When I ask you to understand that sentence, I’m requiring you to take on a pretty significant undertaking, although you do it hundreds of times each day without really thinking about it.

The problem comes with the ambiguity of words.

“I” is pretty straightforward. I’m referring to me. Not much ambiguity there.

“Love” is a little more difficult, but not much. Given the grammatical structure and syntax, we can easily reduce the possible meanings of “love” (of which there are 28, according to Dictionary.com) down to a subset that have slightly different meanings, but all basically translate as, “have a strong liking for.” While there is a margin of error here, it’s minimal.

The word “Jaguars” is a different story. To parse my intended meaning, you have to do some substantial guesswork. According to Wikipedia, there are over 30 potential meanings for the word Jaguar.

For other words, they may pull double duty as both nouns and verbs (love is one example) and the simple subject-verb-object structure indicates that the right candidate would be a noun, not a verb. In this case, we could eliminate all the meanings that are verbs and stick to the nouns. But in this case, all 32 possible meanings for “jaguar” are nouns.

When I say "jaguar" do you know what I mean?

When we analyze language, we subject it to increasingly complex methods of analysis. The first is studying language structure, or grammar. We look at how words are formed (morphology), how phrases and sentences are structured (syntax) and determining the meaning of the word based on how it sounds (phonology). If there is no ambiguity with the words involved, that should be sufficient to interpret the meaning. When we learn to read, we build our skills at this first level of interpretation.

Any potential ambiguity (i.e. knowing the “Dick” in “See Dick run” is a boy named Dick and not one of the other dozen or so meanings, including a detective, an English dessert, or worse) is avoided through the use of accompanying pictures. That’s why we learn to read with picture books. It reduces the linguistic complexity of our world.

But this first line of linguistic analysis comes up short on the challenge I threw at you. We were able to parse the meaning of “I” and “love” but couldn’t determine the right meaning of the word “jaguar”. There were too many alternatives that could potentially fit in the grammatical structure of the sentence.

So now we have to move to the second level of analysis – the study of meaning. Here, we look at the context of words. First, we would look at semantics. Are there inherent clues in how the sentence itself is structures that would help us resolve ambiguity?

For example, I’ve used the plural form of Jaguar, rather than the singular form.

This is a clue that I probably don’t mean the Macintosh operating system, one of two possible comic book characters, one of three music bands, one of two potential films, the chemistry software, the Oak Ridge super computer, the Japanese wrestler or the British research rocket.  If I were referring to any of those things, I would have used the singular form of the term.

Here’s another semantic clue. If I were referring to one specific group, such as a sports team or band, I would probably use the modifier “the”, as in “I love the Jaguars!” But I didn’t. This indicates that I’m not speaking about one specific entity known as “Jaguar” or “Jaguars” but rather a group of things, animals or people known collectively as jaguars.

Because I’ve given you the sentence in written form, there’s one other semantic clue you could use to eliminate potential candidates. I’ve chosen to capitalize the word “Jaguars”, indicating that it’s a proper noun. This would also eliminate the member of the cat family from consideration. But given our rather sloppy approach to capitalization, and the fact that in spoken form, you wouldn’t have had that clue, let’s ignore it for the moment.

So semantic analysis gets us part of the way there, but we still have over a dozen potential meanings to disambiguate, including cars, animals, a game console and several different types of military hardware. Any of these are still a valid semantic fit given the structure of the sentence.

So far, a machine could do what we’ve done. The level of analysis employed to this point relies on rules and comparing alternatives to find the best match, something that machines excel at.  But it’s the next step that really separates the man from the machine.

The Human Element

Given the 20 potential meanings of jaguar, we have to go beyond the clues we find in the sentence itself and use our knowledge of the real world to narrow down the possibilities. This is where things get really interesting.

Now, when I said “I love Jaguars”, chances are you didn’t know there were over 30 potential meanings. You’ve probably never heard of the British elevator research rocket, the Icelandic funk band, the Brazilian cartoonist or the 1979 Filipino film, all known as “Jaguar”. You were probably equally oblivious to the fact that there are at least 6 different pieces of military hardware called a Jaguar.

In this way, in most instances, our limited knowledge of the world helps makes the task of word disambiguation easier. And while you’ve probably heard of the Jacksonville Jaguars NFL team, you probably didn’t know that a international rugby team in the 80’s, a second tier rugby team from Argentina, a Formula One racing team and the teams of Indiana University – Purdue and the University of Houston – Victoria are all also known at the Jaguars. In the case of word disambiguation, ignorance is often bliss.

So, in all likelihood, when I professed my love for Jaguars, you had to choose from three alternatives: the Florida NFL team, the wild animal and the British luxury car. Here, not knowing my personal inclinations, you would have gone with the most likely candidate. And your final choice would have depended on your own world view.

First of all, because I didn’t use the modifier “the”, it’s unlikely that I meant the NFL team. The way I phrased the sentence, it would be interpreted as my loving the individuals that make up the team, rather than the team itself. Without passing judgment on what attracts one human to another, the likelihood of that being my intent is probably pretty low.

Further, even if you missed that subtle semantic clue, if you follow the NFL, you know that it’s been 3 years since the Jaguars made the playoffs. You may also know that I’m from Canada. Therefore, my having a strong attachment to a mediocre team at the opposite corner of the continent seems like a stretch. It’s a possible candidate, but again, without a high level of probability.

Also, in our day-to-day lives, it’s not often that someone walks up to you and declares a strong passion for any particular wild species, especially one we never see. In certain settings (say, walking through a zoo or on an wildlife expedition in Central America) it may make sense, but not within our current frame of reference. Again, the cat is a candidate but not a highly probable one.

That leaves us with the car. Sports cars are something that middle aged men often express feelings of desire for. I fit both categories (“middle aged” and “men”). It’s not unusual for someone to say they love a particular brand of vehicle.

Therefore, given our current understanding of the world, it would make most sense to assign this meaning to the sentence to be understood. While it won’t be the right answer in all cases, it’s the one with the highest probability of success.

I’ve led you through this rather exhaustive analysis of word sense disambiguation to make a point. Understanding language is tough. And, in many cases, it goes beyond applying certain rules; it relies heavily on our experiences as human beings and making a guess about an intended meaning. That’s why it’s so difficult for a machine to do.

Yet, what it’s taken me over a 1000 words to explain, you did in a split second without really thinking about it. The application of multiple tiers of linguistic analysis was done instantly, heuristically and subconsciously. So far, no machine has been able to equal this feat.

For my next few articles, I want to take a deep dive into the problem of language. It’s something artificial intelligence experts have wrestled with for over 5 decades, and while we’ve made some headway with the first two levels of analysis (grammatical and semantic) it’s the final challenge that still lies ahead. How can machines understand language in the same way humans do?

You could consider language the “sound barrier” of artificial intelligence. Until machines can accurately interpret human language, they will be relegated to just being a tool. In humans, true intelligence has always been inextricably linked to language.

For technology to emerge and fulfill the promise laid out for it by writers such as Ray Kurzweil , Kevin Kelly and even Gene Roddenberry, we have to tackle the problem of language. It is an essential part of solving the I/O problem and finding the most human way of interfacing with a machine.

To explore the connection between AI and language, let’s roll the clock back over 60 years to the man widely regarded as the father of computer science and artificial intelligence, Alan Turing.

Meet Alan Turing

Alan Mathison Turing

In a 1950 paper titled Computing Machinery and Intelligence, Turing begins with these words, “I propose to consider the question, ‘Can machines think?”

Knowing that this was a loaded question, Turing quickly proposed an alternative: can a machine imitate a human so successfully that it could fool a human into believing it was human?

A judge would communicate with another human and a machine (both hidden) through transferring text messages (to eliminate the limitations of audio processing at the time).

If after a reasonable period of time, the judge was unable to determine which candidate was human and which was the machine, the test would be successful. This has since become known as the “Turing Test.”

Turing (who died tragically in 1952) would have turned 100 next year. He predicted that machines would be able to pass the test 30% of the time in the year 2000.

His prediction turned out to be overly optimistic. Ray Kurzweil (considered a technology optimist) now pegs the date of successfully passing the Turing Test at 2029.

Regardless of the date, understanding natural language represents one of the most significant challenges facing artificial intelligence. In the next few articles, I’ll be looking at some of the people tackling the challenge and look at how Google and other search engines currently process language.

We’ll see why the current approach to “natural language processing” still falls far short of the bar set by Turing. And we’ll explore why solving the challenge is so important to the future of computing.

By the way, I really am rather ambivalent towards Jaguars of all sorts. This introduces an entirely new challenge in communicating with machines. We don’t always say what we mean. I’ll explore that as well.

More and more, we find search as a relatively inflexible connector between our intentions and our desired actions. Search has struggled to continue as a one size fits all solution to the diversity of our needs. While the search interface has tried to accommodate this by offering more and more types of results through a single portal, it’s unclear how much longer this approach can continue to meet our needs.

One of the main drivers behind this fragmentation of intent is that often are searches are being launched from different devices. And, as Krones found in the study, our search behaviors can vary dramatically from device to device. The required experience on a smart phone can bear little resemblance to success on a desktop or tablet.

Jacquelyn Krones, Microsoft Product Manager

Jacquelyn Krones:

“There’s clearly a different profile of these activities on the different platforms. On desktops and laptops, people do all three of the activities – they conduct missions and excavations and explorations.

On their phones we expected to see lots of missions – usually when you use your mobile phone and you’re conducting a search, whatever you’re doing in terms of searching is less important than what’s going on with you in the real world – you’re trying to get somewhere, you’re having a discussion with somebody and you want to look something up quick or you’re trying to make a decision about where to go for dinner.

But we were surprised to find that people are using their mobile phones for exploration. But once we saw the context, it made sense – people have a low tolerance for boredom. Their phone is actually pretty entertaining, much more entertaining than just looking at the head in front of you while you’re waiting in line. You can go check a sports score, read a story, or look at some viral video and have a more engaged experience.

On tablets, we found that people are pretty much only using them for exploration today. I had expected to see more missions on tablets, and I think that that will happen in the future, but today people perceive their mobile phone as always with them, very personal, always on, and incredibly efficient for getting information when they’re in mission mode.

As a result of these different profiles, you can see that we do different things. Even though we hadn’t put this framework together, we intuitively understood that the kinds of things you want to do on your mobile phone are different than on a PC. We didn’t necessarily anticipate the exploratory behaviour but we certainly did anticipate the missions, which is why right from the home screen of Bing you’ve always been able to get to the kinds of things you’d be doing on your mobile phone very efficiently. If you look at the Bing tablet experience, we built that with the idea and understanding that people would want to explore. It’s a beautiful and delightful experience because it matches what people actually want to do.”

Mission Mode vs. Information Excavation

But how does a search engine know whether you’re in mission mode or gearing up for a major excavation of information? And, if we’re “creating knowledge” through the use of other applications and tools, is a stand-alone search the best match for our needs?

Bill Gates outlines the vision of Longhorn in 2003

Almost 10 years ago, Microsoft was working on a major revamp of their OS that was code named “Longhorn.” Their ambitions were lofty, including integrating contextually intelligent search functionality. Search would cease to be a separate destination and would be a simple right click away from whatever we were doing, without ever having to leave the app.

In fact, Microsoft was working on something called “Implicit query”, where search suggestions would appear in a sidebar based on what you were doing at the time.

While the idea made sense then, and still makes sense today, Microsoft discovered that they were introducing a feature we weren’t ready for. The functionality got scaled back for the release, which eventually became Vista. Yet, the idea still intrigued the folks in Redmond and they decided to re-explore it in their 2007 study:

Krones:

“The last time that we did ethnography, in 2007, we were all excited about this idea – why should people have to go to a destination search site? Search should be everywhere, it should be easy to get to and you shouldn’t even really be thinking about it, you shouldn’t have to break your context in order to conduct a search.

What we found in 2007 was that people weren’t ready for that. They had so much faith and trust and excitement about what search engines were doing for them that they didn’t want it to be where they were. They wanted to go there because it was valuable to them to do that. They felt like that’s where they would have access to not only the best results but also the most comprehensive results. That has changed – there is now an opportunity that wasn’t there in 2007 in terms of users’ expectations and users’ needs and desires.

The best example today would be on mobile platforms, including Win Phone 7.  There’s a search button built into the device and no matter where I am, I hit the search button and it searches, understanding the context that I’m in. If I’m in mail, it searches my mail – if I’m on the web, it brings up Bing and I can do a web search there. I think you’ll see more and more of that.”

Understanding the context of a “Mission” search depends on picking up signals “in the moment” – including location and time. But an “excavation” search presents a much bigger challenge to the search engine. By their nature, these searches involve extended research and aggregation of information.

In the past, I’ve referred to them as “master intents” that we then divide up into individual sub-tasks. We do this because the heavy lifting required resides squarely on our shoulders. Search assists in the role of a “go-fer” but remains ignorant of our overall intent. Krones speculates that this might not be good enough any more:

Krones:

“The interesting thing is that when you think about excavations, there really is no perfect tool for managing your excavation activity. Excavations can range from pretty small to pretty big. If I’m searching for a pair of shoes to wear to a wedding, that can be an excavation for me because I want to find the best pair to match this dress, or it could be a mission or it could be just an exploration.

But maybe I have a health issue and I have to make some really important, life-changing decisions. What tool is there that allows you to search, compare analyze, get input on, save, organize, and come to your ultimate solution? What we found is that search has a very high mindshare for excavations in terms of being a fundamental tool.

Following this research, I had another project that I did with an online panel of 20 people where I had them work on an excavation over a period of 3 to 10 days. Every day they reported back what they did, how they did it and how it went. What I found was, to a person, everybody started with a search engine and almost everyone used more than one search engine throughout their process of excavation. So search is very connected to excavation.

But what it doesn’t do today is allow you to do what users are trying to do on their own – they’re cutting and pasting information from different places, trying to compare, analyze, synthesize and come up with their own answer. For informational kinds of excavations, we hear people say they’re not finished until they start seeing the same information over and over again in different ways from different places and they feel like it all makes sense to them.

You can tell that this is an incredibly effortful process on the part of users. It’s important to them, both from a functional perspective and from their own self-concept as a person who creates their own knowledge perspective as they go. Search is a natural place to start thinking about how you make that easier.”

Specialized Search Apps

It’s these extended online sessions where the greatest promise of what could be exists. The “in the moment” problems are increasingly being addressed by specialized apps that are very good at doing one thing. But let’s raise the bar a little bit. Why shouldn’t there be a perfect application for shopping for shoes or all these other little segmented parts of our lives that can all incorporate search where it’s useful and functional?”

If suddenly we have this broad universe of all these little very specialized applications that we use as required, then you have this need for search to find the right app or maybe the right collection of apps given where we are and what we’re trying to do in our lives at that time. Ultimately, this “meta-app” approach could assemble the right pieces required to help us with the bigger “excavations” as they arise.

Krones:

“One of the reasons why apps have taken off is because instead of having a website that tries to do everything; an app does one thing really well and really easily. Instead of trying to find something, the information is almost instantaneously there, which is a great experience when you’re in mission mode. But you’re also right that what people are essentially doing is creating their own kind of super apps – let’s talk about mobile first, then I’ll talk about PC – by taking all these apps that do one thing really well and stringing them together in terms of a solution that makes sense for them.

So for instance, when we were in San Francisco, almost everybody was a heavy Yelp user. Let’s say the way that I use Yelp is when I’m in a different part of the city that I don’t know very well. I’m with a friend, and I look at what restaurants are around me, then we find one that has a high star rating and a lot of reviews, and we go. And that’s how I use it most of the time.

But let’s say that I’m going to run an errand and it’s a place I don’t know very well, so in addition to picking the restaurant, I also have to figure out how to get there by bus. I think what we would all tend to say, “Yelp needs to integrate bus information,” but what if only 5% of their target user base in any particular city uses the bus information and another 10% walks and another 80% will drive? Rather than make the app like a website that tries to do everything and as a result makes it a little less efficient, people will pair those apps together on their own – they’ll find the restaurant and then they’ll go use their bus app to find the best route.

People are already kind of creating their own customized solutions for the tasks that they do often. And you can see that most clearly on their phones, but people also do this on the Web too. While we would love to say “We’re the perfect shoe store because we can be comprehensive,” that doesn’t really sell well with people. And when you get into something like a health issue where the stakes are so high, that definitely doesn’t sell well with people. We can bring in information that we think is incredibly high quality and trustworthy, but people are still going to want to look at other things.

For things that people do often or that are very important for them, they create their own suite of sites and applications that they trust for this area. When they go out and start searching for a new topic, they know where they’re going to go. They’ll go from site to site to site and do the same query over and over again. Then they use search to make sure they didn’t miss anything – to make sure there’s nothing new and better. But what search isn’t doing for them today is making it easy for them to manage that process they have of using the set of resources they love or trust for a particular topic.”

Microsoft As The Search Underdog

If there was one word that could describe Microsoft’s role in the past, it would be monolithic. The sheer bulk of Microsoft sat at the center of our computing experience. But the world of a million different apps is a far cry from the virtual monopoly that Microsoft has enjoyed in the past. How might Microsoft reinvent itself to be relevant in this new, much more fragmented marketplace?

Krones:

“Our strength is in creating a platform where an ecosystem can do an awesome job of bringing the content and the actual transactional capabilities to play. So our role would really be more about enabling these things to take place across different providers, different publishers and retailers, not to actually create a bunch of small apps on our own.”

Finally, I asked Jacquelyn to look forward 3 years and predict what behaviors she might see when the next study, in 2013, again looks at how we conduct online searches…

Krones:

I really hope that what will have happened by three years from now is that these behaviours that we’re seeing around people creating knowledge and creating their own customized solutions – I hope that the industry has caught up with them.  I hope we see that people are doing this much more efficiently and that they feel empowered not just to create their own knowledge but that the industry has supported them creating their own customized personal solutions. I hope the industry provides what they need as an individual and provides a wonderful experience.

Jacquelyn Krones (Photo: Annie Laurie Malarkey)

I first met Jacquelyn Krones, a Senior Product Manager from Microsoft, at a search show. A mutual friend on the Bing team, Product Manager Stefan Weitz, introduced her to me and said, “You have to meet Jacquelyn. You speak the same language.”

Stefan was right. Jacquelyn started talking about the research project she was then actively engaged in. She explained about the ethnographic approach Microsoft was taking to understanding search behavior in a broader context. Since then, we’ve been able to continue the conversation at various opportunities in the search conference schedule. It’s always been fascinating.

In today’s column, I wanted to share parts of that conversation with you. I had a chance to talk to Jacquelyn recently about some of the findings that have come out of the study. But before that, I want to talk a little bit about the methodology she chose to use.

Qualitative research explores gray, murky areas to uncover insights impossible through more quantitative methodologies. I believe it’s fair to say that qualitative research discovers, where quantitative research either proves or disproves.

As another fan of qualitative research, Ball State’s Michael Holmes, Director of Insight and Research at the Center for Media Design, once said, “Quantitative is essential for refinement and optimization of what you’re currently doing, but it won’t drive reinvention.”

It was the qualitative angle that Krones took with the research that fascinated me. The other thing that was of note was the fact that this was an ongoing project with Microsoft. This is the third qualitative study looking at search user behavior, with the first conducted in 2004, the second in 2007 and the third having just wrapped up, in 2010.

Given the degree of change in not only the core search experience, but also in rapidly emerging areas such as mobile, I was anxious to hear the results from the latest study. But first, I’ll let Jacquelyn explain a little more about the approach they took:

Jacquelyn Krones:

“We needed to be able to look at the search space with new eyes. To me that says qualitative research, specifically ethnography. That would be one of the techniques I would choose first because it allows you to get so deep and so rich because you’re spending time with people in their homes and in the contexts that matter to them.

In addition to being in their homes, we spent time out in the world with them in the places that matter to them. You just pick up a lot more of what’s important to them and what they actually do. You get to find little bits of insights based on seeing them in a way that you won’t see in a quantitative survey or even in a lab setting with a focus group.

We started with a technique called ZMET, which is a technique that’s based more on psychology. The way that ZMET works is that people bring in pictures that represent the topic that you’re talking about. Then there’s a very deep and structured interview process to get at why that picture represents that topic for them. At the end of it, they create a collage and then tell a story about that topic based on their interview and the pictures that they brought in. So that allowed us to create some basic insights about the space before we were even into ethnography.”

So, did the investment in qualitative research pay off for Microsoft? Did they find the big insights they were looking for? According to Krones, the answer was yes.

Krones:

“We actually did come out with pretty big insights for us, the first one being around how we can organize our thinking about the user activities that they’re involved in when they come to search. I like to tell people that nobody wakes up in the morning saying, “I want to go use a search engine.” We really should be looking beyond what they actually do on the search engine if we want to understand what’s important to them when they get to the search engine.

If you look at the breakdown of the number of minutes that people spend online, I think it’s about 5% of minutes are spent with search engines, and we want to understand what’s happening in the other 95% of the time. People are using search engines to make choices about things they do in the real world, it’s not just an online world. Also, we recognize that the definition of what search is has actually expanded –  a lot of what people are using apps for really could be considered search. Even what people are using their social network for could sometimes be considered search.”

The notion of search behavior being dependent on the context of the real world requirements the user brings with them makes perfect sense, but for some reason, it’s not something we typically spend much time discussing when we draft our search marketing strategies. Krones goes on to explain more about the different search “modalities” the research uncovered…

Krones:

“Another big finding was that there are three kinds of activities that are bringing the people to search.

The first one is a mission where people have a specific, well-defined goal and they want to complete a task efficiently.

The second one is excavation where they have a goal that likely doesn’t have a single right answer and they spend more time with it because they want to get the optimal result. They want the best option, the best price, or the most accurate understanding of an issue.

And then there are explorations, which really come in two types – spending time on something that you care about –  continuing to explore that topic, or passing time while you’re waiting in line, in public transportation or waiting for somebody. And in both cases for exploration, people are optimizing for novelty – they want to see things they haven’t done before, they want to be engaged that way.”

The third finding uncovered in the study was particularly interesting because it charted a change in attitudes amongst search users over the span of the three studies. From 2004 to 2007, and on to 2010, the way we looked at search and its ability to answer our questions changed in both subtle and fundamental ways.

Krones:

“I would say the other big insight for us was around knowledge creation, which is also called “sense-making” in the information science academic discipline. We have really seen a shift over the past several years that we’ve been doing ethnography.

In 2004 people really said that knowledge lives with experts and the experts help them make decisions.

In 2007,  people said that search engines actually had all of the knowledge in the world and it was just there for them to go out and pull it out. And now, in 2010, people told us that they created their own knowledge, that even though the search engine never really had all the knowledge in the world, it was linked to information.

People are much more sophisticated now in how they think about that. They say “The search engine’s a great tool for getting access to information, but I need to look at that information and contrast and compare it, and come to my own conclusion about what the right answer is for me. And when I do that, that’s knowledge, but before that, it isn’t knowledge.” People have a sense that knowledge is something that they are actively creating and that is very personal to them.”

If search behaviors can shift that much, depending on the type of search mission we’re undertaking, then it stands to reason that the type of result should also be able to shift to match our intent. Jacquelyn admits that this is a fairly new insight for search in general:

Krones:

“We were really overly focused on always getting the right response in position 1 on the SERP.  That’s still very important, but sometimes that position 1 is an information scent (a cue to information that lies beyond the link) and sometimes it’s actually an answer.

If you’re on a mission search, we might even be able to answer the question for you by either using universal search or in the snippet in position 1 or position 2 or 3, but if you’re in excavation mode, you’re not actually going to find the answer on the SERP and nothing that we can do for you can actually allow you to say that we nailed it on the SERP because people feel like they need to go through this process.

Our biggest insight in 2007 was that in addition to people looking for what one user called “close doors,” which means “Get a definite answer, move on,” people come to search to open doors –  to open their mind and learn about a topic or to explore. Now in 2010 we went even further beyond that, even though we didn’t start out saying, “Okay, what is there beyond exploring and getting an answer?”

What came back, however, was that we found that there were these three activities and that exploration is its own special thing. If you look at the things that are goal-directed, there really is a pretty big difference between searches where you’re looking for a specific answer and then be done and searches where you really want to get the right answer and you’re going to invest a lot of time. There’s a big difference between optimizing for being efficient and optimizing for being thorough.”

I’ve written before in this column about the “Google Habit”, and I had to ask Jacquelyn if this shift in behavior opened the door to breaking this habit. If our habitual search behaviors weren’t yielding the expected results, might we change those behaviors?

Krones:

I don’t think people actually even are aware that they’re changing their behaviour. What they would tell you is that it is more that the environment has changed and that there’s more information available to them, more of what they need available to them in more contexts. They don’t really see it as “I would have done this on my search engine before,” – they see it as a new behaviour.

The second thing is I would say that is that because people are more sophisticated, they’re becoming more demanding about what they want from a search engine. There’s less glow around the power of a search engine than there was in 2007, which is just a natural part of the product category maturing.

If we really want to delight people and surprise them, which are the things that would cause them to break their habit in a conscious way, we have to recognize that the user’s demanding more, they have higher expectations. I think that’s really exciting actually, because we used to feel like any change to the current search model was just going to be immediately rejected by everyone. I don’t think we feel that way anymore.

In my next Just Behave column, I’ll continue my conversation with Jacquelyn Krones, where we explored more about the three types of search activity, how it plays out across different types of devices, searching through apps and what this all might mean for interfaces and advertising in the future.

Checkmate…Finally!

Kasparov and Deep Blue

The real question isn’t why we finally were defeated by a chess playing computer, but why it took so long. Chess is a game that computers should excel at.

The whole point of the game is analyzing possible moves and picking the one that yields the highest probability of a successful outcome. That’s what computers do.

It’s actually rather amazing that humans stacked up so well against the best that IBM could throw at it for as long as we did. The 1997 match was not the first duel between man and machine.

It was simply the one the machine won. Previous to that, Kasparov and others had consistently bested the most powerful computers in the world. And even the 1997 match wasn’t a blow out. Deep Blue won 3 and a half matches to 2 and a half.

How could we compete against something that could process 200 million positions a second? The human brain can’t come anywhere near that level of mathematical dexterity. Our ability to mathematically evaluate positions can be numbered in the dozens per second. If playing chess was all about processing math, we would have been bested long before 1997.

But humans are extraordinarily good at split second processing based on intuition and pattern recognition. What Kasparov could do by instinct took millions of MIPS  (Million Instructions Per Second) of processing horsepower.

In fact, Kasparov protested that Deep Blue had an unfair advantage in that it was able to analyze hundreds of Kasparov’s past matches, looking for patterns, where he didn’t have the same advantage. Also, Deep Blue didn’t do it alone. IBM programmers were allowed to go in and tweak programs in between matches, keeping Deep Blue from falling into traps laid by Kasparov. It’s actually amazing, when you consider the odds stacked against Kasparov, that he did as well as he did. And it wasn’t because he was a better machine than Deep Blue. It was because he was human.

In the 14 years since the match, computers have become exponentially more powerful. And, if we’re benchmarking computer performance against humans, they needed to substantially raise the bar. Because, unlike chess, most of the things we humans do deal with ambiguity and nuance. We were built to deal in messy and uncertain environments. If the advantages humans have allowed us to compete against a computer in a test as mathematically precise as chess, imagine the advantage we have in the organic world.

This Is Jeopardy

It’s that world of ambiguity, represented by human language, that IBM chose as its most recent man vs. machine challenge.  The game show Jeopardy provided the forum, and this time it was a machine called Watson that was the challenger. Watson came to the Jeopardy stage, prepared to take on the all-time champs, Ken Jennings and Brad Rutter.

IBM's Watson on Jeopardy

Jeopardy presented a much bigger challenge to IBM than chess did. To win, Watson had to be able to understand human language, especially difficult given that Jeopardy turns the typical grammatical structure on its head, providing the answer and asking contestants to provide their response in the form of a question.

It we were just measuring the ability to store data (something we humans call memory) there would be no contest. Watson would blow us away. The entire recorded history of man could be stored in its memory bank.

For humans, the limiting factor was the amount of trivia we could cram in our cranium. But for Watson, the challenge was interpreting the question and knowing which information to access and present back as a response.

One of the biggest challenges IBM has ever taken on (the same problem, incidentally, that Google struggles with every day) was something we humans do instinctively, without thinking. It’s another example of how astoundingly efficient our brains actually are.

The Human Part Of Usability

My point, and there is one, is that we consider user experiences and test usability, we have to have a fine appreciation for what makes humans human. All too often, usability testing relies on reams of data, crunched and analyzed in a zillion different ways. We examine bounce rates and benchmarks, as if our users were machines and the answers we seek can be arrived at mathematically.

The irony of usability is that, most often, we try to understand what humans want without ever talking to one directly. We rely on a spreadsheet to reveal the mysteries and subtleties of the human condition. We reduce the magnificence of the human brain to nothing more than a machine, something that can be understood by examining inputs and outputs.

Let me give you one brilliant example of true human based testing I once heard at a conference. Motorola Senior VP and Chief Marketing Officer Eduardo Conrado was talking about how they test radios used by emergency response teams.

Motorola was testing a new model that had just been released. The radio had already gone through their extensive in lab testing and design process. The prototype was now ready for field testing. This is when Motorola actually goes out on first response calls and watches how their radios are used in real life situations.

Despite all the previous testing, Motorola’s researchers soon realized they had a problem. As part of the redesign, they had tried to reduce bulk, introducing a smaller, more efficient radio. The reasoning, which was logical, was that the first responders would appreciate not having to carry around bigger radios. But there was a flaw in that reasoning.

It’s only obvious once you see it…

First response situations are incredibly stressful. They demand an extraordinarily high (sometimes super-human) level of performance on the part of the response team. The human body prepares for this anticipated demand on its resources by cranking up its metabolism. The heart starts beating faster. A lot faster.

Based on a study from Indiana University, it found that during a fire, a firefighter’s heart rate can approach 100% of their maximum for prolonged periods. By comparison, a world-class marathoner typically runs 85 to 90% of their maximum heart rate in a race. The body also signals the release of adrenaline and other neuro-stimulants to allow the body to perform in the required high-stress situation.

For the average first responder, the stress on their body while on the job would be the same as if they had just exercised full out for several minutes. Imagine then the challenge of trying to use a smaller, slimmer radio. The problem was immediately obvious to the field team – “The buttons are too small!” In the lab, the new design was perfect. In the real world, in the hands of real people, it was unusable. The crew’s hands were shaking too much to be able to use the smaller controls on the radio. The design was quickly modified.

So why are these human factors typically absent for much of what passes for usability testing? I suspect it’s because human factors are very difficult to measure. Thing like intuition, habit and emotion, all of which can significantly impact a user experience, can’t be quantifiably measured.

By their very nature, they require human interpretation. It’s the same reason why IBM’s Watson, for all it’s sheer processing power, still can’t chat with you about your upcoming vacation or how your kids are doing in school (which, incidentally, was proposed by Alan Turing as the ultimate test for artificial intelligence).

The only way to understand the human element is to use human-to-human methodologies. It can be a simple as observing behaviors of actual users, or as complex as a large-scale ethnographic study. Whatever route we choose to take, it’s essential that we not loose sight of this human element. We are not machines. We are far more than that. Consider this for a moment. It took IBM researchers and engineers years to create a machine that could best Garry Kasparov at a game of chess.

Eventually, they succeeded. But it was a machine that could only play chess, albeit at a very high level. Kasparov could also protest totalitarian regimes, write books, file a patent application and, one supposes, show love, reciprocate friendship, reflect on sunsets and appreciate art. Deep Blue, or Watson, or any other machine, has never accomplished any of those things.

As these engines boosted revenue (because showing more ads will almost always boost revenue) they also saw their market shares decline. Google has generally respected the 3 ad maximum in the top spot. Over time, Microsoft and Yahoo have also come to abide by this upper limit on ads. So, why is 3 the magic number? Or, is it always the magic number? I’ll come back to this in a moment.

There are user experience dynamics that happen in this relative small piece of real estate that are interesting to explore, because the serve to help us understand the search interaction in greater detail. All the major engines now spend a lot of time thinking about how to maintain top of page relevancy.

Predatory Scanning

First, let’s revisit how we scan a search results page. We have developed a pretty efficient F-shaped scanning strategy, starting in the upper left of the search results page. Why the upper left? It’s because our brains automatically create short cuts based on probability of success. Engines, more often than not, show the most relevant result in the upper left. If this piece of real estate typically delivers what we’re looking for, after awhile we’ll just naturally start there without thinking.

You might argue that in our culture, we read top to bottom, left to right so that would lead to an upper left bias. This contributes, but if Google and Bing suddenly started showing the most relevant result half way down the page, we’d soon adjust our scanning strategy to make that our entry point. Search scanning patterns are much more about probability of success than they are about more arbitrary cultural guidelines.

The Golden Triangle, Enquiro 2003

Scanning a search page is really a hunt for the most useful information on the page, and as Pirolli and Card discovered in the early 90’s at PARC, we have adapted evolutionary mechanisms to assist us there. We use the same basic strategies that we developed to hunt for food.

We look for cues, or, to use the analogous term, “scent”. And efficiency is the rule. We’re none-too-patient in our hunt for usefulness. We start at the top left, scan down the left hand side of the results in a vertical path (the upright leg of the “F”) and, if we find something of interest, scan laterally on the title (the horizontal arms of the “F”).

This creates a triangle shaped scan pattern which I christened the “Golden Triangle.” In our many studies over the years, we’ve found the typical session time, from first scan to first click, to be in the 10 to 12 second range. And in that time, we scan approximately 4 to 7 listings. This provides a clue as to why 3 ads at the top of the page seems to be the upper limit that users will tolerate.

Our Relevancy Consideration Set

When comparing options, humans have built in limits of alternatives we can hold in our working memory. Previous research indicates it’s about 7 discrete pieces of information (more recent research shows it’s probably closer to 3 or 4), plus or minus two. This seems to hold true on the results page, given the number of listings we consider (in the 3 to 5 range, based on our observational studies).

We like to compare a small set of potential results, and then pick our top contender from that set. If nothing in the set seems to be a good candidate, we start moving down the page (we’ll come back to this in a moment).

So, what are we looking for in that first consideration set? We certainly want relevance, but what is our definition of relevance? Here we come to some fairly nuanced biases on the part of the user.

Most users seem to know that there’s a difference between the top results, bounded by a shaded box, and the results that appear below that. The visual boundary between the two sends a cue to the user that they should be considered separately. And although that user may not be familiar with the specifics of a search engine algorithm, let alone the intricacies of an advertising quality score, the majority still know that the top results are more commercial in nature and the lower results less so.

In the user’s mind, less commercial equals more trustworthy, so the top organic result becomes a sort of usefulness baseline. Users use it to compare other results against. And this is where we see why too many ads on top starts to erode user confidence.

If an engine puts 4 or more ads on top to be considered, they haven’t left an available memory slot for the baseline listing in the top organic spot. They’ve forced the user to either consider only ads or to break their natural scanning pattern and skip further down the page (which the brain hates to do). This will almost always generate more clicks in the sponsored ads (at least, it will generate more first clicks) but the engine will pay a price.

By not allowing the user to follow their natural inclination to compare the ads against the top organic listing, they start to erode user confidence, which will eventually erode market share. This is the lesson Ask, Yahoo and MSN all learned the hard way.

Quality Is Not Just About The Numbers

It’s not just the number of ads that factors into user confidence. The quality of ads is also important. Ideally, for a commercial query, the ads should be more useful results than the organic ones. This is why Google has put such stringent quality thresholds on ads that appear in the North position.

This point was driven forcefully home in a study we did. We took a group of users and split them in two. We showed both groups a page of hypothetical search results from a generic engine and asked them to pick the most relevant result. The two pages of results were identical except for one thing – the ad at the very top of the results. The first group was shown a highly relevant ad and the second group was shown a marginally relevant ad. After their search interaction, we asked both groups four questions:

• Would you use the search engine again for a similar type of search
• Would you use it again for any type of search?
• Would you make it your preferred search engine?
• Would you recommend this engine to a friend?

Enquiro, 2009

The difference between the two groups was astounding.

• For the first question – would you use the engine for a similar search, only 5% of the group shown the less relevant ad said yes. 75% of the group shown the relevant ad said they would use the engine again.
• For the second question – would you use this engine for any type of query – 17% of the less relevant ad group would give it a second chance, compared to 68% of the more relevant ad group.
• For the third question – would this engine become your preferred engine – only 5% of the less relevant ad group would add it to their favorites, with 31% of the relevant ad group prepared to make the commitment.
• And finally, the acid test of user satisfaction – would you recommend this engine to a friend? 18% of the poor ad group would spread the word, compared to 53% of the relevant ad group.

These numbers are pretty amazing, considering the only thing different between the two groups was the quality of the first ad. Everything else on the page, the other 20 sponsored and organic results, was identical. It brings into stark relief the importance of quality in that critical first slot.

Commercial Slumming

So, if ads are this important, it speaks to the importance of a huge ad inventory. In the early stages of the Microsoft/Yahoo partnership, Steve Ballmer mentioned something that was critically important:

“The fundamental basis for doing the search deal with Yahoo has to do with critical mass in the advertising marketplace. It doesn’t have to do with technology, or any of these other things, it really is a market phenomenon. Together we would have more advertisers… which means we’d have more relevant ads on our page.”

The study I talked about previously showed the importance of highly relevant ads at the top of the results. But quality is not just about relevance. Quality is also determined by the appearance of trusted brands. In another study we conducted, we found that in markets where major brands have been slow to adopt search, the top sponsored results quickly become the equivalent of a commercial slum.

Major retailers in some markets (including my own home country of Canada) have been regrettably late in coming to the search marketing party. For many searches on retail consumer products, the only sponsored results you’ll find come from online sellers and aggregators. In most cases, the brands are totally foreign to the average consumer. The major retail brands that dominate the bricks and mortar world are conspicuously absent from the top of the sponsored search results.

In the study I mentioned, we again simulated a search experience for users from a well established  international consumer market, this time on Google, and tested click through rates when we showed ads from the market’s biggest retail chains. These chains had never run search campaigns (nor had any of their main competitors) in our test market, so the top of the search results were typically dominated by lesser-known rivals.

Our baseline group was shown real results straight from Google. Our test groups was shown the same results page, but with an ad from the top retailer inserted at the top of the page. We then tested click throughs across the two groups for a number of searches, including common consumer queries such as types of clothing and popular children’s toys.

In the baseline group, we see anemic click through rates of 1 to 3% in the top sponsored ads. Not really surprising, considering the obscure nature of the advertisers. But even we were amazed by the click through rates we saw from our test group that were shown ads from the familiar retailer. For generic product queries, our test ads captured click through rates that exceeded 30 to 35%. One out of every three visitors clicked on an ad from a retailer they recognized and trusted.

In time, with more competition on the page, we would expect to see these click through rates drop down to a more sustainable level (typically in the 8 to 10% range), but this shows how search categories where recognized brands have been slow to play offer a significant early adopter advantage to the ones willing to test the waters.

One last comment on the power of quality and relevance in ads. During one test looking at search behaviors in a group of participants looking for lap top computers, we noticed a fairly high percentage of them quickly scanning the top sponsored and organic results, then, without clicking, looking over at the top sponsored ads on the right hand side. This is relatively unusual. These ads are usually only referred to by 30 to 35% of visitors, and then it’s typically later in the scanning session. But in this study, a high percentage of our subjects were checking out these ads within the first 5 seconds.

Finally, our research coordinator asked one of the participants why they were looking up at those ads, the response was:

“I was looking for the ad from Dell. I don’t see them anywhere in the results and I expected to.”

Sure enough, as we were mocking up the results page, we didn’t include Dell as a test brand. At the time of the study, Dell dominated laptop sales, with over 1/3 of the market. When our users didn’t see a brand they expected to in the results set, their confidence in the quality of the results, and, by extension, in the engine itself, started to erode.

One search executive recently said to me, “I have a whole team of people looking at top of page quality, trying to get the balance just right.” No wonder. When even one irrelevant ad can erode user confidence to the extent we saw in our studies, top of page quality is a matter of life and death for engines.

In my last column, I started to sum up the overarching themes that emerged from these conversations. Today, I wanted to wrap up the journey I started almost a year and a half ago.

Searching From Different Devices

One of the biggest catalysts of change in search has been the adoption of different devices from which we launch our searches. Suddenly our searches drop into a much broader spectrum of contexts. The search we launch from our smartphone can look substantially different than a search launched from a tablet or a desktop.

We have different intents, different expectations and different ways of interacting with the device. One size fits all search just doesn’t fit that well any more.

Stefan Weitz, Director – Bing Search:

“This notion of “doing a search” is rapidly becoming outmoded. My daughter is five and as you can imagine, my house has 12 machines running at any given time. There’s an iPod touch that’s on the web and she uses it to query things on. Her notion of “going to” an engine and searching is pretty comical. She just assumes that everything is always knowable, instantaneously, at her fingertips through some mechanism.

It’s amazing what you can do with one of these devices from a search standpoint when you think of all the information we understand. We understand your latitude and longitude, so we know your location. We understand better your previous queries because unlike PC’s, the mobile device is fairly personal. We can push things to (users) without them even having to ask, which I think is the thing that excites me the most.”

Shashi Seth, Senior Vice President – Yahoo Search Products:

“I believe that in five years or so, mobile search will have overtaken desktop search. And the areas where that is going to grow the fastest is going to be in Asia and Europe.

If you have the kind of interactivity that a smartphone like an Android phone or an iPhone offers in terms of the touch interface—the ability to move from app to app—and if you had a larger screen, would you be able to do significantly more? I think the iPad has definitely answered that question. Given the interactivity of these tablets, you can build an experience that is what you’d want the next generation of desktop experiences to look like. It definitely opens up a range of possibilities that have never existed before.”

Johanna Wright, Product Management Director – Google Search:

“Mobile is a key part of our strategy and opens up many possibilities for the future. With smartphones getting better and better, people are doing more and more searches. Phones also enable new kinds of search input. For example, today users can search by voice by speaking a query into their phones. They can also search by sight by snapping a picture in Google Goggles. They can even search by location by simply clicking “Near me now” on their mobile Google homepage. The phone is a uniquely intimate device, and this invites further personalization and other exciting possibilities for the future.”

A Shift In The Balance Of Power In The Search Biz

A recurring theme that has come up through all of the conversations (instigated by my continual prodding) has been how different user expectations may shake up the search market, which is dominated by Google.

Competitors have been unable to make much of a dent in Google’s dominant (almost monopolistic) position in desktop search, but if one thing has become clear in these discussions, it’s that the days of desktop search are numbered. Can Google shift their decided advantage into the new emerging search markets?

John Battelle, Founder – Federated Media and author of The Search:

“I think that there are significant business model implications there and I’m pretty certain that they are the obsession of the folks at places like Google, because search as a destination, search as a ubiquitous catch-all is the reason that Google’s dropping billions of dollars to the bottom line every quarter. Search as an application where your first search isn’t the search itself but rather the search for the right application is a very, very different use case. You have the market influence and dominance of one player splintered into tens of thousands of players.

This is the one of the reasons that Steve Jobs and Eric Schmidt (note – this quote came from before Eric’s stepping down from the CEO role at Google) don’t like each other very much right now. We really have an interface war going on and I think it’s fascinating.

This is one of the main reasons that Google decided to get into mobile and to do Android. They are saying, “Well, we’ve got to protect our flag by having a significant play in this next evolution of how people engage with computing, how people organize information and make it available.” That’s their corporate mission, and they had to play there—they couldn’t just assume that the HTML web was going to stay static. I thought it was very smart of them to do that.

The iPad coming out has been an inflection point—an ah-ha moment where they realize that there is a new interface to computing coming. It’s very rare that you launch a new device that already has 140,000 applications built for it, and that’s a pretty big deal. Now to my mind, there’s an awful lot of noise and not very much signal in 140,000 applications. So there’s a big search problem there and I think maybe Google would be wise to own that search problem.

I think it’s a phenomenally important piece of real estate, which is why Steve Jobs controls the whole end-to-end experience. Vertical integration is highly profitable. There’s just not going to be any crawling of the iTunes Store from a third-party developer native on iTunes. It’s going to be Steve Jobs who does search for iTunes. He may not do web search, but app search, that’s him. And now I think that there are opportunities to do that better and to do it across platforms, including netbooks and tablets.”

Shashi Seth from Yahoo touched on this same opportunity. I think in the time since these interviews have been conducted, there have been more and more signals coming from Apple that they’re assembling the pieces required to control this new type of search, especially within their own verticalized empire.

Shashi Seth:

“I think (Google) has a vested interest to continue down the path that they have invested in, and that path is much more around one search box, one user, and one form of advertising. I think it’s a lot easier for somebody like Yahoo! or Apple to shift the game a lot. Apple is in a very interesting place because they’re truly a platform play. They clearly have no intent of getting into the content or the search business themselves, but they bring a lot of capabilities to bear that people like Yahoo! and Microsoft and Google need, and clearly with the separation between Google and Apple that is going to be a different kind of war.”

Not surprisingly, Google was very careful in how they answered the question of whether they will continue to dominate the search business:

Johanna Wright:

“We think it’s a good thing that there are many different search options out there. This gives users more choices for how they want to search and discover information. However, on the other hand users also want a more unified and seamless search experience. Users don’t think in terms of “portals” and “properties,” they want information (or “answers” and “to do something with it” as you say).

Over time, integrating different content types into the main Google experience has been a clear win for users. We did this a couple years back with universal search, where users can start finding images, news, books and other content right in their main search results.

With our more recent results page refresh, we’ve made the interface more consistent across different types of information, so when the user clicks “images” she still knows exactly where to find refinement tools, the search box, and other interface elements she’s used to. We don’t anticipate any kind of splinter of the core search experience on Google.”

The Future Of Search Advertising

When we’re talking paradigm shifts in the industry, it’s not just market share that’s up for grabs. It’s also the form that advertising might take, and in search, advertising equals revenue.

This becomes an interesting opportunity for Yahoo, who have always been a pioneer is testing new ad formats and integration across their properties. Now that they have been freed from the tedious duties of maintaining a search algorithm, will they push the envelope in pioneering new types of advertising for our changing search experiences. Shashi Seth’s comments certainly indicate that this might be the case.

Seth:

“With mobile advertising also comes another opportunity, and that is to break down the artificial boundary between search advertising and display advertising. And that’s another stake in the ground – mobile search is not going to look and act much like desktop search. In fact, I believe that mobile search is going to be in the shape and form of mobile apps and that users are going to engage with it and therefore the kind of advertising that lives there is unlikely to be traditional search advertising.

I think the display advertising experience on a search app on an iPad offers up the kind of advertising that even traditional internet display advertising has lacked. It’s the kind of advertising that you see in a magazine like GQ or Vogue, where ads and content start blending so well together that users don’t even think of them as ads, they actually think of it as content. That’s why I think that the next generation of advertising on mobile devices is going to blend display and search advertising, meaning the targeting can be done in one way or the other, for an audience or for keywords—but the kind of formats that you see on these devices are going to be significantly different and will probably bring in new kinds of advertisers that have shied away from this platform traditionally. If you think about a brand like Hugo Boss or Louis Vuitton, who traditionally have not advertised on the internet because they’ve never gotten the kind of canvas that is needed. But if you bring this new kind of canvas that we’re talking about, which could work especially well on an iPad or even an iPhone/Android kind of device, I think the world is your canvas at that point.

It requires that marketers become much more data-centric in their analyses and their targeting, and that the kind of data that publishers like us have to bring to the table for marketers is going to be significantly more sophisticated than has been done in the past. For example, knowing and providing details about users, their habits, what kind of content are they likely to interact with, what their needs might be, and then changing that advertising and marketing message to fit that individual user is going to become increasingly important. And it’s even more important to do it in the right way on the right device. For example, connected devices like the iPad or a connected TV. I think the kind of ads that you bring to the table have to be very different in each of those scenarios and have to tie in the user’s context, the intent, the device, the location, and so on, which makes for a pretty complex landscape. Much like we’ve talked about search changing, I think advertising is going to change a significant amount as well.”

Moving Into The Future

The most forward looking observations in the series of interviews came from John Battelle and Research in Motion’s Hampus Jakobsson. In those conversations, we moved well away from the safety net of what we currently know about search and speculated about what might be further out on the horizon.

The surprising thing was that no matter how wild the predictions seemed, none of them were further out than the next 10 to 15 years. This seems to indicate that we’re truly at an inflection point with our search experiences.

In my interview with Jakobsson, he laid out a vision of “connectedness” that included increased usage of smart personal devices that collect data specific to you and an increasing number of nodes where you can “plug in” to the broader public database that is the Internet:

Hampus Jakobsson, Strategic Alliances EMEA, Research in Motion

“I see the Internet evolving as a database. It’s a repository of information and you become the unique identifier. We’re seeing an interesting grid – seeing this single identity, you, which is another repository for a lot of different information. In the coming years, the number of nodes between you and the Internet, between the “public” to the right and the “individual” to the left, you’re going to see more and more nodes interacting. That node might be your mobile phone or your laptop or your car or your ambient screen.”

Search & The Hidden Persuaders

What becomes interesting, and more than a little frightening, is the form that union of private and public data might take. This ramps up the discussion about privacy significantly. For example, let’s say our lives become even more digitized and that data becomes available to marketers.

They use it to fuel even more sophisticated recommendation engines, in a drive to take the guesswork out of marketing.The goal: provide the right advertising message to the right person at the right time, at scale and with mathematical precision.

All this sounds great, if you’ve got your marketing hat on. But what if you’re the customer? Does the ability to target ad messages so precisely that buying becomes, quite literally, a “no brainer” (or, at least, a “no prefrontal cortexer”) represent a benefit or a very frightening step backwards in personal freedom?

Here’s the thing. The more we understand about how we make decisions, the more we realize that often (always?) we’re subconsciously responding to environmental cues and then rationalizing it after the fact as a decision or action we made consciously and rationally.

There is an increasing body of evidence to show that our notion of free will is not exactly as we imagine it. This is a contentious issue; so I expect many of you to be falling on the opposing side of this debate, but for the sake of speculation, let’s assume this to be the case.

If our brains are wired to respond to cues, and if those cues take the form of exceptionally well targeted advertising messages, coming to us in the guise of recommendations, have we allowed those advertisers full access to our “buy button”? It may strain credulity, but the field of neuromarketing is advancing pretty rapidly. The “hidden persuaders” that Vance Packard warned us about a half century ago may be quickly becoming a reality.

Jakobsson:

“Are we in control of our destiny? Is there such a thing as willpower? What is in a world where people can suggest possible directions, where we can’t control it anymore? Of course we can control it, but it requires a lot of stubbornness and thinking about what you really want all the time.

I think our children and our children’s children would laugh at the way we gave away information and exposed ourselves to recommendation algorithms and other things, persuading us to do things that we didn’t want to do.  That’s going to be the asbestos of the 21^st century. We’re not really realizing this now, just like we didn’t do with asbestos. We will endlessly jump into these mistakes.

There are immense problems that comes with this and I think one of them is that we’re going to create a lot of services, which running on business models and ideas, which might be completely un-thought through in terms of whether it’s a good idea or not. For example, recommendation algorithms – where does the recommendation end and advertisement start?

When I give up all my data to you, that’s extraordinarily scary because you can present me with information and change my behavior on the one hand. This has always been the case with my Visa card and my Amex. They know everything I buy, they know when I’m getting a divorce…they can actually see when people are getting divorced.  They can predict that six months ahead of the divorce, they see that on a purchase pattern.”

John Battelle also sounded the alarm about this impending collision between privacy and convenience:

John Battelle:

“I think personal feeds and the consumers’ ability to say, “Sure, you can have my feeds because I’m going to see value from it and I know that we’re in a trusted relationship”… I think that that handshake is going to be increasingly made in our culture. I think, however, that we need to have a conversation about that handshake and understand it. We’re in the midst of that and it’s going to get more and more interesting over the next decade. I think that the handshake between large services now and what will become a flood of new streams of valuable data from apps, from interactions on other sites and services will allow a Google or a Microsoft to touch and have access to a ton of data about us. But the bond of trust and the cultural contract that we have with those services is going to have to be very well understood. I think we’re sort of slouching our way there, but we’re increasingly having a conversation about that cultural contract and social contract.”

Search & The Singularity

I’ll leave the last word to Battelle, who finished up our conversation with a philosophical riff that touched on no less a subject than the melding of man and machine:

Battelle:

“But it is the greatest question of search, which is how do we get to the point where we can meld man and machine? I think our whole culture is this sort of grand narrative marching down that particular road, and hopefully by the time we get there we’ll be smart enough to know what it means to have essentially created life. And that’s a big, amazing, interesting concept—you know, created life artificially. And that’s I’m convinced, knowingly or not, why most of the people are in this industry, because it’s just got this sort of sexy, immortal vibe to it—we’re working on a big problem that if we solve it could either destroy us or let us live forever.”

For the next two columns, I wanted to sum up what came out of those conversations and find the common themes.

A Shift In Our Expectations Of Success

A number of people talked about a sea change in our expectations of what search should be. The basic paradigm of web search hasn’t changed very much in almost 20 years: one query box and an ordered list of results.

If you held a search result screenshot from 1996 alongside a result from today, you would wonder how anything in the rapidly shifting online world could change that little in a decade and a half.

Today, however, all that is changing. We are expecting much more from our search experiences, driven largely by how those experiences look in different circumstances: launched from a smartphone, launched through an app or launched with a very specific intent in mind.

Traditional “one size fits all” search experiences are hard pressed to keep up with our changing expectations:

Stefan Weitz, Director – Bing Search:

“This idea of a universal notion of relevancy worked really well in the earlier days of the web. We had a smaller web, we had a more static web and we had a web that really was a web in the true sense of that term.

That was the way PageRank was constructed and that algorithm was quite brilliant. I still think that it’s quite brilliant. But it does seem a bit chaotic that we’re using that same notion. If I’m looking for the best hospital to treat cancer, just think about how ridiculous that model actually is. We’re now relying on popularity and in links to determine that? That doesn’t make any sense.”

John Battelle, Founder – Federated Media and author of The Search:

“We had a very, very basic, well-understood use case for 10 years, which was Google or “like Google”—you put in a couple keywords and you get a response back. And that framework of searching and coming back with the best document to answer a query is morphing. People are asking far more complicated questions now and they’re demanding far more nuanced answers, simply because they know they’re out there.”

One idea that I floated in several of the interviews was a concept I call “master intent”.

Often, when we use a search engine for a specific task, it’s actually part of a much bigger and more complex task. Think about the big things we do in life: buy a house or car, change jobs, plan a trip or have a baby.

These life events (and they don’t all have to be this momentous) bring with them a myriad of  needs for specific information. But it’s up to us to keep the master intent – the big objective – in mind and parcel out the individual searches as required.

We plug in the pieces as we gather them, but the heavy lifting is all done by us. Web search acts as a pretty simply minded assistant in all this – going out and gathering relevant information based on the words we feed it.

But what if search knew our “master intent” and offered a much higher level of assistance, going out and gathering all the information, filtering it based on our requirements and guiding us through the entire process?

John Battelle:

“I used the example of trying to figure out which classic car to buy. In a case like that, I don’t even know what I don’t know, and to expect search to tell you what I don’t know is expecting more than search can deliver. That’s a longwinded way of saying I think we’re going to be disappointed with our results because we’re not there yet both culturally and technologically to deliver what we know as consumers is out there. But we’re going to be. I think that we’re in a transition period.”

Not all the interviewees agreed with my assertion that our current standard of search success – relevancy – may not be the most appropriate standard moving forward. I believe search has to become more useful.

Perhaps not surprisingly, the company that largely defined our current search experience, reliant on relevancy, was the one least apt to call it outmoded:

Johanna Wright, Product Management Director – Google Search:

“Whether we talk about “relevance” or “usefulness” is a very subtle difference. Isn’t more “relevant” information also, by definition, more useful? I would also note that relevance is by no means an easy or solved problem. The ordering and presentation of the top web results is a critical part of the search experience, and one of the key reasons users keep choosing to search on Google.”

The Limitations Of Language

There is a challenge in our current paradigm of search, built on the concept of relevancy. It depends heavily on the ability of a machine to understand human language.

And that is no small challenge:

Stefan Weitz:

“If I walk into a conference room of people and ask why are there no jaguars in this conference room everyone is going to look at me like I have two heads. They would say “why would there be a jaguar in this conference room?” Jaguars are mammals, they are carnivorous. They live outside. Or… Jaguars are cars. In any case, it makes no sense to have a jaguar in a conference room. However, if you go to Google, or Bing, or Yahoo and type in “jaguar” and “conference room” you’ll get back in excess of 5 million hits.

Where we need to get to, and where we’re working to get to, is doing better job of having the crawler and the parser really understand the language. When I say Crest White Strips, think of what today’s indexes are going to do. They’re going to find those words in a PageRank and return those results. The system has to know that Crest is a brand and white strips are a way of whitening teeth. Teeth whitening is done by a dentist. And dentists often don’t like using off-the-shelf products.

You have all the things that I know about Crest White Strips, just from casual human understanding standpoint. The engines today don’t know that. So much of the intent calculation we have to do to deliver a good set of results is bound up in this challenge of us imbuing the engines and the index and the parsers with a more human characteristic of understanding what they’re reading. That will get us to intent much faster than a lot of the mathematical tricks.”

But, perhaps, searches reliance on parsing language isn’t as critical as it was before. As we do more things online, we leave more signals to help a search engine determine our intent.

Much of the conversation revolved around this point: that we now gather more information about the current situation of the user: location, their current activity, their place in the social grid. All these things can augment a query, allowing for a more useful experience.

Realtime Search

There are two converging trends that are changing a fundamental aspect of our search experience: a better understanding of who we are and what we’re trying to accomplish and a web that is much more dynamic and timely.

Together, these two things demand that search becomes more temporally relevant, in addition to just being semantically relevant.

Search is moving from being “stateless” to being very much in the here and now:

Mark Cramer, CEO – Surf Canyon:

“For the past 40 years since the inception of search, the way it’s worked is this: user enters a query, that query is matched to an index of documents, tremendous activity is deployed to try to determine the relevancies of those different documents, and a search result set is produced. But that search result set is static. There’s an order to those results. They go from 1 to 10 and then 11 to 50 million, and that order does not change. It’s stateless.

What we have been doing for the past few years is essentially applying state to the search page in order to make the results dynamic. And if you consider “dynamic” to be something resembling a conversation in the sense that the search result page is actually responding to every input from the user to alter the content on the fly, then I think that’s an interesting way of looking at searching.”

For example, conversations in social media happen very much in real time. As we begin to apply what’s happening in the social graph as an additional ranking criterion to the information indexed in a search engine, interesting things begin to happen:

Stefan Weitz:

“What it does do, especially with Twitter or a lot of these real time services, is provide additional signals to an engine. If you think about Twitter, for example, and how fast things rise and fall, the traditional model of ranking simply doesn’t work. It’s not fast enough.

We can use all these signals of UGC as part of an algorithm that takes into account the user’s intent, the most logical response to that intent and that response is determined by a number of signals: what’s happening in real time, what does your social circle think of these things, who is an authority on this particular topic, and what does that person (or entity) either read or write? There are a number of different factors that we look at.”

The Beginning Of Search As An Application

One of the biggest factors driving a new search experience comes with an evolution in the foundation of the web: the nature of data itself.

In the mid 90’s, almost all data was “unstructured”, a hodgepodge of stuff living outside an orderly database that demanded some type of categorization and organization. Traditional search, even with its limitations of language, excelled at this. It could crawl, index and whip unstructured data into shape.

But today, we have more and more “structured” data online. Typically, the commercial value of the data determines the rate at which it will become structured – a kind of economic colonization of the web. And as data becomes structured, it’s potential usefulness extends beyond the limited capabilities of search as we currently  know it.

Our current paradigm of search is that of a destination we go to in search of information. But increasingly, search is becoming a sort of “meta-app” that allows us to do things.

John Battelle:

“We will go do an insanely sophisticated search on Expedia in a matter of three minutes where we determine which flight to take. That’s all structured data and it’s basically an application, right? Expedia is a search application, it’s a decision support application. We do all that work and then we’ve trained ourselves to think about things that way. Then we’ll go to Google, we’ll put in something that has nothing to do with travel but maybe has to do with buying a refrigerator or a car, and we expect search to do that for us. And it’s not that we are consciously expecting it. It’s just that if you put in “1967 Mustang,” you’re hoping the right answer comes back when in fact you need to do a 2 to 3 minutes structured search using an application, not using sort of a vanilla generic search engine.”

The power of this new paradigm, that of search as an application, truly becomes powerful when we start to transfer functionality across different applications:

Battelle:

“Look at the valuable information that you can extract from how any one of us interacts with a well-designed application, and then create a dataset for that. Say I use the New York Transit application to navigate my way through New York for 3 or 4 days… all of the questions and back-and-forth that I use that app for, which is essentially a structured search session—right?

Now, match that against a set of data which is the transit map. I say, “I need to go over here. I want to go over there. I prefer this route over that route,”—that becomes a dataset that should inform other searches that I’m making on things that seemingly are unrelated but may not be. That should be available as metadata for future searches. And figuring how to inform that is as important as parsing the line or the spoken phrase that I’m making in the moment.

Now, if I take that spoken phrase and go and search for “Chicago rental car” four months after interacting with that New York Transit map application, how can we take the metadata from that interaction with New York and inform the appropriate response in Chicago. Perhaps the best suggestions would be, “Hey, you know what? You don’t need to rent a car. You can use the Chicago Transit. Here’s an app for it.

You can get from the airport to everywhere you want to go without having to rent a car. Plus, you’ll save $150 which we know is a goal of yours because you’ve been interacting with the Mint application and it said that a goal of yours is that you want to save $200 a month and here’s a way that you do that”?

Tying all that together, that’s the Holy Grail because then it starts to understand you.”

The Holy Grail indeed. Usefulness starts to expand exponentially as we tie more and more information into our objective. Even our choice of app given our intent starts to provide more signals to help clarify what it is we want to do:

Shashi Seth, Senior Vice President – Yahoo Search Products:

“I think people have narrowed down that master definition or master intent up front by choosing the app that they choose to engage with.

So, for example, if you are looking for local results, you may use a very appropriate local app in order to do those local searches, and therefore there is no guessing to be done anymore. By nature of what you just did, we already know that you are looking for local results and that is the only index that I have. When somebody uses a mobile app and we know what the GPS coordinates are and the fact that they’re using a local app.”

Search & Privacy

So, the future of search may lie in this concept of “usefulness.”

But usefulness depends on a search engine’s (and we may even debate the appropriateness of that label in the future – is a “search” engine truly what we need?) ability to know who we are, what we’re doing and what we hope to accomplish. And that level of transparency comes at a price. What happens to our privacy in this new paradigm?

John Battelle:

“I think personal feeds and the consumers’ ability to say, “Sure, you can have my feeds because I’m going to see value from it and I know that we’re in a trusted relationship”… I think that that handshake is going to be increasingly made in our culture.

I think, however, that we need to have a conversation about that handshake and understand it. We’re in the midst of that and it’s going to get more and more interesting over the next decade. I think that the handshake between large services now and what will become a flood of new streams of valuable data from apps, from interactions on other sites and services will allow a Google or a Microsoft to touch and have access to a ton of data about us.

But the bond of trust and the cultural contract that we have with those services is going to have to be very well understood. I think we’re sort of slouching our way there, but we’re increasingly having a conversation about that cultural contract and social contract.”

Editor’s Note: Batelle recently also had some interesting commentary about whether or not Apple will get into search, and how he saw Apple’s role in the new search paradigm:

In the next column, I’ll continue the summary by looking at how our search activity is increasing coming from a platform that’s not a desktop, what may lay ahead for search advertising and how all this might shift, if not shatter, the balance of power that currently exists in search. And we’ll wrap up with some truly mind blowing projections of what our relationship with technology may become.

As we continued to chat, we went into some fascinating territory that is literally right out of science fiction. TAT (short for The Astonishing Tribe) designed user experiences for new interactive screen technologies.

One of the many fascinating topics we discussed was this idea of personal and public screens and how they may interact together. Jakobbson talked about how our notion of “personal” technology has changed in the last decade:

Jakobbson:  I think there’s a big, big trend moving in one direction, which is the personal screen, the one I am holding in my hand. I think you’re seeing interesting consumer patterns there, for example, with mobile phones actually pushing away what other devices used to be and the way we use them.

We used to call our laptop personal computers but nowadays laptops and other computers are more like workstations in most people’s minds than they are personal computers. It might be mine, legally, but if you want to lend it, sure no problem, that’s not an issue. But if you want to borrow my mobile phone, that suddenly makes me very tense because I have a lot of private stuff on my phone and what if you break it?

So we are creating a new definition of “personal” technology, that’s much more intimate. But, at the same time, there is a new type of interface to the Internet, the public one. Increasingly, we are seeing interactive displays and kiosks that allow us to access tailored experiences, usually commercial in nature.

Jakobbson:  And we’re seeing this coming up, thanks to gesture interaction and also face recognition technologies. The face recognition then would make it easy way to log in, just stand in front of the object and it will recognize your face.

This is Emily (using the example of the interactive mirror from the TAT video), she doesn’t have to log in or anything complicated. It’s a zero click user interface and it comes up with her personal ID. Of course, in a lot of these scenarios, you can’t touch the objects, like the mirror, because you smudge it. And, in a lot of cases, the screen might actually be distant physically or be behind a window.

For example, in a couple of years…you actually can do it now in Korea and some places…you can go to a store which is closed and stand in the front of a window, where you can navigate the different things they have in the store with gestures and buy them at 4:00 AM, thanks to a large monitor that’s facing the window and a camera that recognized gestures.

They have actually put an iris ID scanner or something similar, which allows you to log in and purchase. These things are really happening right now and there are a lot of interesting technologies that are being used and put into play, which have been around for forever.

This is where the worlds of science fiction and fact start to collide. For Steven Spielberg’s 2002 movie Minority Report, he gathered a bunch of futurists in a hotel room and asked them to brainstorm what the world might look like in 2054. One of the outcomes was the famous advertising holographic scene, where disembodied ads recognize Tom Cruise’s character John Anderton and start accosting him as he walks down the street.

The technology Jakobbson is talking about is actually more sophisticated than that, and it’s here 40 plus years ahead of schedule.

Of course, it’s not just in department stores where we find ourselves surrounded by this new “ambient” technology. Even your drive home after work is increasingly becoming a “wired” experience.

Jakobbson:  We work with a lot of car companies right now and one of the interesting things there is that you’re seeing people use a very, very old technology that has been around for at least 50 years, which is eye tracking.  A big problem when you’re sitting in a car, of course, is that you don’t want to be distracted. You can’t have information everywhere.

What’s happening now though is people putting in a small camera that looks where your eyes are looking. With that, it can tell you everything from people falling asleep, (I think that’s already installed on some Volvo cars) to people actually using it also to present information. You see where the driver is looking and you can customize what’s being presented.  Each context requires its own user interface and experience.

I think that there are very vulgar adaptations, where people have taken a game experience, a heavily immersive experience and put that in a car and said that it’s going to be the car of the future, where you drive and Tweet while driving and check your Facebook and read your email. That is, of course, extremely improbable, because then it have to be a world where we’re not actually responsible for driving anymore.

Plugging Into The Grid

This becomes an interesting world, where we have increasing intimate forms of personal technology and more powerful forms of ambient technology. What becomes fascinating to consider is where these two worlds might merge. If we have this grid of functionality and information, and a heavily personalized “key” to it, how might that impact our future experiences?

Jakobbson:  I see the Internet evolving as a database. It’s a repository of information and you become the unique identifier. We’re seeing an interesting grid – seeing this single identity, you, which is another repository for a lot of different information.

In the coming years, the number of nodes between you and the Internet, between the “public” to the right and the “individual” to the left, you’re going to see more and more nodes interacting. That node might be your mobile phone or your laptop or your car or your ambient screen.

This “plugging in” creates a slightly frightening scenario that again smacks of science fiction (i.e. The Matrix – supposedly set in about 2199). It completely upends our idea of “personal” information. And this is not some far-off problem to consider. The technology exists today, as Jakobbson was quick to point out, using a company called FitBit as an example:

Jakobbson:  FitBit is a glorified motion sensor. You clip it to your pants and it measures how you move. It contains the same technology as the Nintendo Wii. The interesting thing is it also does it while you sleep. You can clip it onto yourself when you sleep and you get a perfect pattern – you actually know exactly what happens 24/7.

The data is stored on a web page –  you can view it, it can be private or it can be public.  But the interesting thing is you’re actually digitizing information which is you. It’s your movement pattern and that might be extraordinary beneficial if we look five years ahead for health information. Let’s say that in five years you get a liver issue and you go to your doctor. What if the doctor could get your movement patterns for the last five years, measured every second? It’s data that’s going to be crucial for health in the coming years.

FitBit is just one example of a new form of very personal technology. More and more, aspects of our lives are being digitized and stored in the cloud. With biometric monitoring, there’s a huge potential win for healthcare. And for this reason, we may put aside any privacy concerns. But what about other privacy concerns?

As more aspects of our lives begin publicly available, a converging picture that strips our private lives bare and reveals all becomes likely. And then, I wonder, what will this information be used for? What new business models, under the thin disguise of personal usefulness, will emerge? And, when they do emerge, will it open a Pandora’s box of social impact that will be difficult, or impossible, to close again?

Even if we just consider what this vast database of personal information might mean in terms of advertising targeting, the implications can be frightening.

Jakobbson:  There are immense problems that comes with this and I think one of them is that we’re going to create a lot of services, which running on business models and ideas, which might be completely un-thought through in terms of whether it’s a good idea or not. For example, recommendation algorithms – where does the recommendation end and advertisement start?

When I give up all my data to you, that’s extraordinarily scary because you can present me with information and change my behavior on the one hand. This has always been the case with my Visa card and my Amex. They know everything I buy, they know when I’m getting a divorce…they can actually see when people are getting divorced.  They can predict that six months ahead of the divorce, they see that on a purchase pattern.

Visa and MasterCard and Amex didn’t use to ask me, “Hey, do you want to go to the Caribbean?” They didn’t do that. Now, they do (through targeting and retargeting). They can see patterns and can suggest things, which are going to be subconscious to me. They can see it in my patterns.

The Illusion Of Free Will

There is a debate in the worlds of neurology and psychiatry about whether or not there is such a thing as free will. Do we really control our destiny, or are we simply responding to cues in our environment? There is neurological evidence that our brains send the commands to act much faster than could be possible if we were actually willing our movements.

This theoretical debate becomes more controversial if, in the world imagined by Hampus Jakobbson, our environment is literally being molded on a moment-by-moment basis for us through interpretation of our data patterns and presentation of suggested directions and opportunities.

Does consumerism reach new Pavlovian heights then? Are we leading our lives, or simply being pulled through them? Is the lure of technology too beguiling, and will we be opening the door to an evil that it may take us decades to recognize?

Jakobbson: Are we in control of our destiny? Is there such a thing as willpower? What is in a world where people can suggest possible directions, where we can’t control it anymore. Of course we can control it, but it requires a lot of stubbornness and thinking about what you really want all the time.

When we put in asbestos in our homes, everybody thought it would be great idea. Now we know that asbestos is a really stupid thing, just like DD. Now it’s obvious that it’s really stupid.

I think our children and our children’s children would laugh at the way we gave away information and exposed ourselves to recommendation algorithms and other things, persuading us to do things that we didn’t want to do.  That’s going to be the asbestos of the 21^st century. We’re not really realizing this now, just like we didn’t do with asbestos. We will endlessly jump into these mistakes.

It’s a lot to think about. But we may not have as much time as we imagine. A recurring theme of my conversation with Jakobbson was the accelerating timelines of technology. Things we can imagine and place in the far-off future seem to become reality much faster than we anticipate.

The challenge with this is that the pace of technological possibilities is far greater than society’s ability to determine the impact. As Jakobbson warns us, the things that look wonderful today may carry a far greater price than we’re willing to pay, but we won’t realize it until it’s too late.

I reached out and made contact with Hampus Jakobsson, TAT Co-Founder and now Strategic Alliances EMEA at Research in Motion. Hampus and I spent about 40 freewheeling minutes on the phone and in those 40 minutes, we covered a lot of fascinating territory.

Of course, fate and technology had to intervene. The audio quality of the interview left a lot to be desired. We salvaged a transcription that captured the spirit of the conversation but is not of sufficient reliability to allow for extensive verbatim quotes. So I’ll paraphrase where required, but I think you’ll find the topics covered thought provoking to the point of occasionally being jaw dropping.

The Future Of Search Behavior

As we think about what search might look like, we have to think about where those searches might occur. Today, our online experiences are pretty tightly bound to a handful (literally, in the case of mobile) of devices: a desktop or laptop, a smartphone and possibly a tablet device.

In each case, the device, while a possession of ours, really doesn’t know much more about who we are or what we want than our television, dishwasher or kitchen table does. They are, in most instances, an appliance, despite their digital capabilities.

But this is not the world envisioned by TAT. Here, the world is composed of online nodes, some public and some intensely private. We plug in and plug out of the grid as we live our lives. We may carry bits of technology with us, in our pockets, wearing them as clothing or even have them physically embedded in us (being researched as we speak in an Intel lab somewhere).

The divide between technology and biology is being assaulted on multiple fronts. Nanotechnology is already creating microscopic biological computers. The world of Ray Kurzweil’s Singularity is rapidly approaching.

If we are to consider the future of search as an experience, we have to begin to imagine the environment those experiences will take place in. And that is the ground I wanted to cover with Hampus Jakobsson from The Astonishing Tribe.

I started by asking him about a favorite interest of mine, the behavioral divide between those that grow up with technology (often called Digital Natives) and those that adapt to it as adults (Digital Immigrants). As someone who engineers experiences, how do you approach these inherently different attitudes about technology?

One of the challenges identified by Hampus was in how both groups accept technology. “Will it be used?” becomes a key question. Natives tend to judge new technologies on their functionality and inherent values. Immigrants feel more compelled to learn how to use it.

Jakobsson: They (Natives) look at an object or a phone as a tool and if they don’t like it, they won’t use it. Whereas the digital immigrants feel they need to learn this, they need to use it. (Immigrants) are much more part of the design process for us, through defining the limitations.

The Dualism Of Public & Private

Another distinct difference between Natives and Immigrants are how they define their private and public worlds. Immigrants are apt to build more rigid barriers between these spheres that make up their lives. For natives, technology has been a part of their environment that is largely taken a face value without consideration of broader implications. As PEW Internet researcher Lee Rainie said, for Natives, “technology is like oxygen, it’s just there.”

This can often lead to doing things that “seemed fun at the time” but which can turn into widely accessible and long lasting personal liabilities. Case in point: the infamous and ill thought out Facebook post:

Jakobsson: Some (Natives) are on their Facebook (profile) and they’re saying, “Oh, I had a great party with these guys last weekend,” and they don’t understand that their parents are actually listening, because they’re on the same network.

(Note to my teenage daughters: Your mother and I would never do that.)

The Translation Of Technology

The analogy often used to explain the neuroplastic process that creates the differences between Natives and Immigrants is the acquisition of language. If you learn a language as a child, it becomes your native tongue. If you learn it as an adult, you’ll probably speak it with an accent and it will always be your second (or third) language.

Research has found that the reason we learn language easier as a child is that the neurons best suited to the task are available and the required circuits are rapidly strengthened and embedded. This combination of optimal neural recruitment and adaption, fueled by the massive amount of restructuring our brains go through as children, ensure that the first language learned claims the prime neural real estate.

The second language learned has two big obstacles ahead of it. The primary language is already present in the brain’s language centers. Recent studies seem to indicate that second languages may actually live in a different part of the brain. Secondly, our mature brains are simply not as “plastic” as those of a child. The rapid selection of strengthening of neural circuits that makes learning your native tongue relatively easy is not present to the same extent in adults.

The dominance of the primary language means that it will become our baseline. The acquisition of a second language begins with a translation process – laboriously translating words back to our native tongue so we can grasp the concept, then taking the response and translating it back to the second language. Only with a massive amount of exposure and practice in the new language will we become “fluent”, able to structure our concepts in this language without translation to our baseline.

The “learning” of technology follows the same path. For Natives, technology equals fluency. No translation is needed. That’s probably why Natives accept technology for what it is, while Immigrants feel we have to “conquer” new technologies.

At TAT, they have another term for this translation process. They call it “Filmed Theatre”:

Jakobsson: When TV first came around, when people wanted to create the first TV shows, they thought, what are we going to broadcast? What they did was filmed theatre because theatre was a medium which existed and which was understood. They filmed that and then broadcast it on TV.

“Filmed Theatre” shows how we tend to try to understand new technologies by using a baseline of a technology we’re already familiar with. We’re “translating” it so we can understand the concept. But this translation also limits our ability to function in the new paradigm. It not only forces a much more rigorous mental process, it also limits us because some concepts just don’t translate well. We don’t “get it”. Hampus gives the example of an online publisher who doesn’t “get” the intensively immediate nature of online:

Jakobsson: They’re (the publisher) talking about how they’re going to update their content every 24^th hour. Every night they’re going to push that content to all their users and their consumers. We have to tell them repeatedly that you can’t push things out on a every 24 hour basis. You have to push it out when the news comes in. I mean, you can edit it, of course, and write it, but you can’t do the layout at night and publish it.  That’s the old way of doing media and that’s filming the theatre.

The Dualism Of Tools & Toys

There’s another distinction between Natives and Immigrants that is becoming more prevalent in education and the workplace. In fact, it was this distinction, identified in the classroom, that first led educator Marc Prensky to coin the terms Digital Natives and Digital Immigrants in 2001.

We immigrants tend to think of things in terms of being “Tools” and “Toys”, with little overlap between the two. We use “tools” at work, and we play with “toys” in our spare time.

In the world of the Digital Native, however, the divide between the two is not so distinct. “Tools” and “Toys” are merging in more and more online instances.

As a Digital Immigrant, I maintain two social profiles: LinkedIn and Facebook. Although I have both business contacts and personal friends and family in Facebook, I have almost no personal contacts in LinkedIn. For me, Facebook is much more of a “Toy” and LinkedIn is definitely a “Tool”.

Natives seem not only to be more comfortable with this merging, they expect it. “Edutainment” is becoming the norm rather than the exception in formal education. Social and professional worlds collide on social networks. The precepts of gaming are creeping into more and more business applications. I was recently talking to a developer of a new business lead program and he’s seriously considering embedding the functionality in a game-like interface.

But Who Is Right?

As we talk about Digital Natives and Digital Immigrants from our respective sides of the divide, there is a natural tendency to try to judge the merits of each. Is being “fluent” in technology better?

Or do we Immigrants tend to take a more thoughtful approach to technology because we have to…well…think about it more? Personally, I think the whole debate is rather pointless. There is no right or wrong, no better or poorer. It is what it is.

But as someone who has to create experiences for both Natives and Immigrants, Jakkobson had an interesting perspective:

Jakobsson: People think the digital natives create all of the value, but I think that’s completely wrong. I think the digital natives do create a lot of the value, but it’s often the immigrants that “hook” it into actually how people work.

In my next column, I’ll continue my chat with Hampus Jakkobson by exploring the concept of ambient technology and how it may interface with very private technology.

*Editors’ Note: TAT was recently acquired by RIM (Research in Motion).

Soon, my conversation with Shashi steered into the area of how search might morph as more searches are done from platforms other than the typical desktop box environment. Obviously, mobile search is a hot topic, but I personally have been amazed at how my connected experience has changed on the roomier yet still tactile iPad. It’s a much different type of engagement —more leisurely and more serendipitous. I tend to browse and become less task oriented on my iPad. It’s the type of device you pick up to kill time or entertain yourself. And search in that environment can look significantly different than the typical task driven “in and out” nature of a search launched from a desktop keyboard. As our online time splinters across more connected devices, do we have to tailor experiences to match?

Seth: If you have the kind of interactivity that a smartphone like an Android phone or an iPhone offers in terms of the touch interface—the ability to move from app to app—and if you had a larger screen, would you be able to do significantly more? I think the iPad has definitely answered that question. Obviously, you can do a lot more in terms of experiences like reading a magazine, having apps that give you more meaningful space and therefore you can accomplish a lot more. Given the interactivity of these tablets, you can build an experience that is what you’d want the next generation of desktop experiences to look like. And that’s how we’re starting to treat the iPad, to say what if you took the approach of actually building apps on iPad, seeing how they work out, and then learning from that and then bringing that back to both your traditional smartphone experiences and desktop experiences? And it definitely opens up a range of possibilities that have never existed before.

Of course, if search experiences change, so does the opportunities advertisers have to present themselves in compelling and relevant ways. Shashi soon steered the conversation in that direction.

Seth: It brings up another point—actually—a couple of points. I think the display advertising experience on a search app on an iPad offers up the kind of advertising that even traditional internet display advertising has lacked. It’s the kind of advertising that you see in a magazine like GQ or Vogue, where ads and content start blending so well together that users don’t even think of them as ads, they actually think of it as content. And that’s why I think that the next generation of advertising on mobile devices is completely going to blend display and search advertising, meaning the targeting can be done in one way or the other, for an audience or for keywords—but the kind of formats that you see on these devices are going to be significantly different and will probably bring in new kinds of advertisers that have shied away from this platform traditionally. If you think about a brand like Hugo Boss or Louis Vuitton, they traditionally have never advertised on the internet because they’ve never gotten the kind of canvas that is needed. But if you bring this new kind of canvas that we’re talking about, which could work especially well in an iPad or even an iPhone/Android kind of device, I think the world is your canvas at that point.

One of the biggest challenges with search marketing is that it has struggled to move beyond it’s direct response roots. While people use search to research all types of potential purchases, it’s easiest to measure search’s performance when it comes directly before a transaction or “hard” conversion event. Because of this, Google and the other engines have had difficulties attracting the branding dollars from sexier media such as magazines and television. But a different browsing experience could lead to a different type of search ad, as Seth points out.

I don’t believe, however, that we should be in a rush to emulate a TV or magazine ad on a search platform. One of the beautiful things about a search ad is that it’s (or should be) highly relevant. And that makes it more useful. TV and magazine ads are rarely relevant, no matter how high the production values. And so, perhaps, a richer search experience could combine the best of both worlds: high emotional engagement and on-target relevancy, leading to ads that actually add value through their usefulness.

But the ability to target ads to the right audience based on interpretation of their intent and an assumption of what it is they want to do is not an easy thing to master. It requires search marketers to embrace data in a new way, using it to delve into the psyches and motivations of their marketplace. This isn’t simply about slicing faceless numbers to gain maximum share. It’s about nailing intent with precision. And, with a proliferation of devices, that intent could vary significantly from a desktop to a tablet to a smartphone. Shashi provides his view:

Seth: I think it requires that marketers become much more data-centric in their analyses and their targeting, and that the kind of data that publishers like us have to bring to the table for marketers is going to be significantly more sophisticated than has been done in the past. For example, knowing and providing details about users, their habits, what kind of content are they likely to interact with, what their needs might be, and then changing that advertising and marketing message to fit that individual user is going to become increasingly important. And it’s even more important to do it in the right way on the right device. For example, connected devices like the iPad or a connected TV. I think the kind of ads that you bring to the table have to be very different in each of those scenarios and have to tie in the user’s context, the intent, the device, the location, and so on, which makes for a pretty complex landscape. Much like we’ve talked about search changing, I think advertising is going to change a significant amount as well.

With change comes opportunity. Will Google continue to dominant, or does this change in habits allow for an accompanying change in the order of the search world as well? As I mentioned to Shashi, this could be a brand new horse race. And so I asked him to handicap the contenders, starting with Microsoft…

Seth: Lots of great assets for sure. Microsoft is doing the heavy lifting on the back end side for search for us, and therefore clearly are in the game and in it for the long haul. I see them not so well positioned in other spaces like mobile for example, but nevertheless definitely have the technical capability to pull off.

Then came Shashi’s own mount—Yahoo…

Seth: Somebody like Yahoo! has a lot of great assets. Our Asian assets, which is where I think the bulk of new users and new use cases are going to come about—at least I’m banking on that—but also the fact that we have a pretty robust ad platform and we’re probably the first one to have started to blend search and display advertising. A significant amount of work has gone on in that area and we’re betting big on it. And definitely with the kind of media assets that we can bring to the table like sports, finance, news, etc., I think the blending of that, of local and listings and search, all that starts to make a lot of sense in this new mobile world that we’ve talked about.

I think that is the key opportunity and we as a company are definitely pushing hard on that angle, to say, “Hey, who better than us to own the content and the communications and the search pieces all under one umbrella and bring it to this new world which is going to be very mobile-centric?” And the landscape changes significantly with that.

Can Google, the 800 pound gorilla, be fleet of foot enough to compete in this race?

Seth: It changes the game significantly and, as you probably have heard in the interviews before, I think they have a vested interest to continue down the path that they have invested in, and that path is much more around one search box, one user, and one form of advertising. Right? I think it’s a lot easier for somebody like Yahoo! to shift, somebody like Apple to shift the game a lot, but I think somebody like Google, there’s a lot invested and a lot more to lose. So we’ll have to see how they react to the new world.

And finally, the dark horse—Apple…

Seth: Apple is in a very interesting place because they’re truly a platform play. They clearly have no intent of getting into the content or the search business themselves, but they bring a lot of capabilities to bear that people like Yahoo! and Microsoft and Google need, and clearly with the separation between Google and Apple that is going to be a different kind of war. Somebody like Yahoo! could definitely leverage some of these assets very well and play really well in that space.

And so, this marks my last column for 2010. Best wishes for a great holiday and we’ll see you again in 2011 with more on where search might be headed in the near future!