Five Visionaries Sum Up The Future Of Search

For the past several months in this column, I’ve been asking a number of people the same question: Where will search go from here?

For the next two columns, I wanted to sum up what came out of those conversations and find the common themes.

A Shift In Our Expectations Of Success

A number of people talked about a sea change in our expectations of what search should be. The basic paradigm of web search hasn’t changed very much in almost 20 years: one query box and an ordered list of results.

If you held a search result screenshot from 1996 alongside a result from today, you would wonder how anything in the rapidly shifting online world could change that little in a decade and a half.

Today, however, all that is changing. We are expecting much more from our search experiences, driven largely by how those experiences look in different circumstances: launched from a smartphone, launched through an app or launched with a very specific intent in mind.

Traditional “one size fits all” search experiences are hard pressed to keep up with our changing expectations:

Stefan Weitz, Director – Bing Search:

“This idea of a universal notion of relevancy worked really well in the earlier days of the web. We had a smaller web, we had a more static web and we had a web that really was a web in the true sense of that term.

That was the way PageRank was constructed and that algorithm was quite brilliant. I still think that it’s quite brilliant. But it does seem a bit chaotic that we’re using that same notion. If I’m looking for the best hospital to treat cancer, just think about how ridiculous that model actually is. We’re now relying on popularity and in links to determine that? That doesn’t make any sense.”

John Battelle, Founder – Federated Media and author of The Search:

“We had a very, very basic, well-understood use case for 10 years, which was Google or “like Google”—you put in a couple keywords and you get a response back. And that framework of searching and coming back with the best document to answer a query is morphing. People are asking far more complicated questions now and they’re demanding far more nuanced answers, simply because they know they’re out there.”

One idea that I floated in several of the interviews was a concept I call “master intent”.

Often, when we use a search engine for a specific task, it’s actually part of a much bigger and more complex task. Think about the big things we do in life: buy a house or car, change jobs, plan a trip or have a baby.

These life events (and they don’t all have to be this momentous) bring with them a myriad of  needs for specific information. But it’s up to us to keep the master intent – the big objective – in mind and parcel out the individual searches as required.

We plug in the pieces as we gather them, but the heavy lifting is all done by us. Web search acts as a pretty simply minded assistant in all this – going out and gathering relevant information based on the words we feed it.

But what if search knew our “master intent” and offered a much higher level of assistance, going out and gathering all the information, filtering it based on our requirements and guiding us through the entire process?

John Battelle:

“I used the example of trying to figure out which classic car to buy. In a case like that, I don’t even know what I don’t know, and to expect search to tell you what I don’t know is expecting more than search can deliver. That’s a longwinded way of saying I think we’re going to be disappointed with our results because we’re not there yet both culturally and technologically to deliver what we know as consumers is out there. But we’re going to be. I think that we’re in a transition period.”

Not all the interviewees agreed with my assertion that our current standard of search success – relevancy – may not be the most appropriate standard moving forward. I believe search has to become more useful.

Perhaps not surprisingly, the company that largely defined our current search experience, reliant on relevancy, was the one least apt to call it outmoded:

Johanna Wright, Product Management Director – Google Search:

“Whether we talk about “relevance” or “usefulness” is a very subtle difference. Isn’t more “relevant” information also, by definition, more useful? I would also note that relevance is by no means an easy or solved problem. The ordering and presentation of the top web results is a critical part of the search experience, and one of the key reasons users keep choosing to search on Google.”

The Limitations Of Language

There is a challenge in our current paradigm of search, built on the concept of relevancy. It depends heavily on the ability of a machine to understand human language.

And that is no small challenge:

Stefan Weitz:

“If I walk into a conference room of people and ask why are there no jaguars in this conference room everyone is going to look at me like I have two heads. They would say “why would there be a jaguar in this conference room?” Jaguars are mammals, they are carnivorous. They live outside. Or… Jaguars are cars. In any case, it makes no sense to have a jaguar in a conference room. However, if you go to Google, or Bing, or Yahoo and type in “jaguar” and “conference room” you’ll get back in excess of 5 million hits.

Where we need to get to, and where we’re working to get to, is doing better job of having the crawler and the parser really understand the language. When I say Crest White Strips, think of what today’s indexes are going to do. They’re going to find those words in a PageRank and return those results. The system has to know that Crest is a brand and white strips are a way of whitening teeth. Teeth whitening is done by a dentist. And dentists often don’t like using off-the-shelf products.

You have all the things that I know about Crest White Strips, just from casual human understanding standpoint. The engines today don’t know that. So much of the intent calculation we have to do to deliver a good set of results is bound up in this challenge of us imbuing the engines and the index and the parsers with a more human characteristic of understanding what they’re reading. That will get us to intent much faster than a lot of the mathematical tricks.”

But, perhaps, searches reliance on parsing language isn’t as critical as it was before. As we do more things online, we leave more signals to help a search engine determine our intent.

Much of the conversation revolved around this point: that we now gather more information about the current situation of the user: location, their current activity, their place in the social grid. All these things can augment a query, allowing for a more useful experience.

Realtime Search

There are two converging trends that are changing a fundamental aspect of our search experience: a better understanding of who we are and what we’re trying to accomplish and a web that is much more dynamic and timely.

Together, these two things demand that search becomes more temporally relevant, in addition to just being semantically relevant.

Search is moving from being “stateless” to being very much in the here and now:

Mark Cramer, CEO – Surf Canyon:

“For the past 40 years since the inception of search, the way it’s worked is this: user enters a query, that query is matched to an index of documents, tremendous activity is deployed to try to determine the relevancies of those different documents, and a search result set is produced. But that search result set is static. There’s an order to those results. They go from 1 to 10 and then 11 to 50 million, and that order does not change. It’s stateless.

What we have been doing for the past few years is essentially applying state to the search page in order to make the results dynamic. And if you consider “dynamic” to be something resembling a conversation in the sense that the search result page is actually responding to every input from the user to alter the content on the fly, then I think that’s an interesting way of looking at searching.”

For example, conversations in social media happen very much in real time. As we begin to apply what’s happening in the social graph as an additional ranking criterion to the information indexed in a search engine, interesting things begin to happen:

Stefan Weitz:

“What it does do, especially with Twitter or a lot of these real time services, is provide additional signals to an engine. If you think about Twitter, for example, and how fast things rise and fall, the traditional model of ranking simply doesn’t work. It’s not fast enough.

We can use all these signals of UGC as part of an algorithm that takes into account the user’s intent, the most logical response to that intent and that response is determined by a number of signals: what’s happening in real time, what does your social circle think of these things, who is an authority on this particular topic, and what does that person (or entity) either read or write? There are a number of different factors that we look at.”

The Beginning Of Search As An Application

One of the biggest factors driving a new search experience comes with an evolution in the foundation of the web: the nature of data itself.

In the mid 90’s, almost all data was “unstructured”, a hodgepodge of stuff living outside an orderly database that demanded some type of categorization and organization. Traditional search, even with its limitations of language, excelled at this. It could crawl, index and whip unstructured data into shape.

But today, we have more and more “structured” data online. Typically, the commercial value of the data determines the rate at which it will become structured – a kind of economic colonization of the web. And as data becomes structured, it’s potential usefulness extends beyond the limited capabilities of search as we currently  know it.

Our current paradigm of search is that of a destination we go to in search of information. But increasingly, search is becoming a sort of “meta-app” that allows us to do things.

John Battelle:

“We will go do an insanely sophisticated search on Expedia in a matter of three minutes where we determine which flight to take. That’s all structured data and it’s basically an application, right? Expedia is a search application, it’s a decision support application. We do all that work and then we’ve trained ourselves to think about things that way. Then we’ll go to Google, we’ll put in something that has nothing to do with travel but maybe has to do with buying a refrigerator or a car, and we expect search to do that for us. And it’s not that we are consciously expecting it. It’s just that if you put in “1967 Mustang,” you’re hoping the right answer comes back when in fact you need to do a 2 to 3 minutes structured search using an application, not using sort of a vanilla generic search engine.”

The power of this new paradigm, that of search as an application, truly becomes powerful when we start to transfer functionality across different applications:


“Look at the valuable information that you can extract from how any one of us interacts with a well-designed application, and then create a dataset for that. Say I use the New York Transit application to navigate my way through New York for 3 or 4 days… all of the questions and back-and-forth that I use that app for, which is essentially a structured search session—right?

Now, match that against a set of data which is the transit map. I say, “I need to go over here. I want to go over there. I prefer this route over that route,”—that becomes a dataset that should inform other searches that I’m making on things that seemingly are unrelated but may not be. That should be available as metadata for future searches. And figuring how to inform that is as important as parsing the line or the spoken phrase that I’m making in the moment.

Now, if I take that spoken phrase and go and search for “Chicago rental car” four months after interacting with that New York Transit map application, how can we take the metadata from that interaction with New York and inform the appropriate response in Chicago. Perhaps the best suggestions would be, “Hey, you know what? You don’t need to rent a car. You can use the Chicago Transit. Here’s an app for it.

You can get from the airport to everywhere you want to go without having to rent a car. Plus, you’ll save $150 which we know is a goal of yours because you’ve been interacting with the Mint application and it said that a goal of yours is that you want to save $200 a month and here’s a way that you do that”?

Tying all that together, that’s the Holy Grail because then it starts to understand you.”

The Holy Grail indeed. Usefulness starts to expand exponentially as we tie more and more information into our objective. Even our choice of app given our intent starts to provide more signals to help clarify what it is we want to do:

Shashi Seth, Senior Vice President – Yahoo Search Products:

“I think people have narrowed down that master definition or master intent up front by choosing the app that they choose to engage with.

So, for example, if you are looking for local results, you may use a very appropriate local app in order to do those local searches, and therefore there is no guessing to be done anymore. By nature of what you just did, we already know that you are looking for local results and that is the only index that I have. When somebody uses a mobile app and we know what the GPS coordinates are and the fact that they’re using a local app.”

Search & Privacy

So, the future of search may lie in this concept of “usefulness.”

But usefulness depends on a search engine’s (and we may even debate the appropriateness of that label in the future – is a “search” engine truly what we need?) ability to know who we are, what we’re doing and what we hope to accomplish. And that level of transparency comes at a price. What happens to our privacy in this new paradigm?

John Battelle:

“I think personal feeds and the consumers’ ability to say, “Sure, you can have my feeds because I’m going to see value from it and I know that we’re in a trusted relationship”… I think that that handshake is going to be increasingly made in our culture.

I think, however, that we need to have a conversation about that handshake and understand it. We’re in the midst of that and it’s going to get more and more interesting over the next decade. I think that the handshake between large services now and what will become a flood of new streams of valuable data from apps, from interactions on other sites and services will allow a Google or a Microsoft to touch and have access to a ton of data about us.

But the bond of trust and the cultural contract that we have with those services is going to have to be very well understood. I think we’re sort of slouching our way there, but we’re increasingly having a conversation about that cultural contract and social contract.”

Editor’s Note: Batelle recently also had some interesting commentary about whether or not Apple will get into search, and how he saw Apple’s role in the new search paradigm:

In the next column, I’ll continue the summary by looking at how our search activity is increasing coming from a platform that’s not a desktop, what may lay ahead for search advertising and how all this might shift, if not shatter, the balance of power that currently exists in search. And we’ll wrap up with some truly mind blowing projections of what our relationship with technology may become.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Content | Search & Usability


About The Author: is CEO of Enquiro, a search marketing firm that produces search engine user eye tracking studies and other research.

Connect with the author via: Email


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • wfzimmerman

    Two words: work product.

    I am tired of having search as an intermediate step between me and my goal. I want the computer to accomplish my goal for me.

  • JPsmith

    What about the Jaguar conference room? There’s one in a building down the road.

    Very Friday comment perhaps, but true story.

  • c

    What if I want my search to bring up items not based upon my own narrow assumptions; results that may surprise, delight or even educate me about other perspectives? Will there be the equivalent of an “antonym” search result list?

  • davidhrosen

    I wonder if certain industries would benefit from the “Master Intent” approach before the “I’m having a baby” types were tackled. The RFP process in the business world works a lot like the “parcels,” when what a company is really looking for is a good vendor. (Not that the business world’s problems would be looked at first, snarf, snarf.)

  • Adaptive Consultancy

    Thanks for the post. It’s very interesting to know where leading digital experts think search is headed. I especailly like the refernce to the economic colonization of the web, and the commercial value of data. I do often worry about the manipulation of search and quite agree with what Stefan Weitz had to say about unsuccessfully trying to find a hospital. Knowing how reliant people have become on search engines, there is a major responsibility there to ensure that the best information is ranking the highest and not just the most optimised.

  • James Sanders

    Hello Danny,

    The problem I see relates to state, whereas some things just do not change as quickly as others. Take for instance, information on a chicken. It’s shape and size, although random and unique, mostly does not change that quickly throughout time. It may be slightly larger or slightly smaller from day to day, but I the end, it’s still a chicken, and the same information relates to it from day to day. Over thousands or millions of year, evolution might change the chicken, but that’s not happening tomorrow.

    The danger is when they start to use state and apply it the same way to all information online. State implies timeline and trying to link some relevance to creation date. At that point, we’re stuck with an internet littering scenario again just like article marketing was. In order to keep ranking, we’ll have to continue writing the same or similar content in order to keep up with state. This is not a good thing in my opinion.

  • James Sanders

    Sorry, Hello Gord =) Dang, just commented on one of Danny’s posts and forgot to change the intro =)

  • http://webliquidgroup Paul Burani

    Gord, your notion of the “cultural contract,” I believe, might be getting us close to the true catalyst in the future of search discussion. We need to be asking ourselves, do we in 2011 have the algorithmic wherewithal to make meaningful cultural inferences, on a large enough scale to be marketable? And while we’re at it: can we really make a blanket statement that there isn’t someone out there who specifically needs a jaguar conference room?

    I think that with the launch of Google +1, Google is basically telling the world that they don’t have the mojo — or the desire — to map out this so-called cultural contract. At least not within the next few years. And so they’re relying on people to submit a binary vote of confidence for items of cultural relevance. You pivot that against the other profile data they’re collecting, and you’ve got something interesting.

  • pureau

    The paragraph in which John Batelle described the Holy Grail of Search sounds very much like “User agents (personal agents)” in Artificial Intelligence research. What is different though is these agents are software “robots” that runs on your own personal computer like other software you have installed on your PC and your personal information is protected within that computing environment.

    However, with the advent of cloud computing and with software no longer installed on your PC, I would argue that just because your applications (and data) are now hosted on the cloud, your personal software agent and your search datasets can still be subjected to the security mechanisms now used to protect your online privacy.

    Unless we are not given that choice, why should personal software agents be any different from emails, calendars and other applications hosted on cloud infrastructure?

  • kameir

    Google claims that its mission is to organize the world‘s information and make it universally accessible and useful. At first look this seems to be a valuable goal. However, it is important to keep in mind that this mission is subordinant to the primary goal of any company which is to increase the wealth of its owners (shareholders) by paying dividends and/or causing the stock price to increase. Plagued by this paradigm Google adapted a business model (AdWords; AdSense) that in its consequence led to the development of an entirely new industry: Search Engine Optimization (SEO) which in its various forms – such as content farms – has caused an explosion of digital content that is essentially marketing copy disguised as information. In short: Google consequently organizes mostly the world’s ad copy rather than the world’s information. It is hence an utterly failed attempt to effectively organize information. Unless the “search giant’ is willing to radically change its business model there seems no way out of the corner the company painted itself into.
    I am working on an concept that addresses the inherent conflict while attempting to put the challenge of information organization into a wider scope than simply ‘search’ and hope to push out a minimum viable product out soon.


Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide