Will the “Deep Web” Slay Google?

There are arguably two parallel tracks in the “what will succeed Google?” meme. An emerging one is “will the feds intervene to block Google?” The second is much older and involves the hypothetical “semantic web” or sometimes “deep web.” Essentially this asks, “who will do search better than Google?” (A third line of argument might surround “social search.”) Two pieces in two days in the NY Times reflect the two former strands.

The first article is Everyone Loves Google, Until It’s Too Big and picks up on the monopoly discussion. I responded over the weekend on my personal blog Screenwerk. The second and subject of this post is Exploring a ‘Deep Web’ That Google Can’t Grasp.

From the “Deep Web” article:

Beyond those trillion pages [indexed by Google] lies an even vaster Web of hidden data: financial information, shopping catalogs, flight schedules, medical research and all kinds of other material stored in databases that remain largely invisible to search engines.

The challenges that the major search engines face in penetrating this so-called Deep Web go a long way toward explaining why they still can’t provide satisfying answers to questions like “What’s the best fare from New York to London next Thursday?” The answers are readily available — if only the search engines knew how to find them.

Yes, an engine that can mine all that data and present “answers” to users would be quite exciting. However, as the article points out, Google is also investing in trying to mine more of that so-called “deep web” itself.

There have been many extravagant claims by and about semantic search engines (e.g., Powerset, bought by Microsoft) and deep web projects in the past (Chris Sherman has much more perspective on this than I). But so far, none of them have really borne fruit.

A quote from the “Everyone Loves Google” article is also relevant here and I believe correct about what changes might make a difference at least in the near term:

“Whether we’re slightly ahead or slightly behind Google in core relevance is not a game changer in search,” said Prabhakar Raghavan, Yahoo’s chief search strategist.

Yahoo’s best opportunity, Mr. Raghavan said, is to offer radically new ways of presenting information that will help users finish whatever it is they started before the search, like finding a job or buying a plane ticket. “People don’t want to search; it’s a digression,” he said. “They want to complete a task.”

Search results pages right now are terribly cluttered (and flawed in my opinion). But doing something that truly delivers on the “complete a task” metaphor is challenging in myriad ways:

  • Technology/algorithm
  • Interface design
  • Political: picking winners and losers from among similar sites to a much greater degree than today

Mobile is also an interesting lab for PC-based search. The limitations of the mobile “form factor” and the greater need/immediacy regarding the information will likely hold some eventual lessons for online search.

Microsoft and Yahoo (assuming it doesn’t sell search to Redmond) will continue to make improvements in their algorithms, indexes and interfaces. The more competition the better because search will only become more important as the “deep web” is unlocked.

Related Topics: Channel: Social | Features: Analysis | Google: General | Google: Legal | Google: Other | Microsoft: Bing | Microsoft: Other Search | Search Engines: Powerset | Search Engines: Social Search Engines | Top News | Yahoo: Search | Yahoo: User Interface


About The Author: is a Contributing Editor at Search Engine Land. He writes a personal blog Screenwerk, about SoLoMo issues and connecting the dots between online and offline. He also posts at Internet2Go, which is focused on the mobile Internet. Follow him @gsterling.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://www.compucall-usa.com olivieramar

    Hi Greg,
    I think you’re really talking about the Holy Grail here. I agree that if anyone ever got it right (including Yahoo OR MSN) we would have a whole new ball game. That being said, if anyone can do it and allocate the proper resources to doing it, it’s going to be Google. MSN can’t commit itself to search (as we all heard over and over again at SMX) and Yahoo has other concerns at the moment it seems. Any small player will be gobbled up at the first sign or trouble by the others.
    I also think you make a great point about search clutter. Even though everyone seems to love and be comfortable with Google, the One Box has been a mess on the eyes. Google’s results are starting to feel (to me at least) like Asks.
    Take Care,

  • Jeff

    Or, will semantic search and the deep web slay PPC CTR?

  • bwelford

    I believe there is an even bigger cyberspace that might be called the NOW web. This includes all information packages or Instants that are online, most of which never have a URI. These might be chat messages or text messages for example. This implies the following hierarchy:
    NOW web >> Deep web >> crawlable web.
    I think Google should stick to the latter if it wishes to achieve its business goals.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide