Will the “Deep Web” Slay Google?

There are arguably two parallel tracks in the “what will succeed Google?” meme. An emerging one is “will the feds intervene to block Google?” The second is much older and involves the hypothetical “semantic web” or sometimes “deep web.” Essentially this asks, “who will do search better than Google?” (A third line of argument might surround “social search.”) Two pieces in two days in the NY Times reflect the two former strands.

The first article is Everyone Loves Google, Until It’s Too Big and picks up on the monopoly discussion. I responded over the weekend on my personal blog Screenwerk. The second and subject of this post is Exploring a ‘Deep Web’ That Google Can’t Grasp.

From the “Deep Web” article:

Beyond those trillion pages [indexed by Google] lies an even vaster Web of hidden data: financial information, shopping catalogs, flight schedules, medical research and all kinds of other material stored in databases that remain largely invisible to search engines.

The challenges that the major search engines face in penetrating this so-called Deep Web go a long way toward explaining why they still can’t provide satisfying answers to questions like “What’s the best fare from New York to London next Thursday?” The answers are readily available — if only the search engines knew how to find them.

Yes, an engine that can mine all that data and present “answers” to users would be quite exciting. However, as the article points out, Google is also investing in trying to mine more of that so-called “deep web” itself.

There have been many extravagant claims by and about semantic search engines (e.g., Powerset, bought by Microsoft) and deep web projects in the past (Chris Sherman has much more perspective on this than I). But so far, none of them have really borne fruit.

A quote from the “Everyone Loves Google” article is also relevant here and I believe correct about what changes might make a difference at least in the near term:

“Whether we’re slightly ahead or slightly behind Google in core relevance is not a game changer in search,” said Prabhakar Raghavan, Yahoo’s chief search strategist.

Yahoo’s best opportunity, Mr. Raghavan said, is to offer radically new ways of presenting information that will help users finish whatever it is they started before the search, like finding a job or buying a plane ticket. “People don’t want to search; it’s a digression,” he said. “They want to complete a task.”

Search results pages right now are terribly cluttered (and flawed in my opinion). But doing something that truly delivers on the “complete a task” metaphor is challenging in myriad ways:

• Technology/algorithm
• Interface design
• Political: picking winners and losers from among similar sites to a much greater degree than today

Mobile is also an interesting lab for PC-based search. The limitations of the mobile “form factor” and the greater need/immediacy regarding the information will likely hold some eventual lessons for online search.

Microsoft and Yahoo (assuming it doesn’t sell search to Redmond) will continue to make improvements in their algorithms, indexes and interfaces. The more competition the better because search will only become more important as the “deep web” is unlocked.