This has been a remarkable year. After years of no real dramatic evolution in search, the third generation finally arrived. Google calls it Universal Search, and I’ve been tending to say "blended search" as a generic name for the change that’s now hit all the major search engines. But in doing the agenda for our upcoming SMX West conference, a better term for what’s going on finally clicked: Search 3.0. In this article, I’ll cover the why and what of Search 3.0, taking in Search 1.0 and 2.0 along the way and touch on how Search 4.0 — personal and social refinement — is on the way.
Search 1.0: Location, Frequency, & On-The-Page Ranking Criteria
Let’s go back to the good old days, around 1996, when some of the big names in web search were Lycos, Infoseek, Excite, WebCrawler, Open Text, Hotbot, and AltaVista. Yahoo was there, too — but it was the exception, the search engine that relied on humans to catalog the web, rather than the others that used "crawlers" to automatically build listings.
Those crawlers were robots that automatically followed links across the web. When they came to a new page, they would effectively make a copy of it. All the copies were stored in an "index," which was like a big book of all the pages they had collected. The crawlers indexed millions of pages, which sounds like nothing compared to the billions of pages that are harvested today. But back then, the web was a lot smaller!
When someone did a search, the actual namesake "search engine" component kicked in, using an "algorithm" to sort through all the pages in the index. The algorithm was a system designed to return what a programmer thought the best pages would be. Back then, the best pages were deemed to be those that used the search words a lot in proportion to other words on the page, plus whether those words appeared in key areas of a web page.
Consider a search for shoes. The algorithm might not like this page:
Walking Alone At Night
I hear behind me
Shoes softly scuffle
Friend or foe?
Keys, and I’m safe inside
Maybe the algorithm doesn’t like bad poetry written in a hurry. But see how the word "shoes" appears only once? How relevant is this page really, then, if shoes only gets a single mention. Now consider:
Buying Shoes Online
As the web grows, many are discovering
that shopping can indeed be done online
for all types of things, even shoes!
Yes, footwear fanatics. You can buy
shoes online. But what about trying
them on? Those selling shoes have
thought of that. In this review, we
look at online shoe sellers with great
In the second example, shoes is mentioned several times. Plus, it appears as the headline of the story, and in the opening paragraph. Let’s say it also appears in the HTML title tag. That means shoes not only appears with frequency, but it also appears in key locations on the page. Algorithms back then liked that.
So first generation search? Search 1.0 was largely about looking at the location and frequency of words on individual web pages to measure them against each other.
Search 2.0: Looking At Links & Other Off-The-Page Criteria
It didn’t take long for savvy search marketers to figure out that if the algorithm liked certain things, you should tailor content to suit. Thus, SEO was born — search engine optimization, optimizing content for search engines. Some SEOs focused on making relatively minor changes that still could have dramatic impacts. Other figured if location/frequency was what the search engines wanted, the more the better:
Shoes, Beautiful Shoes!
Shoes are the best shoes
anyone could hope to find
shoes online at our store
with shoes shoes shoes
blue shoes red shoes
walking shoes tennis shoes
Pretty ugly — but that’s OK — this text might be hidden from human view in a variety of ways. The main point is that location and frequency worked better in the trusted environments of libraries and controlled collections of documents that web search grew out. Applied to the open web, where anyone could get anything listed, it started to fall apart. Back in 1999, AltaVista founder Louis Monier told me:
The good answer to the query ‘car’ has nothing to do with the text. I’d rather use a medium with a crystal ball.
Instead of a crystal ball, two guys by the name of Larry Page and Sergey Brin had fired up their own search engine, Google, that relied on link analysis to rank pages. They weren’t the first to look at links, and PageRank was only part of the overall algorithm. But the pair popularized a shift for all the major search engines to make a second generation leap — depending on criteria not actually on web pages to rank which pages are best. Call it Search 2.0.
It’s not just links that can be part of off-the-page criteria. Clickthrough is a metric that has been and may still be used, where you determine if people are clicking on a particular listing more often than one in a particular position normally gets. If so, perhaps the listing should be ranking better. Domain age and general traffic levels to a site are other things that can be used. But it’s really the links that remain dominant.
Google in particular popularized the notion of links as votes, with the web being "democratic" in how those votes were cast. And for a time, it worked. But then we got active campaigns to win votes, Googlebombing that led to things like the miserable failure ranking for President George W. Bush. We also saw an economy of buying and selling votes spring up, which to this day sees Google trying to fight off paid links.
Search 2.0 still works, of course, despite these problems. But things could be better — cue Search 3.0 to the stage, please.
Intermission: What The Hell Is Vertical Search?
I’ve long hated the term "vertical" search, which was popularized by financial analysts more than search engines. I hate it because most people have no idea what it means. I know. I ask audiences all the time if they understand the term, and plenty of hands of those indicating "no" go up, followed by "ah ha" looks after I explain it.
But vertical search as a term is here to stay, continuing to be used and even having its own Wikipedia entry, so let me explain it.
Go to Google, Yahoo, Microsoft Live, Ask.com and do a default search. What you’ve done is a "horizontal" search. You’ve searched across the entire spectrum of the web for a topic. To illustrate, consider just some of the topics that pages are about:
… shopping video news sports weather research stocks lottery …
See, a nice long horizontal line. But if there’s some news event that happens, you don’t want to search the entire web for information. Doing that means you’re hunting through billions of pages to find the relatively few updated with the fresh and relevant information. Far better if you focus only on news content.
To help you do that, search engines build specialized search engines that only go to news sites. Rather than search across the horizontal spectrum of the web, they let you slice down "vertically" only into news — or into sports — or whatever. Search for "fires" in a news search engine after a major fire in a particular area, and pages about that fire will dominate the results, since you can apply a time-based ranking criteria. Search for "windows" in a home improvement search engine, and you shouldn’t get information about how to fix Windows Vista.
Vertical search isn’t new. However, it faces a huge challenge. Searchers simply don’t know vertical search engines exist. Put tabs, buttons, drop-downs, you name it — it’s been tried (see Why Search Sucks & You Won’t Fix It The Way You Think for some illustrations). Users have ignored these options.
Here’s another problem. Consider this:
That’s my "Google 2005" image that I made in 2001, for a keynote to a librarians’ group on search trends. My slide was to illustrate that Google couldn’t keep adding tabs for each vertical search engine it unleashed. More important, even if it did, no one clicked on the tabs. Instead, I explained that search engine would need to learn to push the right tabs for us, behind the scenes, something I eventually called "invisible tabs."
Search 3.0: Blending Vertical Search Results
Even in 2001, we had some of this mixing of vertical results happening, usually at the top of pages in what Google called OneBoxes and others gave different names. But finally this year, search’s third generation really happened. The supremacy of "horizontal" web search results being the default was hit by Google’s Universal Search rollout.
Our Google 2.0: Google Universal Search article from May explains more about the inner workings of Universal Search — and the headline might now seem odd — shouldn’t it be Google 3.0 if we’re talking Search 3.0? Well, Google 1.0 was a Search 2.0 search engine already! That’s my story, and I’m stickin’ to it.
The short story is that Universal Search replaces some of the web search results with listings that come from vertical sources, such as video, local, or news. The results are mixed in, blended as appropriate, as our Search Illustrated column, um, illustrated in July:
It’s not just Google, however. In June, Ask launched its Ask 3D interface that used the "Morph" algorithm to automatically decide which vertical search results to blend into the main page. Yes, web search listings still remain front and center, with vertical search blended around web search — but vertical results became far more prominent.
OK, I am stretching the last two a bit. In both cases, they’ve mainly enlarged the "shortcut" style material that shows up at the top of a search for some queries, such as illustrated below for Yahoo:
And here for Live:
Still, the vertical search listings are growing in how often they appear at Microsoft and Yahoo, as well as with the amount of space they take up on the search page. It’s enough for me to declare Search 3.0 now fully engaged by all the major search engines.
For search marketers, Search 3.0 represents new opportunities. There’s less content in various vertical search engines, so the odds of ranking well naturally increase more. In addition, much of the content in vertical search areas such as local and video seems poorly optimized. With just a little care, people should be able to see improvements — and now improvements that may bring them to the first page of the "regular" search results.
Want to learn more from a search marketing perspective? Aside from how we cover verticals (and thus Search 3.0) through our columns here and features, as I said above, we’re going to have an entire track on Search 3.0 at our SMX West show happening next February, specifically these sessions:
Search 3.0: The Blended Search Revolution
Search 3.0: Video, Images & Blended Results
Search 3.0: Local Search & Blended Results
- Search 3.0: Online Retail & Blended Results
Search 4.0 & Beyond
I’m certainly not the first to use the phrase "Search 3.0," and I know not everyone agree with how I suggest it be used now. Time will tell! But I hope this article explains why I find it useful to explain the latest generation that’s been unleashed, especially in a world where everything seems more easily understood by some if it’s X.0 something.
I’d note that back in October 2005, Robert Scoble wished for a "Search 3.0" world where search engines learned from users, and Read/Write Web wrote in July about "Search 2.0" companies that were using "third generation" social features. That got me to comment:
The third gen has commonly been considered the combination of personal data – either refining results because of your own past history or that of others. Lump it all into social search, and that’s your third gen. And since it’s third gen, please call it Search 3.0 if you must and don’t force it into a Web 2.0 world just to try and mesh some Web 2.0 companies into a Search 2.0 framework.
But the human / social element does have a role to play in search. It’s the other long expected evolution to hit, one that I’ve long written about being alongside vertical search as a third generational leap:
Vertical search is important because it’s one of the two major things I’ve long talked about as being how search will advance. First generation search analyzed words on a page to rank content. Second generation search tapped into link analysis. Third generation search to me is looking at both user input (what we visit; what we click on; personalized results) and making search go more vertical.
So where’s the humanity in Search 3.0? It’s not — I’m shoving it into Search 4.0, a fourth generation leap that Google’s already making and the other major search engines have yet to push into.
Search 4.0 is another track at the show — and the topic for another article next week (Now posted: see Search 4.0: Putting Humans Back In Search). Next time, more on personalized and social search. Until then, here’s some key reading for those who want to look ahead:
- Google Search
History Expands, Becomes Web History
- Mahalo Launches
With Human-Crafted Search Results
- The Promise & Reality Of Mixing The Social Graph With Search Engines