« SearchCap: The Day In Search, April 25, 2007 | Main | Industry Group Proposes Eight Principles To Ensure Click Quality »
Apr. 25, 2007 at 5:07pm Eastern by Chris Sherman
Stop Presses! Google Bested In Search Shoot-Out!
PC World has released the results of its most recent search engine shoot-out, and surprise, surprise, found that "Google is indeed the best search engine."
However...
This wasn't just a casual test, and Google wasn't the best across the board with all types of queries and content. The author tested 55 services in six categories: general (text info), video, mobile/local, news, images, and blogs. The goal was to simulate how "real people" searched, so no special commands or modifiers were used. In the text category, the real surprise was in the two search engines that bested Google:
AlltheWeb and AltaVista, two Yahoo-owned services that are powered by the same Yahoo index. Here's a chart summarizing the results of the tests:
You can also see the full chart with comments online.
As these types of things go, this shoot-out was fairly well done, avoided some of the incorrect assumptions and conclusions that others have fallen into, and generally offers a pretty good snapshot of the state of search today. Of course, all testing of this type is highly subjective and can't be used to draw absolute conclusions about the quality (or lack thereof) of any search engine tested.
Years ago, when Danny and I conducted a similar test we called "the perfect page test" several of the search engines protested that our conclusions weren't fair. AltaVista's then chief scientist Jan Pedersen wrote:
"Correctly assessing the relevance of an Internet search engine is methodologically tricky. All too often people resort to anecdotal tests based on a small number of favored queries --- for example, someone's name. This can be misleading, since no one query is a good predictor of overall relevance and small samples produce unstable and unreliable results. In contrast, truly informative tests, such as those sanctioned by the Information Retrieval community, and such as the one we run internally at AltaVista, average over a large number of diverse queries, typically several hundreds, to produce statistically significant results. We use a random sample of queries to avoid the judgment bias that can creep into a test using a 'representative' query set."
PC World's test was in fact the type of anecdotal test Pedersen describes, and so some of the biases he mentioned do surface in the test results. But the article does serve as a good reminder that there are other services out there besides Google that are in fact as good or better when it comes to searching various types of content. It's a good read and worth the time to read. Read on in PC World's Search Engine Shoot-Out.
|
Like The Story? Vote For It On Yahoo Buzz!
Send me the monthly search newsletter too! (Learn more about our newsletters and feeds) |
Subscribe To Our Search Feed!         
|
| Share & Bookmark This Story! |
By Chris Sherman
Permalink
Jump To Comments
See Related Stories In: Stats: Relevancy
Reader Comments
Wow, am I missing something? The PC World article barely describes the methodology used, though I think (can this really be true?) that they used 10 queries? They used 10 queries to evaluate engines that handle millions of real-world queries. Please tell me I'm misreading something...
The article states: "Google is indeed the best search engine, even though two other services topped it--barely--in our text-search tests." Ahm, OK...That's confusing.
And the notes in the chart are mostly about UI.
I know that every relevancy test has its naysayers and normally I would have let this one pass like others I've seen, but I couldn't because the chart takes it beyond anecdotal.
![[TypeKey Profile Page]](http://searchengineland.com/nav-commenters.gif)
The mainstream press may like to talk more about new Google products or features, but the search quality team at Google is quite focused and works very hard. It's kinda funny; I went to mail the article to a few other Googlers to say "Cool, now how can we improve search quality even more for these types of queries" and another Googler had already beaten me to it. :)