Stop Presses! Google Bested In Search Shoot-Out!

PC World has released the results of its most recent search engine shoot-out, and surprise, surprise, found that “Google is indeed the best search engine.”

However…

This wasn’t just a casual test, and Google wasn’t the best across the board with all types of queries and content. The author tested 55 services in six categories: general (text info), video, mobile/local, news, images, and blogs. The goal was to simulate how “real people” searched, so no special commands or modifiers were used. In the text category, the real surprise was in the two search engines that bested Google:

AlltheWeb and AltaVista, two Yahoo-owned services that are powered by the same Yahoo index. Here’s a chart summarizing the results of the tests:

You can also see the full chart with comments online.

As these types of things go, this shoot-out was fairly well done, avoided some of the incorrect assumptions and conclusions that others have fallen into, and generally offers a pretty good snapshot of the state of search today. Of course, all testing of this type is highly subjective and can’t be used to draw absolute conclusions about the quality (or lack thereof) of any search engine tested.

Years ago, when Danny and I conducted a similar test we called “the perfect page test” several of the search engines protested that our conclusions weren’t fair. AltaVista’s then chief scientist Jan Pedersen wrote:

“Correctly assessing the relevance of an Internet search engine is methodologically tricky. All too often people resort to anecdotal tests based on a small number of favored queries — for example, someone’s name. This can be misleading, since no one query is a good predictor of overall relevance and small samples produce unstable and unreliable results. In contrast, truly informative tests, such as those sanctioned by the Information Retrieval community, and such as the one we run internally at AltaVista, average over a large number of diverse queries, typically several hundreds, to produce statistically significant results. We use a random sample of queries to avoid the judgment bias that can creep into a test using a ‘representative’ query set.”

PC World’s test was in fact the type of anecdotal test Pedersen describes, and so some of the biases he mentioned do surface in the test results. But the article does serve as a good reminder that there are other services out there besides Google that are in fact as good or better when it comes to searching various types of content. It’s a good read and worth the time to read. Read on in PC World’s Search Engine Shoot-Out.