Google Fails To Trounce Bing (Again): The Fallacy Of The Superior Search Engine Revisited

A year ago, I wrote a quick post called, Google vs. Bing, The Fallacy of the Superior Search Engine, in which I selected twenty search-difficult queries and ran a subjective head to head evaluation of the search results from Google and Bing.

The end result confirmed what I had long anecdotally experienced – the difference in result relevancy between the engines was really not that much. In fact, Bing bested Google slightly.

Admittedly, my little test, with n=20, had the academic rigor of a Sarah Palin geography lesson. I was rightly skewered by some readers, including one who called the article a “shitpost”.  But the concept of quality parity between the search giants was so unexpected that some other media outlets picked up the post.

Among all the comments, this one, by Cathy Reisenwitz answered the big question, “if there is such little disparity in quality, why  is there such huge disparity in market share?”

“People aren’t switching to Bing because Bing needs to be much better than Google to make it worth the switch.”

Time to revisit the question – is there really a big quality difference between Google and Bing?  Over the past 12 months, many things in search have changed.

We’ve seen an unprecedented reversal in transparency from the engines, the explosion of social as a factor, modest Bing marketshare gains, stronger anti-competitive allegations and a (an?) hiybbprqag search dragnet.

Google vs. Bing:  Round II

The objective was to test and the search engines’ ability to deliver quality results, admitting that quality is a very subjective term, but includes things like timeliness, 1 click access to info, volume of content and lack of spam.

The Rules

I expanded the data set from 20 to 100 – collecting results over the past 4 months or so on real search terms from yours truly – (thus the inclusion of “clean crayon off a lcd t.v. screen”). Again, I included terms that were obscure, or could be difficult for engines to handle.

For example, “attorney Tom Brady” meant I was looking for an attorney named Tom Brady, but a logic-driven computer might confuse that for interest in the hunky quarterback’s legal troubles.

I also wanted to capture any personalization so stayed logged in to all accounts (which should have given Google a slight edge.) Paid results were not included in the evaluation set.

Scoring

• A number one result earned 5 points, top 3 earned 3 points and a page one showing earned 1.
• Failure to show up on page one cost an engine 5 points.
• 2 bonus points were awarded if the answer to the question was delivered on the SERP page.

An additional 2 bonus points were awarded if at least one of the top 3 results was from an authoritative site, as determined solely by yours truly . . . Wikipedia in, eHow out.

Sampling The Search Data

Interestingly, there were many queries that didn’t return any results. This was certainly due to user error (but is that really the user’s problem?). But in many cases, the content just simply (and surprisingly) didn’t exist on the Web.

For example, Salomon, sells a full line of ski boots in a highly competitive industry. The only explanation I could find for the stiffness index difference between the Salomon Impact 120 CS and the Salomon Impact 110 CS was by subtracting 110 from 120. Hmmmm – not much info to go on when dropping $500 on a pair of boots.

Likwise, the answer to “who was Kim Jong Un’s mother?” was surprisingly difficult to find.

I remain amazed that neither Google nor Bing has really figured out the people/address side of search – my default when looking up someone’s address is still a direct load of 411.com; despite the fact that site invariably tries to upsell me on some affiliate cyberstalking product.

This is particularly painful when updating a Christmas cards list. Perhaps this is the engines’ deliberate decision to mollify those concerned about privacy issues, but it is very frustrating.

I ran some searches that had an unwritten time element to them, like “wii new release rumors”. These searches are particularly difficult for engines to understand the users’ intent. In that case, I was wondering if I should delay a purchase, instead of researching past rumors about wii launches.

For the query, “distribution of type of search engine queries”, Google sent me to Wikipedia, but unfortunately the data on there cited studies from 2001. Likewise, “what percentage of people online use Twitter” – Google sent me to current data while the only results on page 1 from Bing went back to studies from 2010.

The most interesting result was for the query “perpetrators behind rick santorum googlebomb”. I’m not sure if it was the inclusion of “googlebomb” or “santorum”, but Bing initially gave me a no results page. (Although when I checked it again while writing this, they served up Danny’s post on Santorum.)

Another one that begs for a conspiracy theorist was Bing’s results for “google analytics import match type data from adwords”.

Page 1 was littered with anti-Google sentiment, including headers like: “Comparative Analysis:  Omniutre SiteCatalyst, Google Analytcs” and “STOP Google Analytics from Stealing your valuable Adwords Keyword data“, neither of which have anything to do with match types. Did we catch Bing favoring certain results? Or is this a natural occurence?

I could smell the impact of Panda in the results . . . . Bing seemed to heavily favor some weak UGC sites like eHow and a chacha, (which looks like the ugly lovechild of Entertainment Weekly and Quora.)

My favorite garbage content came on eHow – for “how do you change the water filter on a frigidaire professional series”.  “Step 1:  Open the freezer or refrigerator door . . . ”

Having said that, Bing’s overall score was buoyed by a slick incorporation of a travel widget for all airline related queries.

The Results

Same as last year – a statistical dead heat; meaning overwhelming parity between the engines.

Google:  296

Bing:  274

While the marketshare and tech sentiment suggests the score should have looked like the college football West Virginia beatdown of Clemson, it looked much more like Romney squeaking out a win over Rick, errrrrrr Santorum in Iowa. And like Romney, this isn’t a win for Google.  I suspect we will continue to see Bing make very slow inroads during 2012.

One final note – you can see the raw data by downloading the Excel spreadsheet from my test here.

Remember, I ran the searches on my computer at work over a period of four months and you most likely won’t get the same exact results if you recreate some of these searches on your own. In fact, as I reran some queries while writing this story, I got completely different results.