A year ago, I wrote a quick post called, Google vs. Bing, The Fallacy of the Superior Search Engine, in which I selected twenty search-difficult queries and ran a subjective head to head evaluation of the search results from Google and Bing.

The end result confirmed what I had long anecdotally experienced – the difference in result relevancy between the engines was really not that much. In fact, Bing bested Google slightly.

Admittedly, my little test, with n=20, had the academic rigor of a Sarah Palin geography lesson. I was rightly skewered by some readers, including one who called the article a “shitpost”.  But the concept of quality parity between the search giants was so unexpected that some other media outlets picked up the post.

Among all the comments, this one, by Cathy Reisenwitz answered the big question, “if there is such little disparity in quality, why  is there such huge disparity in market share?”

“People aren’t switching to Bing because Bing needs to be much better than Google to make it worth the switch.”

Time to revisit the question – is there really a big quality difference between Google and Bing?  Over the past 12 months, many things in search have changed.

We’ve seen an unprecedented reversal in transparency from the engines, the explosion of social as a factor, modest Bing marketshare gains, stronger anti-competitive allegations and a (an?) hiybbprqag search dragnet.

Google vs. Bing:  Round II

The objective was to test and the search engines’ ability to deliver quality results, admitting that quality is a very subjective term, but includes things like timeliness, 1 click access to info, volume of content and lack of spam.

The Rules

I expanded the data set from 20 to 100 – collecting results over the past 4 months or so on real search terms from yours truly – (thus the inclusion of “clean crayon off a lcd t.v. screen”). Again, I included terms that were obscure, or could be difficult for engines to handle.

For example, “attorney Tom Brady” meant I was looking for an attorney named Tom Brady, but a logic-driven computer might confuse that for interest in the hunky quarterback’s legal troubles.

I also wanted to capture any personalization so stayed logged in to all accounts (which should have given Google a slight edge.) Paid results were not included in the evaluation set.

Scoring

  • A number one result earned 5 points, top 3 earned 3 points and a page one showing earned 1.
  • Failure to show up on page one cost an engine 5 points.
  • 2 bonus points were awarded if the answer to the question was delivered on the SERP page.

An additional 2 bonus points were awarded if at least one of the top 3 results was from an authoritative site, as determined solely by yours truly . . . Wikipedia in, eHow out.

Sampling The Search Data

Interestingly, there were many queries that didn’t return any results. This was certainly due to user error (but is that really the user’s problem?). But in many cases, the content just simply (and surprisingly) didn’t exist on the Web.

For example, Salomon, sells a full line of ski boots in a highly competitive industry. The only explanation I could find for the stiffness index difference between the Salomon Impact 120 CS and the Salomon Impact 110 CS was by subtracting 110 from 120. Hmmmm – not much info to go on when dropping $500 on a pair of boots.

Likwise, the answer to “who was Kim Jong Un’s mother?” was surprisingly difficult to find.

I remain amazed that neither Google nor Bing has really figured out the people/address side of search – my default when looking up someone’s address is still a direct load of 411.com; despite the fact that site invariably tries to upsell me on some affiliate cyberstalking product.

This is particularly painful when updating a Christmas cards list. Perhaps this is the engines’ deliberate decision to mollify those concerned about privacy issues, but it is very frustrating.

I ran some searches that had an unwritten time element to them, like “wii new release rumors”. These searches are particularly difficult for engines to understand the users’ intent. In that case, I was wondering if I should delay a purchase, instead of researching past rumors about wii launches.

For the query, “distribution of type of search engine queries”, Google sent me to Wikipedia, but unfortunately the data on there cited studies from 2001. Likewise, “what percentage of people online use Twitter” – Google sent me to current data while the only results on page 1 from Bing went back to studies from 2010.

The most interesting result was for the query “perpetrators behind rick santorum googlebomb”. I’m not sure if it was the inclusion of “googlebomb” or “santorum”, but Bing initially gave me a no results page. (Although when I checked it again while writing this, they served up Danny’s post on Santorum.)

Another one that begs for a conspiracy theorist was Bing’s results for “google analytics import match type data from adwords”.

Page 1 was littered with anti-Google sentiment, including headers like: “Comparative Analysis:  Omniutre SiteCatalyst, Google Analytcs” and “STOP Google Analytics from Stealing your valuable Adwords Keyword data“, neither of which have anything to do with match types. Did we catch Bing favoring certain results? Or is this a natural occurence?

I could smell the impact of Panda in the results . . . . Bing seemed to heavily favor some weak UGC sites like eHow and a chacha, (which looks like the ugly lovechild of Entertainment Weekly and Quora.)

My favorite garbage content came on eHow – for “how do you change the water filter on a frigidaire professional series”.  “Step 1:  Open the freezer or refrigerator door . . . ”

Having said that, Bing’s overall score was buoyed by a slick incorporation of a travel widget for all airline related queries.

The Results

Same as last year – a statistical dead heat; meaning overwhelming parity between the engines.

Google:  296

Bing:  274

While the marketshare and tech sentiment suggests the score should have looked like the college football West Virginia beatdown of Clemson, it looked much more like Romney squeaking out a win over Rick, errrrrrr Santorum in Iowa. And like Romney, this isn’t a win for Google.  I suspect we will continue to see Bing make very slow inroads during 2012.

One final note – you can see the raw data by downloading the Excel spreadsheet from my test here.

Remember, I ran the searches on my computer at work over a period of four months and you most likely won’t get the same exact results if you recreate some of these searches on your own. In fact, as I reran some queries while writing this story, I got completely different results.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Search Marketing | Features | Features: Analysis | Features: General | In House Search Marketing

Sponsored


About The Author: is the founder of Atticus Marketing - a search agency dedicated exclusively to the legal profession. Prior to Atticus, Conrad ran marketing for Urbanspoon and the legal directory Avvo, which rose from concept to market leader under his watch.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • cerealbowl

    THIS is how you should compare search engines. Blindly, without some arbitrary scoring function, and with more than a single person’s arguably biased inputs.

    http://blindsearch.fejus.com/

  • http://www.contentping.com trinity_hartman

    What I really want to know is whether you were able to get that crayon off your TV screen!

  • Conrad Saam

    Trinity – the answer according to both Google and now confirmed by myself, is Magic Eraser by Mr. Clean and a very stern discussion with a 4 year old.

  • http://www.rimmkaufman.com George Michie

    Conrad, this is a fun post. Some may throw stones, but I think there is value in down and dirty analyses from time to time.

  • http://www.maxminzer.com maxminzer

    Bing sucks. Microsoft should stick to OS and software, shut down IE and Bing (ok, that was a bit of overreaction).

    Great interesting article!
    However, this is “just for fun” type of research and explores just one side. There are way more factors to be considered with way more data in this multi-sided equation.
    I’ll just say that Bing is just too easy of a search engine and (just like IE) is not catching up with reality. You can rank to page one of Bing with 1-page website with duplicate content while being nowhere to be found in Google. Too easy. Not gonna fly. Spam will kill it.

    I appreciate your reminder about where search is going – reading and trying to understand our complicated thought process and intentions. Thank you!

  • Antoine Brunel

    Hello Conrad,

    Very interesting article! But I have a coupe of questions though:
    1- May you share with us the 100 queries?
    2- How do you determine the relevancy of results?
    That’s a very tough question to answer, one can determine what he expects, but how do you know this is the results everybody is expecting?

    In SEO, I think we can learn so much by:
    1- Trying to understand queries, ie what people are expecting, that’s what search engine are trying to do, and it goes way beyond of a simple keyword research!
    2- Once we do understand what people expect, we can always improve website pages in order to fit customer’s expectations

    It seems to me, that most of the times, when referring to SEO, we miss what the customer expect, and this is simply wrong!
    The starting point of SEO is Marketing, Marketing, Marketing, ie trying to read people’s desire through their queries.

  • http://www.pagezero.com Andrew Goodman

    While we’re getting down and dirty, though, I do wonder about your eHow example. It looks to me like pretty good content. It provides all the steps for replacing a water filter on that type of fridge, but you twist it out of context by making fun of the first step of “opening the door”. I have no vested interest in saying so — I just hear a lot of criticism of sites like eHow and in the interests of accuracy it seems fair to defend that result.

 

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide