I can still remember when my when I first switched over to Google on the recommendation of my brother’s girlfriend. She’s literally a rocket scientist and carries a lot of intellectual weight with me; her endorsement was fairly simple – “it returns great results”.

To understand what “great results” means, transport yourself back 10+ years ago and try to remember the search user experience – one dominated by short tailed queries and multiple paginated results until you could find what you were really looking for. We didn’t complain because it was free and much better than that old dewey decimal approach we learned in middle school. In fact, it was magical.

Enter Google and the experience and expectations changed significantly. On the strength of superior results, Google’s market share skyrocketed. Over the past decade we’ve grown accustomed to results so accurate that we rarely peek beyond the top three. Yet, I have trouble believing that Google’s dominance continues to be based on superior search results, especially given the financial and human resources thrown at improving search across multiple competitors.

Danny Sullivan questioned Google search relevancy last year:  How the “Focus on First” Helps Hide Google’s Relevancy Problems. And as far back as 2007, Chris Sherman wrote a review of PC World’s search engine shoot out called Stop the Presses! Google Bested in Search Shoot-out!

My personal search approach uses Google as the default while using other sites for specialty searches. On Bing, image search is far superior and Wikipedia for 101 style information. Is this a factor of inertia or am I really getting a better experience with Google? I set up an objective small sample size evaluation of search quality between Google and Microsoft to see for myself.

Let’s Test The Theory

Methodology: I evaluated 20 different searches split evenly between Transactional and Informational queries and the search engines’ ability to deliver quality results, admitting that quality is a very subjective term, but includes things like timeliness, 1 click access to info, volume of content and lack of spam.

To provide a useful analysis, I’ve upped the degree of difficulty, deliberately testing long tail queries that are fairly specific and potentially confusing to a computer. I avoided easy navigational searches in favor of things that could trip up a computer’s ability to understand the user’s intent with queries like “Attorney Tom Brady”.

Results, including the snippet, needed to convince me of a likelihood of finding what I was looking for by clicking through in order to be considered.

To rank the results, 5 points were awarded for a good quality result ranking first, 3 for second and 1 for third. Two bonus points were added for top 3 results being on a highly authoritative site (as determined solely by yours truly). Five points were subtracted if the entire first page didn’t contain any good results. I only considered non-traditional results (like local, news, or Answers) if it appeared above the fold and I logged out of my gmail account when running queries.

Not too surprisingly, there was not a massive disparity in the results of my little test. In fact, Bing came out on top. Some queries performed very differently than others, for example, Bing was able to tell that my query for “Attorney Tom Brady”, was looking for an attorney and not the pictures of the hunky Patriots quarterback served up by Google.

Bing also did well with date nuances, unlike Google, sending me to the 2011 page for “SMX West Agenda” and a future calendar of concert appearances for “When is Trans Siberian orchestra playing next in Seattle” instead of past shows. Google won points by sending me directly to the IMDB page for filming locations for “What town was Beautiful Girls filmed in?” All in all, the results aren’t too surprising.

Of course, this begs the question why has Google been so successful? Are they still riding their brand laurels? Has Microsoft’s brandings and rebrandings of search hurt them among consumers?

Last week, a CNN article suggested that Google’s stock potential upside is probably no more than 20% – are we seeing the end of the growth in search? While these questions are the subject of intense debate in board rooms and VC water coolers, the conversation no longer centers on superior search results.

While I spent most of this post in retrospective navel contemplation about search, what does this mean tactically for the future of the in-house SEO? Despite the fact that we live in an industry of perpetual seismic change, Google has been a constant dominant player for most SEO’s entire career. Is Facebook the new SEO (as someone at work keeps insisting)?

Is Quora, the newest shiny toy whom the technology press dumped Twitter for this year, going to change everything? Is there a revolution in Local or location based search brewing? Or does consumer behavior change so much more slowly than technology (watch Quora trip on results for “Who is Lindsay Lohan” for example) – providing Google with a long lasting competitive advantage.

Your answers to these questions should shape your long term online marketing strategy.

Postscript: See also 89% Find Search Engines Do Good Job Finding Information, But “Noise” Is Issue

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Search Marketing | In House Search Marketing | Stats: Relevancy

Sponsored


About The Author: is the founder of Atticus Marketing - a search agency dedicated exclusively to the legal profession. Prior to Atticus, Conrad ran marketing for Urbanspoon and the legal directory Avvo, which rose from concept to market leader under his watch.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Dr.No886

    I just have two comments to make, first, just a small thing, the score is actually 53 to 62 as according to your scoring system, you did not deduct 5 points from Bing in the last query.

    Also, while it is much harder to test, to truly asses a search engine you would need to take into account the more advanced things that they do, such as using your history to influence results, for example it is likely that a Patriots fan who regularly searches for team updates and Patriots news would see a different search result than someone who has been recently searching for and visiting many different pages of lawyers in the search query attorney Tom Brady. This effect is of course a double edged sword and could negatively affect results depending on the situation, and is hard to test, but is a source of error nonetheless.

    Overall it’s not surprising that other engines have as good or better results than Google, but they will need significantly better results to compete.

  • Duane Forrester

    Excellent article Conrad! Certainly gets folks thinking – or, should get them thinking. Perpetual change is really the only constant we know. :)

  • Conrad Saam

    Dr. No – thanks for pointing out math error. I’ve updated.

    -Conrad

  • http://www.clientsforyourbiz.com Damon L. Rice

    Good article. I think the branding aspect of the popularity between the two is undeniable. People are sheep and tend to think and respond so accordingly when it comes to any kind of trending product. In a world of quality search vs. quantity, I think the interest will remain with whoever is most actively providing the newest, freshest resources to its users (Google). No matter who may actually have the better “specific” feature. People want the all-inevitable, consolidated, “all-in-one” convenience when it comes to most activities in our busy lives today.

  • http://www.michael-martinez.com/ Michael Martinez

    “Enter Google and the experience and expectations changed significantly. On the strength of superior results,”

    That’s an odd thing to say. Google’s search results weren’t THAT great back then. They grew their market share primarily by partnering with Yahoo!.

  • http://cathyreisenwitz.com/birmingham-seo-blog/ cathy reisenwitz

    People aren’t switching to Bing because Bing needs to be much better than Google to make it worth the switch.

  • http://mattcutts.com/blog/ Matt Cutts

    Hey Conrad, I picked out one query: [smx west agenda]. For me, Google had the 2010 agenda at #1 and the 2011 agenda at #2. When I did the query on Bing, here were the first 10 results:

    SMX West 2010 Agenda
    Search Marketing Expo – SMX West 2010 Agenda: Coming Soon! The agenda for SMX West 2010 will be posted the week of December 19. Want to be notified? Sign
    searchmarketingexpo.com/west/2010/agenda · Cached page

    Search Marketing Expo – SMX West
    SMX West news links… Attend 60 sessions on search engine and social media marketing. See the agenda here. SMX West is THE place to network – take a look at the schedule of events.
    https://searchmarketingexpo.com/west · Cached page

    Search Marketing Expo – SMX West Agenda Now Available: Multi …
    SMX West features 60 sessions on search engine optimization, paid search advertising, social media marketing and more; coming to San Jose, CA, March 8-10, 2011
    news.yahoo.com/s/prweb/20101222/bs_prweb/prweb4914864_1 · Cached page

    SMX West Agenda Now Available – Register Today for Super Early …
    ‘Tis the season of giving! Our gift to you? The Search Marketing Expo – SMX West agenda … and the opportunity to save $400 by registering while Super
    https://searchengineland.com/smx-west-agenda-now-available-register-today-for-super… · Cached page

    SMX West 2008 Agenda Up: 3 Days, More Than 50 Sessions!
    Check out the just-posted agenda for Search Marketing Expo – SMX West and see why SMX West is the must-attend search marketing event of 2008. SMX West will be
    searchengineland.com/smx-west-2008-agenda-up-3-days-more-than-50-sessions-12707 · Cached page

    Search Marketing Expo – SMX West 2010 Conference Agenda Now …
    Attend SMX West and you’ll learn from internet and search marketing pros how to get more customers to your web site … and make more sales. There are sessions for all skill levels …
    http://www.prweb.com/releases/smxwest2010/agenda/prweb3387314.htm · Cached page

    SMX West 2011 Agenda Goes Live With Super Early Bird Registration…
    Search Marketing Expo, SMX West, runs from March 8-10, 2011, in San Jose, CA, and on Wednesday they made their agenda live, and announced their ” Super Early Bird” rates.
    http://www.huliq.com/3257/smx-west-2011-agenda-goes-live-super-early-bird-registration-savings · Cached page

    Search Marketing Expo – SMX West 2010 Conference Agenda Now …
    Search Marketing Expo – SMX West 2010 Conference Agenda Now AvailableAttend SMX West and you’ll learn from internet and search marketing pros how to get more customers to your web …
    http://www.docstoc.com/docs/21158881/Search-Marketing-Expo—SMX-West-2010-Conference-Agenda... · Cached page

    Super Early Bird Rates Set to Expire for Search Marketing Expo …
    The agenda is available at http://searchmarketingexpo.com/west/agenda. SMX West is programmed by search marketing authorities Danny Sullivan and Chris Sherman, the editors of Search …
    news.yahoo.com/s/prweb/20110103/bs_prweb/prweb4932324_1 · Cached page

    Search Marketing Expo – SMX West Agenda Now Available: Multi …
    Search Advertising Expo – SMX West Agenda Now Obtainable: Multi-track Program Covers All Search and Social Media … SMX West features 60 sessions on search
    linkthe.com/2010/12/28/search-marketing-expo-–-smx-west-agenda-now-available-multi… · Cached page

    The 2011 agenda wasn’t even on the first page for me. For that query, you gave Google 0 points and Bing 5 points. To make this a valid comparison, did you use (for example) an incognito version of Chrome or InPrivate browsing on IE? I know that Bing has said that they use clicks in rankings, so if you’d clicked on results for these queries in the past, Bing could be showing you previously-clicked results higher, for example.

  • http://mcarter@receptional.com Miles Carter

    I agree totally with Cathy – this shows a higher quality of results, but not by enough to make people consider switching. It is foreseeable for the situation to be reversed within a few months too.

  • http://emerologio.com Andreas Kontokanis

    I don’t agree. Bing has also problem with global search. I am greek and my searches are 80% 20% english-greek. In my Greek searches I have seen so many times “We did not find any results for your search. I believe that happens too in Spanish that in US is so popular. So if they cannot find at all results in other languages they cannot beat google at any market. I still believe in very special search queries, it would be the same problem with bing. Maybe in 2-3 years your article would be a not bing biased article. I believe that I can find 20-30 search queries that the results show the opposite.

  • http://wiltjk.wordpress.com Jim Wilt

    Google vs. Bing – While I prefer Bing; Google has one feature I use: Filter results by date – Latest, Past 24 hours, Past week, etc.

  • http://www.glammedia.com Bernard Desarnauts

    In my mind this is more of a behavioral than technology question. For many consumers (my 11 year old boy as an example) Google has replaced the word “search” itself… Displacing Google will require a lot more than just better “search” technology. Only a new paradigm shift aka where information finds you type thing will do the trick…

  • http://About.me/RafaelSantos SantoTheWriter

    I don’t know, maybe I drank the kool-aide yet I refuse to use Bing. I do admit i find it strange to go to the Google homepage to do a search. Most of my web search is done on my Motorola Droid where my location is utmost importance to the results I want or on my iPod touch as I have it laying on nightstand. On the desktop I use Chrome and usually search right from the omnibox. I do find that there are some searches that a standard search engine will not return good results.

  • http://www.christian-fey.com Christian Fey

    Great article! I agree with Santo and refuse to use Bing for no real reason aside from Microsoft already running most of the computers out there. Apple gets media, google gets search, and microsoft gets windows! lol

  • Conrad Saam

    Matt –

    The irony in looking for the upcoming agenda for SMX was actually the impetus behind this article. Google is now showing up #2 for the 2011 agenda, but when I ran it about a month ago, it wasn’t (promise). As for Bing, I very rarely use it, so my past behavior wouldn’t include this query. To be fair, I ran the query a few more times (today), using combinations of hidemyass, different computers and a completely cleared browser; no matter what I do, I still get a good result from Bing. I sure hope there isn’t a (hide_if_location=mountain_view) for SEO conference queries component to the MSFT algo.

    -Conrad

  • http://www.MikeEvanchik.com Michael Evanchik

    somebody likes tom brady, LETS GO JETS

  • http://www.adventuresinsearch.com Elisabeth Osmeloski

    For @ConradSaam & @Matt_Cutts –

    Something to consider here may be freshness factor as well on more time sensitive queries. For the purpose of what you two are discussing, a little more info -

    When Conrad first searched “about a month” ago the agenda URL for SMX West was likely a ‘holding page’ – Now, of course that holding page URL was probably the same as for SMX 2010, so there is maybe some long term relevance behind it.

    But the actual details for the “2011 agenda” didn’t get posted until 12/21 – so promotion & viral link spread of the “2011″ qualifier didn’t happen at least until that day, or maybe even slowly until after the holidays, so relevance for that specific query could have been impacted by that as well.

    just wanted to toss that out there for you:)

    PS @Evanchick – Brady all the way!

  • usagemayvary

    this is the most garbage post I’ve ever seen.

    you’re basing google vs bing off of 20 questions?

    just a hint: they have billions and billions of possible avenues and you’re basing it off 20 questions.

    this is like basing “it’s snowing today aorund the world” by looking out your window.

    what a shitpost.

  • http://searchengineland.com/ Danny Sullivan

    Google, of course, counts clicks as well. Just as clicking on some things in Bing may influence results there, so might clicking on things in Google, since it personalizes all results regardless of whether you’re logged in or not. Testing the results set is always tricky.

    I also think it’s a mistake for anyone to take away from this that Google is worse than Bing, or that Bing has bested Google. It was an extremely limited set, which Conrad himself has acknowledged.

    I think the more important point he’s trying to make, certainly what I took away, is that Google is not always better. There’s no question about that. On some queries, Bing will be better than Google. On some queries, Google might be better than Bing. But in some quarters, it has been assumed that Google is simply always the “superior” search engine to any other. That has never been the case, and that’s the fallacy this article is highlighting.

    It would be nice if we did finally get a battery of relevancy testing done in a manner that both Bing and Google would approve of, so that we’d finally have some accepted metrics that we could refer to over time, to take the place of the terrible things we get stuck with instead: ego searches, limited tests and anecdotal accounts that can mean nothing. I wrote about this more here in a past article on Search Engine Land:

    http://searchengineland.com/blekko-launches-spam-clock-to-keep-pressure-on-google-60634

  • http://illogicalordering.com David Lawrence

    Just as a one off let’s have a look test, I put in your search about replacing water filters in Frigidaire products. I typed it word for word, double checked it was the same. On bing, while a frigidaire page came top, it had no info on actually replacing the filters and the main link was to their online shop to buy new filters. In google on the other hand, the top link was to a specific frigidaire professional series fridge which has a manuals section, the manuals containing the desired info. To me, Google returns significantly better results and I was absolutely underwhelmed when I tried Bing for a couple of weeks…

  • Conrad Saam

    Elisabeth – I checked the create date on my original document and it was 12/28 (life moves very quickly). Also – I used a selection of my real searches for this study, so shame on you for outing my Tom Brady man crush.

    Danny – the relevancy testing ideas is very interesting and I’d love to work with you to put something like that together. The intellectually curious point I’m trying to end this piece with is that, regardless of any parity in quality (and acknowledging an admittedly scientifically atrocious sample size of 20), Google still wins in the marketplace. (See today’s story on Google being #1 in all but 5 countries – http://searchengineland.com/google-nunber-one-czech-republic-5-countries-left-61174 for a shining example.)

    The pick up to this story has been surprising and as I read some of the follow up stories and comments I turn inward to contemplation about the future. What would it take to really change the landscape of an industry (with a dominant market leader). What should companies be doing to prepare or gamble on that fuzzy future? What might unseat the leader among the general populace? Stay tuned, its the theme of the next post and because it involves serious crystal ball gazing, will be even less scientific than this one.

  • karatekidkudi

    Bing or Google – Google or Bing???
    I prefer Google, and for me, right now is much better Google than Bing. But Bing is doing a great job and giving fight to google, we shoukd keep an eye for their next steps

    Here more info about this in spanish… Greetings from Spain!
    http://www.readwriteweb.es/tecnologias/mejor-bing-google/

  • http://nitinchan.blogspot.com Nitin Chandak

    Can it be that google receives bing search result signals too? And is Quora not a bit personal opinion website more than say correct answer providing..
    WE need a search engine which can provide us what we want in the shortest time possible and google has done it till now so its survived at the top …

 

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide