Study: Bing More “Biased” Than Google; Google Not Behaving Anti-Competitively

Does Google favor its own sites in search results, as many critics have claimed? Not necessarily. New research suggests claims that Google is “biased” are overblown, and that Google’s primary competitor, Microsoft’s Bing, may actually be serving Microsoft-related results “far more” often than Google links to its own services in search results.

In an analysis of a large, random sample of search queries, the study from Josh Wright, Professor of Law and Economics at George Mason University, found that Bing generally favors Microsoft content more frequently, and far more prominently, than Google favors its own content. According to the findings, Google references its own content in its first results position in just 6.7% of queries, while Bing provides search result links to Microsoft content more than twice as often (14.3%).

The results from the new study by Wright, sponsored by the International Center for Law & Economics (ICLE) are important, especially given the challenges Google has recently faced from an FTC Inquiry over its business practices, antitrust complaints and Senate hearings looking into its alleged anti-competitive behavior.

The findings of the new study are in stark contrast with a study on search engine “bias” released earlier this year. That study, conducted by Harvard professor Ben Edelman concluded that “by comparing results across multiple search engines, we provide prima facie evidence of bias; especially in light of the anomalous click-through rates we describe above, we can only conclude that Google intentionally places its results first.”

How can the conclusions from two prominent scholars be so different? And, perhaps more importantly, given recent interest (and potential oversight) by lawmakers and regulators in search engine activities, what’s going on behind the scenes here?

A Tale Of Two Studies

First, some background. Harvard’s Ben Edelman has serious chops when it comes to search. He’s done thoughtful research into many important issues involving the dark side of the internet, including deceptive advertising, spyware, and so on. But: He’s also been a longtime paid consultant to Microsoft.

The new research from professor Wright, who has a deep interest in antitrust law and economics, was sponsored by the International Center for Law & Economics, with a mission “to create the academic underpinnings for a regulatory environment that ensures the protection of property rights from inefficient interference by government agencies and private parties in high priority markets.”

So, Microsoft has a paid consultant in its court. Did Google sponsor the new research, or influence its outcome? Google continues to increase spending on lobbying U.S. government officials to advocate its interests and ward off the attacks from competitors—was this another, more subtle approach to fend off critics?

Apparently not.

Although ICLE, which sponsored the research, has received financial support from several companies, organizations and individuals (including Google) Geoffrey A. Manne, Executive Director organization responded to my inquiry with this statement:

“The study was not done at Google’s request, and they had no involvement in the design, methodology or conclusions. Rather, the idea for the study and its execution were entirely Josh’s. It was undertaken independently and supported, as all of our affiliates’ supported work is, with an unrestricted grant from ICLE.”

Harvard’s Edelman released the results of his study in January of this year. As said above, Edelman concluded that he found “prima facie evidence of bias” that Google was promoting itself when it “shouldn’t” have been based on other alternative search results.

Search Engine Land’s editor-in-chief Danny Sullivan skewered Edelman’s results, writing that “statistics can easily be turned to whatever you want them to be. I feel like Edelman is turning his study into the most negative view possible. I’m just looking to provide some balance to that.”

Wright’s study had two missions: First, replicate the Edelman study to test its findings, and second, expand on it to eliminate perceived problems with methodology and conclusions. For the first part—replicating the Edelman study, Wright found that Google references its own content more favorably than rival search engines for only a small fraction of terms, whereas Bing is more likely to do so. “For example, in our replication of Edelman & Lockwood, Google refers to its own content in its first page of results when its rivals do not for only 7.9% of the queries, whereas Bing does so nearly twice as often (13.2%),” the report said.

For the second part of the study, Wright employed a much larger, random sample of search queries, rather than the small set (32 different searches) that Edelman performed. Wright’s expanded study found that Bing generally favors Microsoft content more frequently—and far more prominently—than Google favors its own content. Google references own content in its first results position when no other engine does in just 6.7% of queries, while Bing does so over twice as often (14.3%).

So, what conclusions to draw? Wright says that “analysis finds that own-content bias is a relatively infrequent phenomenon”—meaning that although Microsoft appears to favor its own sites more often than Google, it’s not really a major issue, at least in terms of “bias” or “fairness” of search results that the engines present. Reasonable conclusion: Google (and Bing, though less so) really are trying to deliver the best results possible, regardless of whether they come from their own services (local search, product search, etc) or not.

The Bigger Issue: Is “Search Neutrality” A Good Thing?

The study also looked at the whether search engines should be “neutral”—being “fair” to websites in terms of ranking rather than attempting to ferret out the best results for searchers.

It’s both a tradition and a blood sport in the U.S. for traditional media to transform the popular image of scrappy startups into “evil empires” when they’ve grown successfully into a dominant player. Microsoft has been through this; Google is now on the cusp of being the new creepy champ in the minds of many.

A large part of the media and regulatory oversight is now focused on whether Google is “anti-competitive“—typically shorthand for favoring its own content over that of other search engines. The Wright study concludes that the complaintants are simply wrong, saying “many of these complaints ignore the fact that search engine users self-select into different engines or use multiple sources for different types of searches when considering the competitive implications of search rankings.” Well said, and for most sophisticated internet users, probably true.

Why? Consider this scenario: You are going out to dinner in a new town tonight. So, to get recommended restaurants, are you going to Google that—or tweet a request for suggestions, or look up reviews on Yelp or Open Table or Chowhound or… My guess is that most people are going somewhere other than Google for this type of information. Or, for other types of information, going to WebMD or SeatGuru or IMDB or Wikipedia or countless other specialized sites (or mobile apps like Alfred or Ness) when they want more nuanced results than Google typically delivers. No question: Google has a lock on basic queries, and it’s really, really good for those. But the web is huge and asking Google is more often than not like asking the gas station attendant how to get somewhere when your GPS has died. In other words, internet users are smarter than critics give them credit for. We have options, and many are increasingly aware of our non-Google options (hello, Siri?).

But just because a company has grown into a dominant position doesn’t mean they’re doing wrong, or that governments should intervene and force changes that may or may not be “beneficial” to users or customers. I’m not going to rant about this. But in light of the findings of these studies, the pedigree of the researchers and the starkly contrasting opinions they offer, I’d encourage you all to read these analysis pieces, and form your own opinion—and contribute to the comments in the dialog below.

Link to the full study from professor Wright: Defining and Measuring Search Bias: Some Preliminary Evidence (pdf).

More reading (thanks to Gary Price for the links):

Related Topics: Channel: Industry | Features: Analysis | Google: Antitrust | Google: Business Issues | Microsoft: Bing | Top News

Sponsored


About The Author: (@CJSherman) is a Founding Editor of SearchEngineLand.com and President of Searchwise LLC, a Boulder Colorado based Web consulting firm. He also programs and co-chairs the Search Marketing Expo - SMX conference series.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Shivaun

    I am CEO and co-founder of SearchNeutrality.org and Foundem, one of the companies at the center of the European Commission’s formal antitrust investigation into Google.

    In October 2009, we defined search neutrality as the principle that search engine results should be comprehensive, impartial, and based solely on relevance. Clearly, no two search engines will produce the same search results; nor should they. But any genuine pursuit of the most relevant results must, by definition, preclude any form of arbitrary discrimination. The problem for Google is that its Universal Search mechanism and its increasingly heavy-handed penalty algorithms are both clear examples of arbitrary discrimination.

    Anyone who is interested in the more general issue of whether or not Google favors its own sites in its search results, might be interested in viewing our annotated highlights of the recent US Senate Antitrust hearing into Google, available here:
    http://www.youtube.com/watch?v=BslAhJ5-C9g&hd=1

    Senator Lee’s questioning of Mr Schmidt referred extensively to a Foundem study of the comparative Google rankings of the US’s leading price comparison services, and the above video provides some useful insights and context to Mr Schmidt’s testimony.

    Shivaun Raff
    CEO and Co-Founder, Foundem and SearchNeutrality.org (a Foundem initiative)

  • TimmyTime

    “You are going out to dinner in a new town tonight. So, to get recommended restaurants, are you going to Google that—or tweet a request for suggestions, or look up reviews on Yelp or Open Table or Chowhound or… My guess is that most people are going somewhere other than Google for this type of information”
    ******************
    Your guess is worthless since we know that 92% of people use search engines (Google, Bing and the likes) and 3/3rd use them daily, so maybe they end up on Yelp but most likely after searching for “Italian restaurant zip-code.” If that wasn’t the case you wouldn’t have companies bid to have their search box everywhere. Funny, how what Google says ends u here and gospel.

    I can’t comment on the study because I haven’t studied but I know GoogleLand is not objective so people look at the study. My own anecdotal evidence is that you have to navigate like in a minefield to find an organic SERP in Google commercial searches.

  • TimmyTime

    sorry about the above grammar and spelin mustakes ;)

  • http://www.seo-theory.com/ Michael Martinez

    For what it’s worth, Joshua Wright co-authored a paper that argues Google’s practices are PROcompetitive (as opposed to ANTIcompetitive). Cf. papers.ssrn.com/sol3/papers.cfm?abstract_id=1577556

  • http://www.seo-theory.com/ Michael Martinez

    And THIS pro-Google paper co-written by Joshua Wright discloses that ICLE has received financial support from Google. http://www.laweconcenter.org/images/articles/search_neutrality_manne_wright_final.pdf

    I think this is sufficient to call into question the neutrality of Professor Wright’s point-of-view.

  • TimmyTime

    Just read the “study,” and Chris you should be ashamed of yourself. No doubt Matt Cutts and other Googlers will tweet this and maybe give a you candy by ‘checking out’ a site of yours not doing well in Google but your article SUCKS.

    As Michael pointed out, this guy is biased and funded by Google so you should take back this line “So, Microsoft has a paid consultant in its court.”

    Two, instead of spending half of the article to parrot Google’s talking points why didn’t you read the phony study yourself? Why didn’t you link at it?

    Did it ever occur to you that Microsoft actually has a lot more content than Google? Like all those Windows and Office guides or great articles and stories from MSN and MSNBC that have been online for a decade plus? What content does Google have? Other than youtube and Spam Places and we know how well they are doing?

  • http://searchengineland.com Aaron Wall

    If one pulls Compete.com downstream referral traffic for Bing & Google one can see that YouTube gets a lot higher traffic % from Google.

    Google also sticks product search at slot #3 over and over again…so ok that might not rank in position #1, but it is certainly still a heavy bias.

    I have also seen search results that had 3 or 4 or 5 listings (in a single SERP of 10) from books.google.com in them…and some of those also had plus.google.com in the same SERP!

    “Did Google sponsor the new research, or influence its outcome? … Apparently not.”

    The ICLE funding should count for something if Microsoft’s funding of Ben Edelman is a conflict.

  • http://europeforvisitors.com Durant Imboden

    I doubt if we’re ever going to get a study that definitively shows whether Google or Bing is more “biased,” so I’ll take both the Edelman and Wright studies with a grain of salt. Still, since we’ve heard the Edelman study ballyhooed, it’s probably useful to hear that the Wright study draws a diffferent conclusion.

    My own take on the matter is that, as at least one federal court has ruled, search rankings are “opinions” and are therefore protected by the First Amendment. If Google wants to bake YouTube and Maps into its Universal Search interface, or if Bing wants to boost official Microsoft pages for searches on “Windows” or “Office,” so be it.

  • TimmyTime

    “My own take on the matter is that, as at least one federal court has ruled, search rankings are “opinions” and are therefore protected by the First Amendment. If Google wants to bake YouTube and Maps into its Universal Search interface, or if Bing wants to boost official Microsoft pages for searches on “Windows” or “Office,” so be it.”

    My own take is that money makes the first amendment claim very weak. For example if Microsoft banned Chrome from Windows because it was bad in their “opinion,” it wouldn’t fly. The same way if Google rigs results to make more money then they have no first amendment claim. Do you think Google put their own links there because they know that they are the best or do they do it for money? What does your gut tell you?

    and “so be it” doesn’t fly either on monopolies, just ask Microsoft or Google that helped DOJ go after Microsoft.

  • http://europeforvisitors.com Durant Imboden

    Timmy, that’s a specious argument. Blocking Chrome from Windows–or Internet Explorer from the Chrome OS, for that matter–isn’t remotely the equivalent of making an editorial judgment that’s protected by the First Amendment. {Money has nothing to do with the First Amendment, either: Fox News doesn’t get less First Amendment protection than I do just because News Corp. is richer and has a bigger audience.)

  • http://searchengineland.com Aaron Wall

    “Fox News doesn’t get less First Amendment protection than I do just because News Corp. is richer and has a bigger audience”

    Right. Just the opposite is true. They get more of it. Talk to the OWS protestors getting beat & maced.

    Might makes right. TM

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide