Study Disputes “Bing It On” Claim That 2:1 Prefer Bing To Google

A new study appearing on the “Freakonomics” blog aggressively disputes the claim that people prefer Bing to Google and especially the statistical contention that they prefer the search engine over Google 2:1. In an article explaining the study, law and economics professor Ian Ayers argues that the Bing television and related media campaigns based on the Bing It On challenge constitute deceptive advertising.

Ayers says the 2:1 claim comes from a study of just 1,000 users. Using Mechanical Turk, he sought to replicate it with help from students at Yale Law School where he teaches:

[T]ogether with four Yale Law students, I set up a similar-sized experiment using Microsoft’s own BingItOn.com site to see which search engine users prefer. We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were “ties”) …

We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred. We randomly assigned participants to search for one of three kinds of keywords: Bing’s suggested search terms, popular search terms, and self-suggested search terms . . .

The following chart shows the results of the Ayers study, which tested three categories of search queries:

The Ayers study found that Google was preferred most when study participants “used popular search terms or selected their own search terms.” Those who searched with the Bing-suggested queries “preferred Bing and Google in nearly equal numbers.”

There are two potential, contradictory reactions to the Ayers study:

• It either conclusively or largely disproves the Bing preference claims
• Putting aside the Bing advertising claims, the search engine performed relatively well vs. Google

Google won 53 percent of the time and Bing won 41 percent of the query tests, with a tie in 6 percent of instances. That suggests that Bing has the capacity to gain much more market share than it currently has (67 percent vs. 18 percent).

Ayers points out that the more assertive “prefer Bing 2:1” claim has been replaced on the Bing It On website with the more limited claim that “people prefer Bing.” A report with the full methodology and findings of the Yale study is available here.

Postscript: Microsoft provided the following comment on the Ayers/Yale study (attributed to Matt Wallaert, Behavioral Scientist, Bing):

The professor’s analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100% accurate and we’re glad to see we’ve nudged Google into improving their results.  Bing it On is intended to be a lightweight way to challenge peoples’ assumptions about which search engine actually provides the best results. Given our share gains, it’s clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service.