Focus Group Study Offers Good & Bad News For Bing

TechCrunch yesterday reported on the results of a focus group conducted by NY design firm Catalyst Group comparing usability on Bing and Google. The study showed that in many instances Bing was preferred but that users already familiar and comfortable with Google would likely stay with Google. What’s partly interesting about this is that the results can be read either as confirming the challenge Microsoft faces in gaining adoption for Bing or to argue that for the first time Microsoft has a “real contender” on its hands and that Google should be concerned.

The focus group consisted of 12 people in New York who used Google as their main search engine. No one in the group had used Bing before. Users were asked to conduct a “hotel search” and a “shopping search for digital cameras” on both sites. According to the discussion in the slides, “users completed both searches on one site and then repeated the searches on the other site. The task order was rotated to mitigate bias effects.” The firm also collected eye tracking data. Users were interviewed about their experiences and there was also a short written survey they filled out. (One must be cautious about generalizing too much from a focus group, but the results should not be dismissed either.)

The following were the Catalyst findings:

1. Most users preferred the Bing visual design and thought that it was slightly better in terms of organization and refinement options.
2. However, despite some positive factors in Bing’s favor, users thought that both search engines produced equally relevant results. Overall, most users stated they would continue using Google over Bing and were unlikely to switch.
3. For the camera search, Bing attracted 150% more user viewing time to the ad space at the top of the search result.

Here’s a slide that shows the survey scoring and user preferences:

Note that on “visual design,” “organization” and “refine & filter options” Bing is preferred (the dark boxes). Relevance is a tie for most people in the group. Also note that the “overall reasons for preferring” seem to contract the scoring in the individual categories. Google wins 8 to 4 based on:

• Familiarity
• Use of other Google apps (probably Gmail, Toolbar, Maps, Calendar)
• Enhancements in Bing not enough to convert

Overall people like the design and interface features somewhat more with Bing.

In the run up to the Bing launch I had a couple of conversations with Microsoft in which I provided my response to the site and user experience. The things I liked about Bing are mirrored in the feedback provided by the focus group above. Microsoft echoed back to me that the design and UI scored well among testers. And “early returns” from comScore, Hitwise and others have indicated that Bing has gained some traction with the general population of searchers.

Let’s assume for argument that these focus group results are representative of a broader population of searchers and that the look and feel and certain features of Bing are indeed preferred. Let’s also assume that relevance is also basically perceived as a tie or that users cannot see significant differences between the engines. If you’re Google you can’t feel too good about any of that. It adds some weight to the oft-repeated yet otherwise empty remark “the competition is just a click away.” Yet if you’re Microsoft you see these generally positive results together with the “we’re unlikely to switch” statements and you’ve got to feel frustrated. But maybe you’re also encouraged.

In the context of this study, perhaps one of the most interesting findings is the reference in the Google preferences column to “already using other Google apps,” showing how Google’s larger “ecosystem” of tools, sites and apps emerges as a reinforcer of loyalty and continued usage. Related to that is the “inertia” surrounding familiarity with Google. Usage begets usage.

It would appear then that if Bing can gain some regular usage it will increase its own “familiarity” among searchers and a potentially build a cycle of increasing usage. Advertising and the several “default” search deals that Microsoft has done (HP, Dell, Lenovo, Verizon) would appear to have meaning then in exposing people to Bing and getting them to try and use it.

Again, I don’t believe that Bing is a threat to Google in any major way today. By the same token, without taking this too far, these focus group results suggest to me that despite points for “familiarity” there is some fatigue or even perhaps boredom with the Google UI.

If you want to take a look at the heatmaps and other specific comparisons, you can see and download the Catalyst Group slides here.