ChaCha Defeats Google, Bing And Siri In “Answers Quality” Study

A study published today by Indiana-based Butler University compared the performance of a range of search engines and Q&A sites on mobile devices. It found that “ChaCha delivered the highest quality responses consistently across the largest group of categories and question types.” The study was sponsored by ChaCha. However, ChaCha CEO Scott Jones told me via email […]

Chat with SearchBot

ChaCha logoA study published today by Indiana-based Butler University compared the performance of a range of search engines and Q&A sites on mobile devices. It found that “ChaCha delivered the highest quality responses consistently across the largest group of categories and question types.”

The study was sponsored by ChaCha. However, ChaCha CEO Scott Jones told me via email that the company had no involvement with the study methodology, nor did it seek to influence the outcome in any way. Given that this is sponsored research, many people will undoubtedly be skeptical of the results, so I’ve embedded the full report below.

The following are the relative coverage and accuracy rankings of each of the companies involved in the study. After ChaCha, Ask.com was found to be second most accurate while Siri and Quora were near the bottom. Bing also beats Google.

Butler search quality test

A total of 3,960 questions were asked in the course of the study to generate the scoring and rankings above. The report contains more detailed discussion of the performance of each of the companies across a range of content categories. However, most of the report is an explanation of the study methodology:

To assess the quality of the responses of these mobile Q&A platforms, we conducted mobile research, which included the following activities:

  • Posing a sample set of questions to each of the Q&A platforms using a mobile application when one was available and a mobile website when no application existed
  • Recording all responses
  • Rating the responses from each Q&A platform for coverage and accuracy
  • Analyzing data and tabulating it in a summarized format 

Unfortunately, the report doesn’t include the list of questions/queries asked or even examples. So, it’s difficult to fully assess the study on the basis of the report alone. Regardless, it’s a provocative and unexpected outcome. The results certainly confound popular expectations, which would put Google closer to the top, if not at the top.

Read the report and let us know what you think of these findings.

Postscript: Butler has published a new version of its report that includes all the questions in an appendix. The report is available as a download (no registration) here.

Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.


About the author

Greg Sterling
Contributor
Greg Sterling is a Contributing Editor to Search Engine Land, a member of the programming team for SMX events and the VP, Market Insights at Uberall.

Get the must-read newsletter for search marketers.