ChaCha Defeats Google, Bing And Siri In “Answers Quality” Study

A study published today by Indiana-based Butler University compared the performance of a range of search engines and Q&A sites on mobile devices. It found that “ChaCha delivered the highest quality responses consistently across the largest group of categories and question types.”

The study was sponsored by ChaCha. However, ChaCha CEO Scott Jones told me via email that the company had no involvement with the study methodology, nor did it seek to influence the outcome in any way. Given that this is sponsored research, many people will undoubtedly be skeptical of the results, so I’ve embedded the full report below.

The following are the relative coverage and accuracy rankings of each of the companies involved in the study. After ChaCha, Ask.com was found to be second most accurate while Siri and Quora were near the bottom. Bing also beats Google.

A total of 3,960 questions were asked in the course of the study to generate the scoring and rankings above. The report contains more detailed discussion of the performance of each of the companies across a range of content categories. However, most of the report is an explanation of the study methodology:

To assess the quality of the responses of these mobile Q&A platforms, we conducted mobile research, which included the following activities:

• Posing a sample set of questions to each of the Q&A platforms using a mobile application when one was available and a mobile website when no application existed
• Recording all responses
• Rating the responses from each Q&A platform for coverage and accuracy
• Analyzing data and tabulating it in a summarized format 

Unfortunately, the report doesn’t include the list of questions/queries asked or even examples. So, it’s difficult to fully assess the study on the basis of the report alone. Regardless, it’s a provocative and unexpected outcome. The results certainly confound popular expectations, which would put Google closer to the top, if not at the top.

Read the report and let us know what you think of these findings.