ChaCha Defeats Google, Bing And Siri In “Answers Quality” Study

ChaCha logoA study published today by Indiana-based Butler University compared the performance of a range of search engines and Q&A sites on mobile devices. It found that “ChaCha delivered the highest quality responses consistently across the largest group of categories and question types.”

The study was sponsored by ChaCha. However, ChaCha CEO Scott Jones told me via email that the company had no involvement with the study methodology, nor did it seek to influence the outcome in any way. Given that this is sponsored research, many people will undoubtedly be skeptical of the results, so I’ve embedded the full report below.

The following are the relative coverage and accuracy rankings of each of the companies involved in the study. After ChaCha, Ask.com was found to be second most accurate while Siri and Quora were near the bottom. Bing also beats Google.

Butler search quality test

A total of 3,960 questions were asked in the course of the study to generate the scoring and rankings above. The report contains more detailed discussion of the performance of each of the companies across a range of content categories. However, most of the report is an explanation of the study methodology:

To assess the quality of the responses of these mobile Q&A platforms, we conducted mobile research, which included the following activities:

  • Posing a sample set of questions to each of the Q&A platforms using a mobile application when one was available and a mobile website when no application existed
  • Recording all responses
  • Rating the responses from each Q&A platform for coverage and accuracy
  • Analyzing data and tabulating it in a summarized format 

Unfortunately, the report doesn’t include the list of questions/queries asked or even examples. So, it’s difficult to fully assess the study on the basis of the report alone. Regardless, it’s a provocative and unexpected outcome. The results certainly confound popular expectations, which would put Google closer to the top, if not at the top.

Read the report and let us know what you think of these findings.

Postscript: Butler has published a new version of its report that includes all the questions in an appendix. The report is available as a download (no registration) here.

Related Topics: Channel: Consumer | Search Engines: Help Engines

Sponsored


About The Author: is a Contributing Editor at Search Engine Land. He writes a personal blog Screenwerk, about SoLoMo issues and connecting the dots between online and offline. He also posts at Internet2Go, which is focused on the mobile Internet. Follow him @gsterling.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • JLishere

    I’ve never heard of ‘ChaCha’ until now.

  • http://twitter.com/pmeinertzhagen Peter Meinertzhagen

    No, neither had I, and my brief experience with it just now was greeted with an irrelevant answer to my question (I asked “How can I generate new blog ideas” and was given the answer “The first thing you need to do is to create a YouTube account. Once you have one, you can create a video blog by sharing videos.” and an interface that didn’t work properly (I was asked to sign in but the sign in button is unresponsive).

    I’ll certainly give it more of a test drive though to see what I think.

  • http://www.distressedvolatility.com/ Dvolatility

    interesting.

  • http://www.facebook.com/drdadman Dan Roberts

    Not surprising at all. For all the hype around ‘semantic search’ and ‘AI’ solutions like Siri, the truth is they have a long way to go before they can handle natural language queries as well as ‘human-assisted’ or ‘crowdsourced’ solutions – which this study would seem to confirm.

    That said, there is tremendous room for improvement and innovation as the current top contentders, Chacha and Ask both rely heavily on automation – and consequently often spit out answers that kind of answer the question asked – but not really.

  • http://www.seo-theory.com/ Michael Martinez

    Numbers in a report with no data to back them up. And this was newsworthy because…?

  • http://twitter.com/muddylemon Lance Kidwell

    In the study they reveal that the source questions came from two sources 1) Questions asked to the ChaCha service and 2) Questions collected from the android knockoff of Siri, Iris.

    It’s hardly surprising that they scored well in the metric of ‘having answers to questions that people have specifically asked you.” Half the questions were their own questions and the other half from the service most like their system – responding to questions asked over the phone.

  • http://www.facebook.com/emeka.sebastine.9 Emeka Sebastine

    at JLishere. i have never heard of chacha until now. anyway i grabed something tangible. http//www.unn.edu.ng

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!