Report: ChaCha Voice Search Beats Google, Yahoo/Vlingo In Accuracy, Reliability

A new report finds that mobile search and answers service ChaCha beat Google and Yahoo’s voice search tools in terms of overall accuracy and reliability across a range of query types in a controlled study. The study was conducted by Albright Communications/MSearchGroove in January of this year.

The report, which can be downloaded from the ChaCha site, was sponsored by ChaCha but the authors said, “While ChaCha sponsored the study, it was fully removed from the planning, testing, analysis, and reporting of this work.” The methodology was very specific and relatively elaborate. I summarize it below.

Both Google Voice Search and Yahoo oneSearch with Voice (powered by Vlingo) use automation for voice search. ChaCha uses humans to interpret voice queries, which are submitted through an 800 number: 1-800-2-ChaCha. Queries can also be submitted via text or online.

Here are the bottom line findings from the report:


Source: Albright Communications/MSearchGroove

Here’s how the two categories were defined:

  • Natural language queries: queries submitted as questions using normal conversational syntax
  • Keyword queries: queries seeking the same results as the natural language queries but reduced to the least number of keywords . . . necessary

What the table and findings above show is that in a “natural language” context, ChaCha was substantially better than Google and somewhat better than Yahoo/Vlingo. In a keyword query context the results are somewhat closer, but ChaCha still comes out on top.

One might expect this result given the involvement of human beings at ChaCha vs. pure automation with the others. From a pure user experience perspective, putting aside the back end, the report finds that ChaCha offers greater accuracy and reliability than either search engine when it comes to voice input.

The methodology:

All searches were performed by a single individual on the iPhone 3G. Here’s an abridged but otherwise verbatim description of the methodology used by the report’s authors:

To assess overall performance of the voice-enabled search services in a typical range of use cases and scenarios, we created 18 queries representative of mobile search usage and trends.

The queries covered search categories considered common in the mobile environment, such as navigation, directions, local information, general information on timely topics, and specialized or unusual long- tail topics. We included specific queries that represent the most popular mobile search terms in 2008, based on mobile search data publicly reported by AOL and Yahoo, as well as social queries. We did not include transactional queries . . . associated with downloading music and applications . . . and purchasing goods and services via a device.

Microsoft’s Tellme and Nuance Voice Control were not included in the study.

Related Topics: Channel: Mobile | Google: Mobile | Google: Voice Search | Microsoft: Bing 411 | Search Engines: Mobile Search Engines | Yahoo: Mobile & Go


About The Author: is a Contributing Editor at Search Engine Land. He writes a personal blog Screenwerk, about SoLoMo issues and connecting the dots between online and offline. He also posts at Internet2Go, which is focused on the mobile Internet. Follow him @gsterling.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide