Report: ChaCha Voice Search Beats Google, Yahoo/Vlingo In Accuracy, Reliability

A new report finds that mobile search and answers service ChaCha beat Google and Yahoo’s voice search tools in terms of overall accuracy and reliability across a range of query types in a controlled study. The study was conducted by Albright Communications/MSearchGroove in January of this year.

The report, which can be downloaded from the ChaCha site, was sponsored by ChaCha but the authors said, “While ChaCha sponsored the study, it was fully removed from the planning, testing, analysis, and reporting of this work.” The methodology was very specific and relatively elaborate. I summarize it below.

Both Google Voice Search and Yahoo oneSearch with Voice (powered by Vlingo) use automation for voice search. ChaCha uses humans to interpret voice queries, which are submitted through an 800 number: 1-800-2-ChaCha. Queries can also be submitted via text or online.

Here are the bottom line findings from the report:

Source: Albright Communications/MSearchGroove

Here’s how the two categories were defined:

• Natural language queries: queries submitted as questions using normal conversational syntax
• Keyword queries: queries seeking the same results as the natural language queries but reduced to the least number of keywords . . . necessary

What the table and findings above show is that in a “natural language” context, ChaCha was substantially better than Google and somewhat better than Yahoo/Vlingo. In a keyword query context the results are somewhat closer, but ChaCha still comes out on top.

One might expect this result given the involvement of human beings at ChaCha vs. pure automation with the others. From a pure user experience perspective, putting aside the back end, the report finds that ChaCha offers greater accuracy and reliability than either search engine when it comes to voice input.

The methodology:

All searches were performed by a single individual on the iPhone 3G. Here’s an abridged but otherwise verbatim description of the methodology used by the report’s authors:

To assess overall performance of the voice-enabled search services in a typical range of use cases and scenarios, we created 18 queries representative of mobile search usage and trends.

The queries covered search categories considered common in the mobile environment, such as navigation, directions, local information, general information on timely topics, and specialized or unusual long- tail topics. We included specific queries that represent the most popular mobile search terms in 2008, based on mobile search data publicly reported by AOL and Yahoo, as well as social queries. We did not include transactional queries . . . associated with downloading music and applications . . . and purchasing goods and services via a device.

Microsoft’s Tellme and Nuance Voice Control were not included in the study.