A new report finds that mobile search and answers service ChaCha beat Google and Yahoo’s voice search tools in terms of overall accuracy and reliability across a range of query types in a controlled study. The study was conducted by Albright Communications/MSearchGroove in January of this year.

The report, which can be downloaded from the ChaCha site, was sponsored by ChaCha but the authors said, “While ChaCha sponsored the study, it was fully removed from the planning, testing, analysis, and reporting of this work.” The methodology was very specific and relatively elaborate. I summarize it below.

Both Google Voice Search and Yahoo oneSearch with Voice (powered by Vlingo) use automation for voice search. ChaCha uses humans to interpret voice queries, which are submitted through an 800 number: 1-800-2-ChaCha. Queries can also be submitted via text or online.

Here are the bottom line findings from the report:

Source: Albright Communications/MSearchGroove

Here’s how the two categories were defined:

• Natural language queries: queries submitted as questions using normal conversational syntax
• Keyword queries: queries seeking the same results as the natural language queries but reduced to the least number of keywords . . . necessary

What the table and findings above show is that in a “natural language” context, ChaCha was substantially better than Google and somewhat better than Yahoo/Vlingo. In a keyword query context the results are somewhat closer, but ChaCha still comes out on top.

One might expect this result given the involvement of human beings at ChaCha vs. pure automation with the others. From a pure user experience perspective, putting aside the back end, the report finds that ChaCha offers greater accuracy and reliability than either search engine when it comes to voice input.

The methodology:

All searches were performed by a single individual on the iPhone 3G. Here’s an abridged but otherwise verbatim description of the methodology used by the report’s authors:

To assess overall performance of the voice-enabled search services in a typical range of use cases and scenarios, we created 18 queries representative of mobile search usage and trends.

The queries covered search categories considered common in the mobile environment, such as navigation, directions, local information, general information on timely topics, and specialized or unusual long- tail topics. We included specific queries that represent the most popular mobile search terms in 2008, based on mobile search data publicly reported by AOL and Yahoo, as well as social queries. We did not include transactional queries . . . associated with downloading music and applications . . . and purchasing goods and services via a device.

Microsoft’s Tellme and Nuance Voice Control were not included in the study.

Tellme has been in the speech, automated directory assistance and “voice search” segment for a decade. Two years ago the company was acquired by Microsoft for roughly $800 million. Since that time the two companies have been working on integration and developing next-generation services. Tellme powers Microsoft’s LiveSearch 411, among other services.

Today, two years to the day after the acqusition, Tellme made two related announcements about improved speech technology and an upgraded mobile user experience. (Here’s the enterprise-facing release.) On the consumer side of the equation, Tellme has introduced a “one-button” approach to initiate calls, dictate text messages and conduct searches with Microsoft Live Search:

The new service puts many of the most popular phone functions behind a single button.  Windows phone users just press the side button of their phone to:

• Send a text by saying “text” to open a text box, then speak the text message and send to call anyone in their contact list
• Initiate a call simply by saying “call” and then the name of anyone in their contact list
• Search the Web with Microsoft Live Search by speaking your request, such as “weather in San Francisco, California”, “Pizza in Kansas City” or “mother’s day gift ideas”

The Live Search mobile client has had a voice front end for some time, but that’s now been upgraded. Tellme’s technology is also being integrated at the network/OS level into Windows Mobile 6.5, which will reportedly ship later in the year.

Theoretically anything that you could do with search on the handset using Microsoft Live Search you can now do with voice faster and more easily. I saw the service demo’d briefly and it appeared to work very well. Tellme is arguing in its press materials that voice offers a better user experience than “touch” (i.e., the iPhone). However the iPhone has voice search clients available from Vlingo and Google. Vlingo is also behind Yahoo’s voice-powered oneSearch client.

There is clearly enormous potential for voice search to transform the mobile user experience and increase search volume dramatically. So far that has not happened. But the promise is there. Tellme’s deeper integration into Windows Mobile could represent a strategic differentiator for Microsoft and Windows Mobile phones but we’ll have to see once we get a chance to “live” with the service.

Tellme’s capabilities will also be an integrated feature of the new Kumo/Kiev/Live Search successor in mobile. I have some additional thoughts in a post at LocalMobileSearch.

Microsoft (so far) has been unable to buy Yahoo or Yahoo search assets, so it instead continues to take search talent away from the company. Last year Microsoft successfully lured engineering lead Qi Lu from Yahoo to become head of the company’s Online Services Group. Redmond also recently nabbed Sean Suchter from Yahoo.

Now Bloomberg and CNET report that another Yahoo executive, Larry Heck, will be working on Microsoft search. Heck was, until now, Vice President of the Search & Advertising Sciences at Yahoo Labs. Before that he was Vice President of R&D at Nuance Communications, “where he led teams responsible for natural language processing, speech recognition, voice authentication, and text-to-speech synthesis.”

This suggests more emphasis on voice search (read: mobile) at Microsoft. But the company has an overabundance of speech assets already, having acquired Tellme in March, 2007.

Last Friday, shortly after the Microsoft bid for Yahoo was made public, Danny and I had an opportunity to discuss the proposed acquisition with Yusuf Mehdi, senior vice president of strategic partnerships for Microsoft. Our questions to Mehdi often returned to post-acquisition “integration” issues. I brought up local, maps, and mobile as areas where both companies had made significant investments but also had significant duplication and redundancy. Mehdi responded, “We have not made decisions at that level, such as what to do with maps or mobile products. Every alternative is on the table. We will evaluate all those options and decide. We’ll pick whatever we think is the best options.”

Fair enough.

Yet in many ways, local is a perfect microcosm of the entire issue of product integration and the challenges surrounding determination of what brands or properties to retain and which to fold. Assuming Microsoft is successful with its bid, it won’t have to act quickly to promote or shutter any of the combined company’s local properties. But executives will be making choices during the transition about where to deploy resources and thus will be picking “winners” and “losers.” Here’s a tour of the local properties and the choices that may need to be made.

Local search destination sites

At one time Yahoo Local was the undisputed leader in local search and dynamic maps; it was the most innovative product in the market. It still has a great deal of community content that is not equaled by Microsoft’s newer Live Local property. It also has been creative and more expansive with what I like to call the “product definition,” including a range of additional content types, including events.

Yahoo Local:

Live Local/Maps:

While Yahoo Local is a more “differentiated” product – Live Local is similar to Google Maps – Microsoft’s mapping platform, Virtual Earth, is much deeper and more powerful at this point. In addition, Yahoo Local has more “brand recognition,” consumer awareness, and engagement than Live Local. Thus, I would expect some sort of combination of the two products, replacing Yahoo Maps with Virtual Earth and seeking to integrate Yahoo community content and features into a redesigned offering.

Yellow pages

The “yellow pages” sites of Yahoo and Microsoft continue to exist because they still serve a segment of the population that is looking for “name and number” information. Yahoo Yellow Pages was one of the earliest and most successful online yellow pages, but now looks very tired, in contrast to the other local resources in the market. By comparison, Microsoft yellow pages is a slightly modified version of Live Local and runs on that same platform.

Yahoo Yellow Pages:

MSN Yellow Pages:

I would expect both of these sites to disappear over time, especially Yahoo Yellow Pages. The combined local search product would probably support a perfunctory “yellow pages” offering, in the way that MSN handles it today. But Microsoft would likely direct traffic to a hybrid Yahoo Local-Live Maps site as discussed.

(There is a kind of parallel “shelf space” strategy that might apply to this entire discussion, where multiple properties are maintained and run off unified platforms and databases, which decreases costs and preserves consumer traffic volumes across the “network.” However, traffic trends will likely indicate and help determine whether a particular destination is preserved or folded.)

Verticals (Jobs, Cars, Real Estate)

Though many people don’t include these verticals in the discussion of “local,” they are explicitly about local markets; it’s the internet driving leads and sales offline. (For that matter, one could equally include travel in this discussion, and travel as a category will present similar challenges because both companies have strong travel assets.)

Unlike the maps category, where Microsoft clearly has the stronger product, in jobs, cars, and real estate there is a kind of rough equivalence between the competing sets of properties. If I had to choose one as a cluster, I’d probably say that the Yahoo properties are stronger and somewhat more fully developed.

Yahoo HotJobs (tied into the newspaper consortium):

MSN Careers (in partnership with CareerBuilder):

Yahoo Autos:

MSN Autos:

Yahoo Real Estate:

MSN Real Estate:

As you look at and use these three sets of properties, you’ll see that it’s difficult to differentiate them visually and in terms of functionality. As suggested, you would want to maintain both sets of destinations at least for a time, but doing so over the long term might not make sense if you’re trying to build strong, unified consumer brands.

Search

One of the questions near to the hearts of readers of this site is the question of “core search” and how a combined company would handle it. Microsoft has invested very heavily in search but remains the number three player. Yahoo is number two. I’m not an engineer, but let’s assume that Microsoft’s technology and relevance are on course to surpass Yahoo at this point. Yahoo remains the stronger brand and has more market share. You thus wouldn’t want to shut down Yahoo in favor of Live (or some successor search brand). As with Maps, you might substitute Microsoft’s platform behind Yahoo’s brand. That probably would be a common scenario if the deal goes forward.

Yahoo (local) search (”best sushi new york“):

Live (local) search (same query):

Finally, Microsoft would presumably inherit the newspaper consortium deal, which is a local ad network and ad serving deal with most of the major US newspaper groups except Gannett, New York Times, Washington Post, and Tribune Company. It’s a huge local asset and set of relationships, although unwieldy.

Mobile

Mobile includes a heavy dose of local, of course. And here the assets are arguably more complementary than duplicative. For example, Yahoo has no voice search or free directory assistance product. Microsoft’s 1-800-Call- 411 (built on the Tellme back end) is a best-of-class offering that fills a hole for Yahoo. By the same token, Yahoo’s new mobile development platform “Go 3.0” goes head-to-head with Google’s Android, and to a lesser degree, Windows Mobile.

Yahoo has broader and deeper mobile carrier relationships around the globe than Microsoft, and its consumer brand strength is greater even in the nascent mobile market. Furthermore, beyond adopting the “open” Go platform initiative, Microsoft could also favor Yahoo mobile content on Windows Mobile devices.

Overall, you can see how the question of which products to support and which ones to fold reproduces itself again and again in virtually every area of local. There promises to be considerable drama before Microsoft can close a deal with Yahoo. But assuming it does, as these examples show, that may turn out to be the easy part.

Greg Sterling is the founding principal of Sterling Market Intelligence and a senior analyst at Opus Research’s Local Mobile Search program. He publishes Screenwerk, a blog focusing on the relationship between the Internet and traditional media, with an emphasis on the local search marketplace. The Locals Only column appears on Mondays at Search Engine Land.

Late last month, at its “Searchification” event, Microsoft introduced a wide range of search enhancements and improvements. The company also said there would be additional updates to follow. Those are now rolling out and include feature and content enhancements for maps, local, and mobile services. The biggest news is the introduction of a new voice search product called “Live Search 411,” built on Tellme’s content and voice capabilities. In addition to local business listings, the service provides movie show times, weather, and airlines information.

We haven’t had a chance to test drive or play with any of these new features, but here is the list from the official Microsoft press release: Maps and Directions

• Live Search for use with BlackBerry® Devices. Live Search designed for use with BlackBerry devices is in beta and will be available for download for free on Tuesday, Oct. 16. It is a new offering designed for use on BlackBerry OS 4.0.2 or later phones, and customers with GPS-enabled phones can use location awareness to enhance their searches.

Of note here is the expansion of 3-D coverage, combination of Birds Eye and 3-D views, and the introduction of user generated content, including the use of Google’s KML (now a standard). Regarding the mobile developments, there’s more available information at LiveSearchMobile.

We’re also told there will be additional information on the full range of enhancements later this evening on the Live Search blog and the Virtual Earth blog.

Postscript: New Live Search Maps Features Coming is now up from the Virtual Earth blog; the press release is also now live and discussion is going on Techmeme.

Seemingly in parallel with the land grab going on online, mobile M&A is heating up. Earlier this month, Microsoft acquired EU ad firm ScreenTonic and prior to that voice services and directory provider Tellme. Today, mobile ad network Third Screen Media went to AOL and will become a subsidiary of Advertising.com. Microsoft had been seen as a Third Screen suitor earlier in the year.

Third Screen is a strategic acquisition for AOL, as mobile becomes a more important category and advertising becomes the source of more and more revenue for AOL.

Microsoft announced earlier today that it had acquired six-year-old European mobile ad platform company ScreenTonic. Last February, Microsoft acquired European mobile search provider MotionBridge. This past March Microsoft bought Tellme. Thus, the company is putting together an impressive suite of mobile assets for both consumers and carriers.

Microsoft said it intends to deploy the ScreenTonic technology globally. And while Google and Yahoo are largely perceived as threats to carriers (justified to some degree by recent consumer research from iCrossing), Microsoft is seeking to position itself as a partner, directly and indirectly through these acquired companies.

From the release:

[Steve] Berkowitz said it is important for Microsoft to deliver ad experiences that are mutually beneficial to publishers, mobile operators and consumers alike. ScreenTonic has relationships with some of the largest mobile operators in Belgium, France and the U.K.

In the U.S. Microsoft has a strategic relationship with struggling mobile carrier Sprint.

Microsoft, which recently acquired Tellme Networks, announced three new local business search services, all accessible via phone. The new services include:

Tellme by Voice: Call 1-800-555-TELL and just say “business search” to find a business listing or search for a particular category, such as “flower shops,” from any phone, including a map for the selected listing. Callers can also choose from many other great services like movie tickets, ring tones, sports, news and more.

Tellme by Text: Send a text message to TELLM (83556) with a business request such as “starbucks san francisco ca” or “pizza boston ma” and get a text message back with the listing and a link to a map.

Tellme by Mobile: Speak your request and then see the results displayed on your phone screen for a voice + visual search, including maps and driving directions. Download the Tellme by Mobile beta by visiting Tellme.com’s web page or from your mobile phone.

Screenshots of the new local search services are available here.