Google (Now) Beats Siri And Cortana For Direct Answers — Study

Stone Temple Consulting (STC) published a study in which it compared mobile answers to “3086 different queries” on Siri, Google and Cortana to determine how each performed relative to the others. Google was the winner with more direct answers and more complete information.

Eric Enge, who wrote up the results, says this was not a “personal assistant comparison.” Rather, it was a “knowledge box” or “knowledge panel” comparison. We’ll instead refer to these as “answers,” which would be the layperson’s view of the information.

The queries tested were mostly factual, such as “What’s the tallest mountain in the world?,” “How long is the Gettysburg Address?” and “When is the next Stanley Cup?” Although there were occasional opinion questions such as “What is love?”

I’m not going to discuss the study methodology in depth. One might critique it in various ways, but I accept the overall findings as valid. Below is a summary of the major findings.

STC noted that Google returns more enhanced results vs. Cortana and Siri when asked questions expected to produce such a result (“knowledge box”).

Google also bested the other two in terms of providing most direct and complete answers to fact-based queries. In nearly 90 percent of instances, Google did a better job of clearly and directly answering the query posed. STC said that it was very strict in grading this portion of the test and didn’t accept as correct answers that partially or indirectly responded to the question.

One of the more interesting findings reports that in cases where Google is presenting an “answer,” STC says that it only gives third-party attribution 25 percent of the time. The other 75 percent of answers had no source.

In summary, STC concludes that Google “has a clear lead in terms of the sheer volume of queries addressed, and more complete accuracy with its queries than either Siri or Cortana … [and that] Google is progressing the fastest on all fronts.”

Siri is not a search engine but a combination of natural language understanding (NLU) technology and “artificial intelligence” to help deliver third-party information. There is no Apple index to draw from; Siri “routes” queries to information or answers — including web search results from Bing. Indeed, it’s somewhat ironic that Microsoft is aggressively marketing Cortana (vs. Siri) to promote Windows Phones because Bing is the source of Siri’s web search results.

The original vision for Siri was a smart voice front end tied to multiple structured data sources on the back end to completely bypass the SERP in mobile and yield “actionable” information. Siri’s arrival a few years ago was one reason Google has pushed deeper into structured information — in order to provide more “answers” in mobile. That and the general movement away from “10 blue links” also motivated Google to put more answer boxes and knowledge graph results on PC SERPs.

Siri has not lived up to its promise, and the question for Apple is how much of a search arms race it wants to get into. Regardless, it needs to “fix” Siri. For its part, Microsoft intends to make Cortana an interface across platforms and devices, including Xbox. The personal assistant metaphor and the provision of “answers” are slowly taking over from basic search and conventional results.

The movement toward answers is also a major source of publisher frustration and the sense of “betrayal” at the heart of much of the recent anti-competitive criticism of Google.

About The Author

Greg Sterling

Greg Sterling is a Contributing Editor at Search Engine Land. He writes a personal blog, Screenwerk, about connecting the dots between digital media and real-world consumer behavior. He is also VP of Strategy and Insights for the Local Search Association. Follow him on Twitter or find him at Google+.