Voice assistant study: Microsoft’s Cortana offers most answers, Google Assistant proves most accurate

Perficient Digital ran 5,000 queries on seven devices including Alexa, Cortana, Google Assistant (Home, smartphones) and Siri.

Chat with SearchBot

Perficient Digital released the latest version of its now annual Digital Personal Assistants accuracy study. It compared responses to roughly 5,000 queries on seven devices including Amazon’s Alexa (Echo and Echo Show), Microsoft’s Cortana, Google Assistant (Home, smartphones), and Siri.

More answers, less accuracy. At the highest level, Google Assistant performed the best, but Cortana attempted to answer the most questions. Alexa also showed improvement in answer attempts. But accuracy declined on all devices, according to the study.

The chart below compares devices by number of answers attempted, which means they didn’t respond with “I don’t know that” or “I can’t help with that yet.” Three years of data show Cortana and Alexa have grown the most in answer attempts, with Cortana edging Google for most questions answered (though not always correctly).

Answer Attempeted 2017 Vs 2018 Vs 2019 1
Source: Perficient Digital 2019 DPA accuracy study

Alexa the second-most accurate assistant, after Google. The most accurate assistant is Google (on a smartphone). Alexa comes in second. However, accuracy seems to have declined across the board and most for Cortana, which could be related to its attempt to answer more questions. Siri also suffered a meaningful decline in accuracy.

Fully And Completely 2017 Vs 2018 Vs 2019 1
Source: Perficient Digital 2019 DPA accuracy study

Perficient Digital explored the use of featured snippets by the assistants. It defines snippets as “answers provided by a digital personal assistant or a search engine that have been sourced from a third party” (with attribution).

Decline in use of snippets by Google. Google served up the most snippets, with Google Home beating out the Assistant on smartphones. However, Google Assistant on the smartphone also saw a significant decline use of snippets. It was the only platform to see such a decline.

Featured Snippets 2019 1
Source: Perficient Digital 2019 DPA accuracy study

Finally, Alexa and Siri tied for the most jokes offered in response to queries. Accordingly, they were deemed to be “the funniest” assistants.

Why we should care. Perficient Digital concludes in its discussion of the data that “progress has stalled to a certain degree.” The company explains, “We’re no longer seeing major leaps in progress by any of the players,” adding, “The next significant leap forward will likely require a new approach.” Google may have found that approach in BERT, which helps the search engine better understand the meaning of more conversational queries.


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.


About the author

Greg Sterling
Contributor
Greg Sterling is a Contributing Editor to Search Engine Land, a member of the programming team for SMX events and the VP, Market Insights at Uberall.

Get the must-read newsletter for search marketers.