Perficient Digital released the latest version of its now annual Digital Personal Assistants accuracy study. It compared responses to roughly 5,000 queries on seven devices including Amazon’s Alexa (Echo and Echo Show), Microsoft’s Cortana, Google Assistant (Home, smartphones), and Siri.
More answers, less accuracy. At the highest level, Google Assistant performed the best, but Cortana attempted to answer the most questions. Alexa also showed improvement in answer attempts. But accuracy declined on all devices, according to the study.
The chart below compares devices by number of answers attempted, which means they didn’t respond with “I don’t know that” or “I can’t help with that yet.” Three years of data show Cortana and Alexa have grown the most in answer attempts, with Cortana edging Google for most questions answered (though not always correctly).
Alexa the second-most accurate assistant, after Google. The most accurate assistant is Google (on a smartphone). Alexa comes in second. However, accuracy seems to have declined across the board and most for Cortana, which could be related to its attempt to answer more questions. Siri also suffered a meaningful decline in accuracy.
Perficient Digital explored the use of featured snippets by the assistants. It defines snippets as “answers provided by a digital personal assistant or a search engine that have been sourced from a third party” (with attribution).
Decline in use of snippets by Google. Google served up the most snippets, with Google Home beating out the Assistant on smartphones. However, Google Assistant on the smartphone also saw a significant decline use of snippets. It was the only platform to see such a decline.
Finally, Alexa and Siri tied for the most jokes offered in response to queries. Accordingly, they were deemed to be “the funniest” assistants.
Why we should care. Perficient Digital concludes in its discussion of the data that “progress has stalled to a certain degree.” The company explains, “We’re no longer seeing major leaps in progress by any of the players,” adding, “The next significant leap forward will likely require a new approach.” Google may have found that approach in BERT, which helps the search engine better understand the meaning of more conversational queries.
