Loup Ventures has published the results of its annual “smart speaker IQ” test. Google Home was the overall winner, but the surprise is that Apple’s HomePod and Siri bested Alexa and Cortana.
800 questions asked. The firm asked 800 questions to the virtual assistants and graded them on the basis of whether the question was understood and whether there was a correct response. Questions were organized into five content categories:
- Local – Where is the nearest coffee shop?
- Commerce – Can you order me more paper towels?
- Navigation – How do I get to uptown on the bus?
- Information – Who do the Twins play tonight?
- Command – Remind me to call Steve at 2 pm today.
The smart speakers tested were: a 2nd generation Amazon Echo, Google Home Mini, Apple HomePod and Harman Kardon Invoke. The following table shows the results comparison. Google Home understood all the queries and was able to correctly answer 88 percent of them. However Cortana was only able to answer 63 percent of questions correctly.
Google beat Amazon in Commerce category. All the virtual assistants improved from past tests conducted by Loup. The firm said that “Alexa had the most noticeable improvement.” In terms of specific areas, Cortana did best in the Information category and so did Alexa, where it nearly matched Google’s score. Siri performed best in Local and Navigation, but also did well in the Command category. Google outperformed all of its rivals in the Commerce category, where Alexa was surprisingly the worst performer despite the Amazon connection. 
Loup credited Google’s overall win partly to featured snippets. “Google Assistant’s outperformance stems largely from the search function ‘featured snippets’ . . . Where others may answer with, ‘here’s what came back from a search’ and a list of links, Google is able to read you the answer.”
Why you should care. Smart speakers and voice search have been more rapidly adopted than any preceding technology. They offer significant potential as search, content discovery and commerce platforms. The improvement in accuracy and capabilities reflected above will likely lead consumers to do more and ultimately help them realize their innate potential as a marketing channel.
For a specific discussion of what kind of innovations need to occur before that can happen, read our interview with LiveRamp VP Rebecca Stone.
About The Author
Greg Sterling is a Contributing Editor at Search Engine Land. He writes a personal blog, Screenwerk, about connecting the dots between digital media and real-world consumer behavior. He is also VP of Strategy and Insights for the Local Search Association. Follow him on Twitter or find him at Google+.