| Jay Baer: | Hey guys, it’s Jay Baer from Convince & Convert, and joined today by my friend Eric Enge who’s the CEO of Stone Temple Consulting, an unbelievably effective and famous SEO content organization. Eric, it’s great to talk to you. You and your team put together this new report recently that . . . it’s stunning to me that you even did this. I know it’s the second year you’ve done it, but I was still shocked. It’s called Rating the Smarts of the Digital Personal Assistants in 2018. You go through and figure out what’s the most accurate and actually useful version of Alexa, Siri, Google Home, and Microsoft’s Cortana. I still can’t believe this. You sort of lined up the devices and asked them a bunch of questions. Thanks so much for talking about this. How did this come together? |
| Eric Enge: | Great question. First of all thanks for having me Jay, thrilled to be doing this together with you and talking about this, we always have great fun chatting. We have a set of 5,000 questions that we developed and that set of 5,000 questions are questions about informational topics as drawn from things that we happen to know, Google provides featured snippets for or that they might likely be providing future snippets for. Well to correct that, these are questions we thought there was some possibility they might. This is how those questions originally came together. |
| Jay Baer: | But the array of questions is pretty broad. I mean there’s a lot of different types of questions and intentionally so. |
| Eric Enge: | Yes, it is intentionally so. It is meant to be a broad range across any manner of different topics from history, to recipe, to . . . I don’t know how something is spelled, or kind of all over the map really. Broad by intent because we wanted to test a wide range of capabilities. Then what we did is we literally asked, using human voice, these 5,000 different queries of each device. We did it for Google Assistant running on a smartphone, Google Assistant running on Google Home, Alexa running on the Amazon Echo, Cortana running on the Harman Kardon Invoke speaker, and then Siri running on an iPhone, 25,000 questions that were manually asked. We took this set of questions and we did all this cataloging of all these things including did you get a verbal response from the device or the personal assistant? Did the response indicate that the device thought it understood the question and therefore tried to answer it? If it did so did it answer the question correctly? If it got it wrong then what kind of wrong answer was it? It was an extensive amount of work in analysis done on a query by query basis. |
| Jay Baer: | I tell you what, I think you told me you had 10 people working on this, just asking questions and logging the responses. That is a tremendous amount of human capital put into this project. |
| Eric Enge: | Yes absolutely, I mean for me actually I’m an intensely curious person. I want to know the answers to questions like this. It turns out a lot of other people wanted to know answers to these questions as well because we’ve gotten a lot of visibility out of the study. The fact that we did it last year and did it again this year, we kind of have an index now that we’re measuring how these things are progressing. |
| Jay Baer: | Yes that was the fascinating thing, I think the conclusion this year, is that Google is sort of the “best”, and obviously that’s circumstantial and things like that, but if you sort of had to pick one that Google probably performs the best today. At one point Siri was maybe better and now it’s not as good as it was. It’s not a static condition. That was the most interesting thing looking at last years report versus this years report that there really is quite a lot of variance year to year, which means either some of these things are learning, as machine learning would make you think as in the name and getting better, but others are perhaps getting worse and I’m not quite sure how that happens. |
| Eric Enge: | Well I don’t think anything actually got worse per se. In fact, the personal assistant that made the most progress was Alexa, so they made huge strides towards expanding the number of questions that they responded to and their overall accuracy. Cortana has expanded a lot and had actually a pretty good step forward as well, both in terms of numbers of questions answered and the accuracy in answering the questions. |
| Siri used to be the leader but they were the first out and that’s a few years back now. They just kind of didn’t push it the same way everybody else has. How something gets worse, so I’ll give you an example though. Alexa’s accuracy rate was actually down a bit from last year, but on the other hand, they were answering far more questions. The total number of questions . . . |
| Jay Baer: | It almost stands to reason that your accuracy would go down a little bit. |
| Eric Enge: | Yes that’s exactly how you might see a drop and that, in fact, happened with Alexa. |
| Jay Baer: | Do you feel like there is a real advantage for Alexa because it does have so much market share in the smart speaker category, and certainly Google has so many more installed Android devices because even people who are not using Android are using iPhone, using Google search or Google maps on their iPhone, and as we know some 40% of local searches now are driven by voice search. Do you feel like those data points are sort of helping them get better, they’re sort of ingesting more queries and therefore they can build out better AI? |
| Eric Enge: | Yes, I think there definitely is an advantage being able to leverage crawling the web. You get so much data available to you, but what comes with that is when you’re crawling websites, just because it’s published on the internet doesn’t mean- |
| Jay Baer: | Garbage in, garbage out. |
| Eric Enge: | Right, so you have to qualify that somehow and that’s a tough challenge. Google’s been working on that for years as we’ve also documented in some other studies that we do. Amazon is doing something and I can’t say what it is because I don’t know, but they’re clearly getting access to more information than just Wikipedia. You can see that based on the questions they’re answering today. |
| Jay Baer: | Yes, it’s pretty interesting. If you had to buy a personal assistant for somebody as a Mother’s Day gift or something and you’re like, “All right, I can only buy one of these,” which one would you buy? Which one would you tell somebody to purchase? |
| Eric Enge: | Well if I’m going to base it on how smart it is in answering questions, Google Assistant still does have the lead. On the other hand, I have both multiple Alexa units and multiple Google Home units at home and we use them for home control, so controlling lights and thermostats and stuff like that. Alexa is better at that, so the real nuance . . . |
| Jay Baer: | Better recipes for now, a little bit of a head start on that side of it too. |
| Eric Enge: | Yes exactly, so I think it depends on what you’re using it for. If you’re looking for home control I’d go with Alexa. If you’re looking for the raw intelligence, which is what our study focused on, then yes Google Assistant is still there. |
| Jay Baer: | One of the things you have in the study and again, it’s called Rating the Smarts of Digital Personal Assistants in 2018, you can get it on the Stone Temple website, stonetemple.com. You list sort of the question sets, not that you necessarily asked in the study, although you mention that as well, but what people ask of these assistants generally speaking. It shows that a lot of the questions today are somewhat banal. It’s what’s the weather going to be tomorrow, though I’m certainly guilty of that. I use my Alexa for that all the time even though I’ve got multiple other ways to determine the weather tomorrow, it’s just easier. Do you feel like over time as humans become more comfortable with this technology, perhaps more trusting of it, that the types of questions we ask will change? |
| Eric Enge: | I do, so we’re at very early stages and frankly for this whole space there is kind of a big thing that’s being sorted out right now, which is people getting comfortable speaking to devices, and those devices being able to have real conversations with people because people don’t always use the formulaic phrases that the device is expecting. This is a tricky process, getting that human-machine interaction to work. |
| Jay Baer: | Right because at some point it is our error because we don’t phrase the question. In fact, I probably shouldn’t record this but my wife and I are always fighting about Alexa because I know how to phrase a question because I’ve been in digital marketing and search for so long, so I can phrase the question in a way that I have a better chance of it getting returned. She doesn’t typically phrase it that way and then she gets super frustrated. “This stupid Alexa doesn’t know anything,” and I’m like, “Well but if you said it this way.” She’s like, “I don’t want to say it that way. I don’t want to have to change the way I speak because of some relational database.” It’s sort of like this whose fault is it? Is it stupid or is it us? |
| Eric Enge: | No it’s absolutely the case, and it is impacting how broad the usage is of these things. There’s no question that it’s having that impact. The whole thing about going to voice is okay, we have the decades where we learned to type the thing into Google using fewer words to have a better chance of what we want and we all get trained to do that. When we’re using voice we don’t want to have to do that, but maybe we will get trained to a certain degree and maybe they’ll get better and maybe both will happen and we’ll meet in the middle somewhere. |
| I really do think that’s very much going to happen. It’s just you have to get to the big vision of this thing and the big vision is we’re already at a point where something like 75% of the world’s internet connected devices are something other than a smartphone, PC, or a tablet. That’s an incredible amount of opportunities to interact with the internet, and if I’m going to use something like my watch here I ain’t typing it in. If I could access my Google Assistant through this thing . . . Well that’s a little unfair, it’s an iWatch but that’s beside the point. Basically, I just want to use my voice, I want it to know it’s me, and go. The technology in the personal assistants already exists, it’s already out there, they can connect from every single device that you connect to, and you’re going to be reaching the exact same personal assistant. |
| Integrated experience that can start setting up a reservation on my phone, I could finish it when I hop in my car through the internet connectivity I have there, and it’s all one session. With that level of opportunity it is just incredibly compelling and I really firmly believe that that’s the direction this is going to go. Right now there’s an awful lot of call mom, call dad, set a timer, what’s the weather, very basic stuff, but we’re getting used to it. |
| Jay Baer: | You’ve been in SEO a really long time, as have I, I feel like we’ve seen this move before. If you look at early day Google, Yahoo, Ask Jeeves, search queries, they were all two, three, four-word strings. Then over time, your average search query length got longer and more detailed and more specific as well. I feel like that’s a parallel to what we’re going to see in voice. You’re going to see more detailed, more nuanced questions. |
| Eric Enge: | I agree and just pulling your analogy out a little further or drawing it a little further, we also saw that the search engines capability to process evolved dramatically and their ability to deal with different kinds of language constructs and these kinds just changed underneath our feet. Some of the algorithms we know, things like RankBrain we’ve heard about and other algorithms like that, natural language search. They already were dealing with this even separately from the whole voice conversation. |
| Jay Baer: | Eric I wanted to ask you before we go about the Alexa skill that you have built to answer SEO questions, which I find hilarious and awesome and amazing, and I’m going to, when we’re done here, go upstairs and sit in front of my device and go to Eric Enge Stone Temple SEO school. Tell us about that process and what we can ask it, etc. |
| Eric Enge: | We have a couple hundred, maybe about 250 SEO related questions, so it might be something like what is a new index tag? How do you implement a no follow? What is a 301 redirect? Very common questions every household person wants to ask. |
| Jay Baer: | Everybody needs to know that. Most common questions are what’s the weather tomorrow and how do I do a 301? Those are the two questions. |
| Eric Enge: | We actually have built that out for Alexa. We also have one for the Google Assistant, and Alexa we have an SEO quiz where you can actually take a quiz and get your SEO skills graded. We developed it in-house. There are tools to help you do that. There’s a website you can go to called diagflow.com, which will walk you through the entire process of building what they call an actions on Google app for personal assistant. It’s not easy, it’s definitely some things to figure out but it’s not terribly hard, and when you’re done you can actually export from that code, which with very simple modifications can be used immediately on Alexa. You actually do it in one place and you get the . . . worked on for both. |
| One of the cool things about this is people are actually using them, not that it’s an enormously popular activity as we joked about a moment ago. We’re getting visibility out of it. We actually got articles written about it, some press, which was cool. In addition, on the Google Assistant they have a mode called implicit queries and if you check the box when you set up your app that you want that, somebody can ask Google a question without invoking our actions on Google app. They might just say, “How do you implement a no-follow tag?” Google might come back and said, “Stone Temple has an answer for that, do you want to hear it?” |
| Jay Baer: | Nice. |
| Eric Enge: | Yes, which is nice. It’s free visibility. |
| Jay Baer: | It’s a top-down funnel, yes I like it. |
| Eric Enge: | Yes, and I know at this point we have something like 1,000 people who have been prompted that way and accepted it at this point. |
| Jay Baer: | I mean that’s pretty strong. I mean that’s a pretty tight target. I mean no one’s asking about no follow tags on accident. |
| Eric Enge: | Right, for this particular B to B application, which is kind of what our business is. It’s actually awesome. There’s a big opportunity here because when you look an Alexa skill or an actions on Google app for Google, what you have is an ability to get on the ground floor of being an information provider to Google and to Amazon. Both cases they’re both looking for reputable sources of information to answer user questions. They’re going to have their Wikipedia relationships, Google might use crawling, Amazon is probably doing some other things to get data to people, and the people who provide these apps are another information source. They’ll draw on you if your app is getting good enough scores, however they’re scoring it. Another way to get visibility inside of the digital marketing atmosphere. |
| Jay Baer: | Do you think that being one of those information providers on the voice side would improve your topic authority on the regular web search side or have you seen the evidence of that? |
| Eric Enge: | No evidence to date, I think it’s too early for that to have happened at this point. I certainly think that some level of validation in a third party . . . Well it’s not a third party. I should say some level of validation, which is on the Google Assistant or the Alexa I think it could matter absolutely. |
| Jay Baer: | Yes it certainly can’t hurt is the way I look at it. |
| Eric Enge: | Definitely isn’t going to hurt and if you’re delivering reliable metrics and people are asking for you to give them those answers or the assistant, that’s a signal. |
| Jay Baer: | Yes I love it. Thanks so much for putting all the time and effort into this, doing the work that everybody is curious about but nobody would put that kind of effort into it. Appreciate that you and your team, Eric at Stone Temple, are willing to sit around and ask 5,000 questions times five devices, for a total ladies and gentleman, of 25,000 questions. That is a labor of love that is for sure. |
| Eric Enge: | No question about that. It was fun doing it. |
| Jay Baer: | Grab yourself a copy of Rating the Smarts of the Digital Personal Assistants 2018, super interesting findings from Eric Enge and his team at Stone Temple Consulting. My friend thanks for being here. Great to talk to you as always. |
| Eric Enge: | All right, thanks Jay. |
| Jay Baer: | See you bud. |
| Eric Enge: | Yes, bye. |
