(Image: CNET)
The Amazon Echo was the first mass-market, consumer-level device responding to voice control with general purpose functions, such as playing music. It was launched in the US in 2015.
Then, in 2016, Google released its competing device, Google Home.
Framingham, Mass.-based digital agency Stone Temple collected 5,000 questions to ask both Google Home and Amazon Alexa. Itverbally asked the devices each question and noted many different possible categories of answers.
It wanted to check if each device answered verbally, retrieved the answer from a database (such as Google’s Knowledge Graph), or sourced the answer from a third-party source (“according to Wikipedia”, etc).
The test also noted how often each device did not understand the query and when the device tried to respond to the query but got the answer wrong
The Stone Temple test was not a test of all the capabilities of both devices. It simply wanted to focus on the ability to provide direct answers to specific user questions. It tested the ability of each device to provide factual answers to questions.
Each of the devices may get its information from one of two sources in order to provide answers. It could use a knowledge database like the Knowledge Graph, or it could extract its answers from third-party websites, which Google calls “Featured Snippets.”
The company did not test other areas of functionality in the devices, such as home entertainment, the smart home controller, or the personal assistant.
It noticed some significant differences in its study. It discovered that Google Home answered more than three times the amount of questions compared to Amazon Echo (3,383 versus 1,030).
Out of all the questions that Google Home answered, 89.5 percent of the questions were answered correctly. For the Amazon Echo, 86.9 percent of answers were correct.
Out of all the questions Google Home answered, 2.3 percent of the questions it answered were incorrect versus 6.3 percent for the Amazon Echo device.
A query might have multiple possible answers to a question. Also, the the device may have responded with a joke, or it might simply get the answer wrong.
It was obvious that Google Home had the edge on knowledge due to Google search: 2,228 of Google’s answers came in the form of Featured Snippets, which means that the results were sourced from a third-party website.
However, not all of the 2,228 Featured Snippets were 100-percent correct. Only 2,122 of the correct answers for Google Home were from snippets.
Only 556 of Amazon Alexa’s correct answers came from third-party websites, 527 of which were sourced from Wikipedia.
So, Google Home appears to be the smarter device. With access to Google’s repository, this is hardly surprising. Amazon Alexa performed well, seemingly without relying on Google for its answers.
It is only a matter of time before the next voice-enabled physical device — I’m guessing a Microsoft Cortana Live perhaps — comes along and beats both of these devices.
We will have to wait and see.
VIDEO: Try these fun (and completely useless) Amazon Alexa commands
