Last year, Cortana performed poorly on Loop Ventures’ yearly IQ test, answering just 48% of answers correctly and coming in last place. This time, there’s good and bad news.
Last year, Siri answered 66.1% correctly and Google Assistant 74.8%. A test in July 2018 introduced Alexa, which scored 61.4% against new scores of 52.4%, 78.5%, and 85.5% from Cortana, Siri, and Google.
This year, Google Assistant has jumped to 87.9%, Siri to 74.6%, and Alexa to 72.3%. Cortana’s closest competitor is still almost 10% ahead, and Google is almost 25.
A Shift in Focus
For those who have been following the story of Cortana, this isn’t entirely surprising. While the other three players focus on smart speaker functionality, Microsoft is looking to take its digital assistant in a different direction. Rather than focusing on assistance, it wants to shape it into a productivity aid.
This makes a little more sense when you see the breakdown by query category. Cortana’s biggest weakness was in commerce, and it’s main strength in information, where it edged ahead of Apple.
Interestingly, Google dominated in the commerce category, despite Amazon’s roots in that area. According to the researchers, this is because their questions were skewed towards product research, rather than direct purchasing.
Obviously, each consumer puts a different weight on an assistant’s ability to do certain tasks. Some use them exclusively for information gathering or music playing, while others use a whole host.
However, it seems that Cortana is below average at best in most categories, and it’s likely the gap will widen as Microsoft focuses on those even less. Thankfully, Microsoft’s speech recognition is still better than Alexa, with Cortana understanding 99.4% of questions versus Alexa’s 99%, Siri’s 99.6%, and Google’s 100%.