Piper Jaffray “Street Test” Of Google vs. Siri Misses The Point
No sooner did I hit “publish” on my earlier post, Google’s “Voice Assistant” Not Quite Siri-Smart But Most People Won’t Notice, than people on Twitter started responding that I was out of my mind. A couple of people cited a Piper Jaffray study, published yesterday, comparing the accuracy and breadth of Siri “search results” vs. […]
No sooner did I hit “publish” on my earlier post, Google’s “Voice Assistant” Not Quite Siri-Smart But Most People Won’t Notice, than people on Twitter started responding that I was out of my mind. A couple of people cited a Piper Jaffray study, published yesterday, comparing the accuracy and breadth of Siri “search results” vs. Google in a “street test” in Minneapolis.
Google Gets a “B+” Siri Gets a “D”
In short Google was graded with a “B+” in terms of accuracy, while Siri got a “D.” It’s really an “apples to oranges” comparison, however. Let’s take a look at the methodology.
Piper Jaffray analyst Gene Munster and his minions asked an iPhone (Siri) 1,600 questions, half of which were in a room and half of which were on a noisy street. The same questions were input into Google. However it’s not clear whether this was in Safari, on a Google iPhone app or on an Android device. It’s also not clear whether Google voice search was used.
Fortune summarized many of the findings:
- Google understands 100% of the questions (not surprisingly, since they are keyed in)
- Google replies accurately 86% of the time
- Siri comprehends 83% of queries in noisy conditions, 89% in a quiet room
- Siri answers accurately 62% of the time on the street and 68% in a quiet room.
According to the test the following are the sources of Siri’s information: 60% of answers come from the default search engine (Google), 20% come from Yelp, 14% from WolframAlpha, 4% from Yahoo (e.g., Weather, Stocks) and 2% from Wikipedia.
Siri Isn’t a Search Engine
The incorrect assumption and fundamental conceptual error that the test makes is that Siri is a search engine and should be judged as such. It’s not. And most people don’t use Siri as a substitute for Google at present.
Most people initiate calls, send texts and emails using Siri. Searching the web is a minority use case on Siri. People use apps or Google on the iPhone to “search” the mobile internet.
However, I would agree with another of Piper Jaffray’s implied assertions: the boundaries between what Siri does and search are starting to break down, especially as Apple adds more structured data to Siri’s knowledge base. It goes the other way too, as Google seeks to emulate the Siri “assistant” capability on Android devices.
Siri can act as a voice front end for Google, simply by directing it to “search the web for X,Y,Z.” And, as mentioned, Siri defaults to Google (or other designated search engine) when it doesn’t have a structured database to draw from.
Google Is a Search Engine
By contrast, Google is a search engine and has a massive corpus of data from which to draw — still mostly in the form of links to third party documents and sites. Apple simply doesn’t have the same data and information to make available to Siri.
Siri’s mission isn’t to “organize all the world’s information.” Rather Siri has a much more narrow range of functions as an “intelligent assistant.” True, one of those functions is to deliver information in certain circumstances. But nobody involved with Siri would likely argue that it’s a substitute for Google in all but a handful of situations.
That’s mainly because Siri has a quite limited range of datasets that it’s working with, which are admittedly being expanded in iOS 6.
Results Are No Surprise
Because Siri can only access limited data silos — although its primary functions don’t involve retrieving information on the web — it should come as no surprise that Siri’s “ass got kicked” by Google.
A more “apples to apples” test would be to: 1) compare the capabilities of Siri/Nuance and Google voice recognition and task completion and/or 2) compare Google and Siri in categories where Siri has access to a structured database.
It still might be that Google would do a better job of retrieving relevant information than Siri, but it would be a more accurate reflection of their relative capabilities. For a discussion of how Google’s new speech-based “assistant” performs vs. Siri, see my earlier article.