Piper Jaffray “Street Test” Of Google vs. Siri Misses The Point

No sooner did I hit “publish” on my earlier post, Google’s “Voice Assistant” Not Quite Siri-Smart But Most People Won’t Notice, than people on Twitter started responding that I was out of my mind. A couple of people cited a Piper Jaffray study, published yesterday, comparing the accuracy and breadth of Siri “search results” vs. Google in a “street test” in Minneapolis.

Google Gets a “B+” Siri Gets a “D”

In short Google was graded with a “B+” in terms of accuracy, while Siri got a “D.” It’s really an “apples to oranges” comparison, however. Let’s take a look at the methodology.

Piper Jaffray analyst Gene Munster and his minions asked an iPhone (Siri) 1,600 questions, half of which were in a room and half of which were on a noisy street. The same questions were input into Google. However it’s not clear whether this was in Safari, on a Google iPhone app or on an Android device. It’s also not clear whether Google voice search was used.

Fortune summarized many of the findings:

  • Google understands 100% of the questions (not surprisingly, since they are keyed in)
  • Google replies accurately 86% of the time
  • Siri comprehends 83% of queries in noisy conditions, 89% in a quiet room
  • Siri answers accurately 62% of the time on the street and 68% in a quiet room.

According to the test the following are the sources of Siri’s information: 60% of answers come from the default search engine (Google), 20% come from Yelp, 14% from WolframAlpha, 4% from Yahoo (e.g., Weather, Stocks) and 2% from Wikipedia.

Siri Isn’t a Search Engine

The incorrect assumption and fundamental conceptual error that the test makes is that Siri is a search engine and should be judged as such. It’s not. And most people don’t use Siri as a substitute for Google at present.

Most people initiate calls, send texts and emails using Siri. Searching the web is a minority use case on Siri. People use apps or Google on the iPhone to “search” the mobile internet.

However, I would agree with another of Piper Jaffray’s implied assertions: the boundaries between what Siri does and search are starting to break down, especially as Apple adds more structured data to Siri’s knowledge base. It goes the other way too, as Google seeks to emulate the Siri “assistant” capability on Android devices.

Siri can act as a voice front end for Google, simply by directing it to “search the web for X,Y,Z.” And, as mentioned, Siri defaults to Google (or other designated search engine) when it doesn’t have a structured database to draw from.

Google Is a Search Engine

By contrast, Google is a search engine and has a massive corpus of data from which to draw — still mostly in the form of links to third party documents and sites. Apple simply doesn’t have the same data and information to make available to Siri.

Siri’s mission isn’t to “organize all the world’s information.” Rather Siri has a much more narrow range of functions as an “intelligent assistant.” True, one of those functions is to deliver information in certain circumstances. But nobody involved with Siri would likely argue that it’s a substitute for Google in all but a handful of situations.

That’s mainly because Siri has a quite limited range of datasets that it’s working with, which are admittedly being expanded in iOS 6.

Results Are No Surprise

Because Siri can only access limited data silos — although its primary functions don’t involve retrieving information on the web — it should come as no surprise that Siri’s “ass got kicked” by Google.

A more “apples to apples” test would be to: 1) compare the capabilities of Siri/Nuance and Google voice recognition and task completion and/or 2) compare Google and Siri in categories where Siri has access to a structured database.

It still might be that Google would do a better job of retrieving relevant information than Siri, but it would be a more accurate reflection of their relative capabilities. For a discussion of how Google’s new speech-based “assistant” performs vs. Siri, see my earlier article.

Related Topics: Apple: Siri | Channel: Mobile | Google: Mobile | Google: Voice Search

Sponsored


About The Author: is a Contributing Editor at Search Engine Land. He writes a personal blog Screenwerk, about SoLoMo issues and connecting the dots between online and offline. He also posts at Internet2Go, which is focused on the mobile Internet. Follow him @gsterling.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://twitter.com/thingsiseeonnet sameermal

    BOTTOMLINE….get an android.!

  • http://www.dfwrealties.com/ Omni – Dallas Real Estate

    Greg, this is a nice post. Google products are usually good. Android is a nice phone.

  • http://twitter.com/gsterling Greg Sterling

    Yes, Android is good and Google’s Jelly Bean innovations are very good. 

  • Grimmjow

    Why should they do the test according to Siri’s strengths? Google is not an assistant, its a search engine. They did not even show at as an assistant in the keynote.

  • http://www.seo-theory.com/ Michael Martinez

    You haven’t made a case for why Siri isn’t a search engine.  You might as well argue that IxQuick or Dogpile are not search engines.

  • http://twitter.com/theandrews11 JOe

    Funny thing is.. everyone says how inaccurate siri is, but last I checked.. Siri is in BETA phase! i.e. TESTING  People are idiots..

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide