How Google Now, Siri & Cortana Predict What You Want

Google, Apple and Microsoft all have agents that want to be your personal assistant. But how well Google Now, Apple’s Siri and Microsoft’s Cortana can predict your needs depends on how much you want to share, how wedded to particular platforms you want to be and, in some cases, how much you actively want to help make those predictions happen.

Going Under The Hood

Ferreting out how these agents work is a challenge. The companies all have pages that describe Google Now, Apple Siri and Microsoft Cortana. But those are generally focused on what they can do, not how exactly they do it.

There are various help and privacy pages, but those aren’t always helpful. Google Now information is, in part, spread out across a page for the Google App for iOS and another area for Android. Cortana has a nice single page on privacy but still lacks specifics on some points. Apple’s privacy page notes that it doesn’t mine mail for ads but neglects to mention mail is used for Siri predictions.

The “how it works” information more often comes out in developer documentation or in association with press interviews, events and announcements. I’ve used such information as a primary source. I also sent everything I’m summarizing in this article to each of the major companies for review, in order to ensure it’s as accurate as I can make it.

Below, a deeper look at how these tools form their predictions and some concepts that are involved. At the end, you’ll also find a comparison chart covering key areas.

Predictions & The Data They Depend On

The purpose of these apps is to help you generally. The predictive aspect that’s one part of them is to help you even before you know you need help. When that works, it’s incredibly helpful.

For example, Google Now routinely alerts me when I should leave my home or hotel to the airport to catch my flights. I love it. But for this magic to happen, Google Now needs to understand:

1. That I have a flight booked
2. What airport I’m booked out of
3. My current location

Google Now figures this stuff out consistently without me doing any work. Cortana’s also done it for me, though less predictably than Google. The latest version of Siri with its “proactive” suggestions is promising the same.

If you want to see more specific examples of predictions these tools can do, please see our other articles:

• The Amazing “Google Now” — When Google Searches Before You Think To
• Life With Cortana, Microsoft’s Predictive Search Challenger To Google Now & Siri
• With iOS9, Apple’s Siri & Spotlight Search Get Smarter

The purpose of this article is to delve more under the hood. How do these tools learn about our flights to make such predictions, along with other things they might suggest? With our permission, they learn by tapping into various sources of information.

We’ll explore those in a moment. But first, let’s cover the important issues of passive versus active help and cloud versus device-based profiles.

Passive Vs. Active Assistance

A good assistant, in the virtual or real world, would seem to know exactly what you need before you even ask. It wouldn’t require you to explain anything. It would just know. That’s what I call “passive” assistance. The tool anticipates your desires without your having to teach it things.

Among the digital assistants I’m covering here, Google Now is the most mature and the most passive. Google Now rarely asks you to help it to create predictions. It figures out what it thinks you want without requiring you to do much, if any, work. The trade-off for this ease is embedding yourself in Google’s services and trusting in the company.

Microsoft’s Cortana and Apple’s Siri require more active work. Yes, both will try to automatically detect things from sources like your email, but what they use that for is more limited compared to Google. News articles they surface are also generally limited to categories you actively select, rather than things they passively detect you might be interested in.

Cloud Versus Device Profiles

For all these assistants to help, they need to build up a profile about you. Some store your profile in the cloud, which means you can go from device to device and get the same predictions. A device-specific profile means your phone, tablet and laptop might not all understand you in the same way.

Google Now is fully cloud-based. Whatever it learns on any device or in any way you interact with Google all goes into a common profile in the cloud. If you were to switch from an Android phone to iOS, where Google Now is part of the Google Search App, all your history used for recommendations would be carried over seamlessly.

Cortana also uses a cloud-based profile, so you get the same seamless transition as Google Now offers. However, as I’ll explain in a bit, the sources that Cortana draws from are not always cloud-based themselves.

Siri is device-based. Whatever Siri learns about you to form recommendations on one device is not used to help you on another. This means if you have an iPhone and an iPad, Siri might not work the same way on both, especially in terms of things that use location history. It also means that if you move to a new device and start fresh, without using a backup of a previous device, you’re starting over with Siri.

Now let’s talk about various data sources these tools use.

Email

Email is a fantastic source for an assistant to figure out what you need. Of the assistants, Google Now does the most with email. If you have a Gmail account, it will automatically scan what’s in your email — in the cloud — so that it can detect things like flights, package tracking codes and more, in order to give you predictive information requiring little to no active work. But if you don’t use Gmail — or don’t allow Google Now to tap into it — you won’t get as much out of Google Now.

Cortana also uses email, but in a more limited fashion than Google. It only reads email contained within the Mail app on Windows devices. If you read your mail on the Web before it downloads to the Mail app, or through a program other than Mail, then Cortana misses all that. Cortana on Android also can’t tap into your mail.

Siri also uses email on your device to help with suggestions. For example, if it spots a calendar invite in your email, it will add that to your calendar and, in turn, use that to generate a reminder that can take into account traffic conditions. If you get a call from someone not in your contacts, but you have information about them in your email — such as their phone number in a signature — that can be used to identify them.

By the way, Gmail, Windows Mail and Apple Mail are able to import email from other services. If you use Gmail, you can also read that mail through the Windows Mail app or the Apple Mail app, which in turn helps Cortana and Siri. Similarly, if you forward your Outlook or Apple email to Gmail, Google Now can learn of it that way.

Search

What you search for is another important source for digital assistants. Google Now regularly suggests content I might be interested in, based on my search activity. It’s been very useful.

Google Now taps into your search history on Google, whether you’re searching using Chrome, the Google Search app or other means where you’re signed in and Google knows who you are, as a Google Now user.

Cortana does the same, only using your search history on Bing, whether you use Internet Explorer or Edge, the Bing search app or some other logged-in mechanism.

Siri doesn’t use search at all, to my understanding. That means Siri’s much more limited in what it can recommend. It also means for those concerned about privacy, Siri knows less about you.

iOS9 does have many helpful additions that have been added to both Siri and Spotlight when you actively search. But those are much different from using your search history to generate predictive suggestions.

Browsing

Where you go on the Web can be a useful source for predictive digital assistants. Google Now looks at your browsing history within Chrome, if you’re signed in. It will also understand browsing history if you’re somehow logged into other Google tools that might record this.

Cortana is like Google, also using browsing history but only based on what you do in Edge. If you still use Internet Explorer, browsing history there doesn’t help Cortana.

Siri doesn’t use browsing history at all. It’s again how Siri is perhaps limited in what it knows to help you but also provides more privacy, if you’re concerned about that.

Calendar & News

What’s on your calendar is an age-old source used by assistants long before they became digital, and calendars remain useful.

Google Now looks at whatever’s on your Google Calendar, regardless of the source. In other words, if you sync an outside calendar into Google Calendar, all those events are used.

Cortana looks at whatever is on the Windows Calendar app. The events can be pulled into that app from another provider, such as Google, and still help Cortana. But only what’s in the Windows Calendar app is used. If you use Cortana for Android, it’s going to miss events stored in other calendar programs.

Siri will look at whatever is on your Apple Calendar. Like Cortana, it can use those events even if they flow in from other sources like Google.

An important note here is that while Siri is device-specific, the help it does with your calendar may go up to the cloud and down to other devices. For example, Siri saw that I have a flight confirmation email in my Apple Mail app. It then automatically created an entry on my calendar for that, on my iPhone. Because my calendar syncs to the cloud, that device-specific assistance flowed out to my iPad.

As for news, all the digital assistants will suggest stories you may wish to read. Google depends on what you like in Google News or the associated app, as well as what you search for. That makes Google Now the assistant most likely to suggest the broadest topics. Cortana taps into what you deliberately select or save as part of its Cortana Notebook. Apple depends on what you do within its Apple News app.

Location

One of the most powerful sources for predictive suggestions is your location. When Google Now saw I was in Iceland earlier this year, it proactively presented me with the local exchange rate, things to see and a few words of greetings. All these tools are also designed to understand where you work or live, in order to make location-specific suggestions that relate to those places, like traffic times.

Google Now understands your location based on what’s logged when you use Google apps or devices. An Android phone may log a location in order to help it be found, if lost. That also helps Google Now. Using Chrome on a computer when logged into Google will send your location. Using the Google Search App on iOS sends a location. All of this means that Google Now knows where you are better than any other assistant. If you’re concerned about privacy, that may worry you. If so, you can opt out on an account-wide basis. But leaving it on means Google Now can help you the most.

Like Google, Cortana is logging your location in a variety of ways. Using Cortana itself sends your location. Using another Microsoft product when logged in may send your location in a way for Cortana to use. Location history is also mandatory to use Cortana.

Siri is unique in only using the location history on your device. If you take your iPhone out to many places, Siri understands this and can use that location history to suggest things. But that history is not synced with a profile in the cloud. That means Siri on that iPad you never take from your house will know less about your movements. Again, that’s helpful for those concerned about privacy, but it also means Siri can’t use location as much to predict things.

Apps

Google Now is the most mature of the tools in using information from third-party apps to help form some of the suggestions it provides.

Siri will try to predict the apps you use on your phone. The most cited example is that if you plug in your headphone, it’ll suggest listening to whatever you last heard, like songs in Apple Music or a podcast.

Siri will also suggest the apps it thinks you’re most likely to use when you pull down to search or swipe to the left  from your home screen, in iOS9.

Cortana seems to do little with third-party apps, at the moment.

Conclusion

Time to wrap things up. Above, you’ll find a comparison infographic that covers the major sources that each of these tools use. Thanks to Kyle Pouliot on the Search Engine Land team for pulling that together.

Having used them all, I would say Google Now is the assistant that will most easily give you a robust range of predictions, assuming you’re happy to embrace the Google ecosystem.

If you don’t use Google services, Cortana is an interesting alternative, especially now that it’s out for Android. Just don’t expect it’ll do as much as Google Now.

If you love privacy, Apple’s Siri is appealing. But the trade-off for that privacy is going to be far fewer predictions, having to teach Siri all over again if you switch devices or have to fully reset an existing one.