Will [Not Provided] Ever Reach 100% In Web Analytics?

With the news today that Chrome is moving to secure search, marketers may be wondering about the larger picture of using search data in Web analytics.

The searches people do that lead them to a site is valuable data to an organization. What other data source do we have that gives us direct access to our customers’ wants and needs? Sure, we can ask them, but customers can lie. Or choose not to answer. By looking at exactly what visitors have searched for, we can learn a lot about who our customers are, what they really want, and if we’re providing it to them.

Search marketers have valued this data for a long time, but it’s useful for many other parts of the organization as well, from traditional marketers to product managers, to support. Shouldn’t everyone want to understand more about their audiences? I think this data is so great, I wrote an entire book about it: Marketing in the Age of Google.

But now, this information is going away. Sort of. Should we freak out? Is there anything we can do to get this data back?

Why Are We Losing Search Data?

First, let’s recap where we are. When someone types words into a search engine, then clicks on a page in the search results, the URL that “refers” the visitor to that page typically includes what the visitor searched for. You can see the referring URL in the site’s server logs, and if you have a Web analytics package installed, it will parse what the visitor searched for from that referring URL and provide it in a handy report.

Just over a year ago, Google stopped including the search term in the referring URL in certain cases. Instead, the referring URL just looked like www.google.com. Web analytics programs didn’t have a way of knowing what the visitor searched for, just that that person came from Google. So Web analytics programs count the visit as Google organic search, but list the search term as “not provided.”

What are these certain cases?

• May 2010 – Google launches encrypted search at a separate URL from www.google.com; you have to go to encrypted search directly to keep your search terms from being included in the referrer.
• October 2011: Searchers on www.google.com who are logged into a Google account are routed through Google secure search. At the time, Google estimated this would impact less than 10% of searches.
• March 2012: Searchers on other Google properties beyond www.google.com (such as international domains) are routed through Google secure search.
• July 2012: Firefox 14 launches, which uses Google secure search for all searches.
• September 2012: Safari in iOS6 begins using Google secure search. Because Google handles secure search differently on mobile devices, this traffic doesn’t show a referrer at all (even that it came from www.google.com), so this traffic shows up in Web analytics programs as direct traffic.
• January 2013: The next version of Chrome (version 25 — not yet officially released, but available for beta and developer use) uses Google secure search for those searching from the address bar. (Those not logged into a Google Account searching from www.google.com will not be routed through secure search.)

Google discussed this latest addition of secure search in the Chromium blog today.

What Percentage Of Traffic Is Obscured By Secure Search?

So, what percentage of Google search traffic comes through as “not provided” and will that total ever reach 100%?

As you can see, how much search data is obscured for your site depends on your audience. Do they tend to be logged into Google when they search? Do they use Firefox or Chrome? Some studies looking at this data represent the shift as a steady increase, and if you follow the line up and to the right, it just keeps going up. But this isn’t a gradual increase, it’s an abrupt change as each source of data shifts to secure search, and then stays flat until the next shift.

For instance, over the holidays, a large percentage of traffic from iPhone users shifted into the “direct” bucket because a lot of people got new phones with iOS6 preloaded.

It’s likely that more data sources will shift to secure search, so no doubt the numbers will increase.

At my company, Nine By Blue, we took a sample set of sites from different industries and with vastly different audiences, to see where things stand right now. No surprise, we found that the percentage of searches reported as not provided has gone up over time. We looked at three dates: December 2011, June 2012, and December 2012 and we segmented US sites from international (English) sites, as secure search was launched initially only the the US:

Below you can see the breakdown for each site as a comparison between December 2011 and December 2012:

For one site (which has a tech audience), 54% of Google organic search terms are reported as not provided, although most sites (particularly the more consumer-oriented) have a much lower percentage of search terms showing up this way.

Should We Freak Out? Can We Get This Data Back?

You should not freak out. Yes, it’s great data and tough to lose, and yes, the percentage obscured is likely to increase, but we will continue to have enough data to make the same decisions and determine the same actions.

One of the most important things we’re losing is the ability to see trends. I always recommend looking at categories of search queries, rather than just at individual terms, but if you look at, say, branded searches, or “how to” searches over time, the trend will likely look as though it’s going down, when in reality, a substantial amount of that traffic has simply moved into the not provided bucket.

Using Google Webmaster Tools Data

One way to get the data back is to look beyond your Web analytics platform. The key issue here is that the search term is no longer in the referrer, but Google still stores the search terms in aggregate in their query logs. They make this data available to you in Google Webmaster Tools. (Disclosure: I helped build webmaster tools originally when I worked at Google.)

You can see the top 2,000 search terms for your site for the last 90 days. Google points out that for 98% of sites, webmaster tools shows 100% of search terms. Of course, the remaining 2% are large sites which are likely the most interested in this data, but even for those sites that don’t see all the search data in webmaster tools, the trends tend to be accurate. So, you can still get a very good sense of what your audience is looking for, and in what topic areas you’re beginning to falter.

I think webmaster tools data is so valuable (see disclosure above about my bias; heh) that my company has built search analytics software that provides insights on top of this data. Blueprint (Disclosure: I sell this product) segments webmaster tools query data into logical topical categories and stores it over time (so you aren’t limited by Google’s 90 day restriction).

Webmaster tools gives you a different view of this data than your Web analytics program does, showing you impression data, ranking, and click through rate for each query. By looking at search queries through this lens, you can tell not only what your audience is interested in, but if traffic drops are due to ranking issues or simply are due to seasonality.

Below, for instance, you can compare these data points for 2012 for branded traffic. Ranking stayed fairly consistent throughout the year, but traffic changes were due in part to search volume changes (because of seasonality) and changes in click through rate from the search results (the site added rel=author markup in September).

Using Paid Search Data

As I mentioned above, it’s only the referral that no longer includes the search term. So, not only do Google’s query logs still have the data available (and used as the source for webmaster tools), but Adwords has this data available as well. If you use paid search, you can get similar audience insights as you can from organic search data. This doesn’t help you if you don’t use paid search, of course.

Using Data From Other Search Engines

Yes, I realize Google likely brings in most of your search traffic. But if what you’re looking for is better understanding of your audience — what they are looking for and if you are providing it — don’t forget that Bing (and Yahoo and the rest) are still sending the referrer in all cases.

Using Web Analytics Data In New Ways

How to Turn (Not Provided) Into Useful, Actionable Data talks about different views of Web analtytics data beyond lists of search terms.

What Does the Future Hold?

Since clearly, more sources are moving to secure search (and it’s possible other search engines may one day follow), it’s important to start making plans now. Take a look at your processes.

Can you get to the same conclusions and action plans with a subset of query data? Can you use webmaster tools and other data to approximate what you really need to make decisions? You can augment audience data with keyword research. You can augment on site behavioral data by looking at a page-based, rather than query-based view. The key is to really think through what you need to accomplish and look at what other pieces of data will help you accomplish that rather than focus on the specific data that’s lost.