Will [Not Provided] Ever Reach 100% In Web Analytics?

google-not-provided-200pxWith the news today that Chrome is moving to secure search, marketers may be wondering about the larger picture of using search data in Web analytics.

The searches people do that lead them to a site is valuable data to an organization. What other data source do we have that gives us direct access to our customers’ wants and needs? Sure, we can ask them, but customers can lie. Or choose not to answer. By looking at exactly what visitors have searched for, we can learn a lot about who our customers are, what they really want, and if we’re providing it to them.

Search marketers have valued this data for a long time, but it’s useful for many other parts of the organization as well, from traditional marketers to product managers, to support. Shouldn’t everyone want to understand more about their audiences? I think this data is so great, I wrote an entire book about it: Marketing in the Age of Google.

But now, this information is going away. Sort of. Should we freak out? Is there anything we can do to get this data back?

Why Are We Losing Search Data?

First, let’s recap where we are. When someone types words into a search engine, then clicks on a page in the search results, the URL that “refers” the visitor to that page typically includes what the visitor searched for. You can see the referring URL in the site’s server logs, and if you have a Web analytics package installed, it will parse what the visitor searched for from that referring URL and provide it in a handy report.

Just over a year ago, Google stopped including the search term in the referring URL in certain cases. Instead, the referring URL just looked like www.google.com. Web analytics programs didn’t have a way of knowing what the visitor searched for, just that that person came from Google. So Web analytics programs count the visit as Google organic search, but list the search term as “not provided.”

What are these certain cases?

  • May 2010 – Google launches encrypted search at a separate URL from www.google.com; you have to go to encrypted search directly to keep your search terms from being included in the referrer.
  • October 2011: Searchers on www.google.com who are logged into a Google account are routed through Google secure search. At the time, Google estimated this would impact less than 10% of searches.
  • March 2012: Searchers on other Google properties beyond www.google.com (such as international domains) are routed through Google secure search.
  • July 2012: Firefox 14 launches, which uses Google secure search for all searches.
  • September 2012: Safari in iOS6 begins using Google secure search. Because Google handles secure search differently on mobile devices, this traffic doesn’t show a referrer at all (even that it came from www.google.com), so this traffic shows up in Web analytics programs as direct traffic.
  • January 2013: The next version of Chrome (version 25 — not yet officially released, but available for beta and developer use) uses Google secure search for those searching from the address bar. (Those not logged into a Google Account searching from www.google.com will not be routed through secure search.)

Google discussed this latest addition of secure search in the Chromium blog today.

What Percentage Of Traffic Is Obscured By Secure Search?

So, what percentage of Google search traffic comes through as “not provided” and will that total ever reach 100%?

As you can see, how much search data is obscured for your site depends on your audience. Do they tend to be logged into Google when they search? Do they use Firefox or Chrome? Some studies looking at this data represent the shift as a steady increase, and if you follow the line up and to the right, it just keeps going up. But this isn’t a gradual increase, it’s an abrupt change as each source of data shifts to secure search, and then stays flat until the next shift.

For instance, over the holidays, a large percentage of traffic from iPhone users shifted into the “direct” bucket because a lot of people got new phones with iOS6 preloaded.

It’s likely that more data sources will shift to secure search, so no doubt the numbers will increase.

At my company, Nine By Blue, we took a sample set of sites from different industries and with vastly different audiences, to see where things stand right now. No surprise, we found that the percentage of searches reported as not provided has gone up over time. We looked at three dates: December 2011, June 2012, and December 2012 and we segmented US sites from international (English) sites, as secure search was launched initially only the the US:

Not Provided Over Time

Below you can see the breakdown for each site as a comparison between December 2011 and December 2012:

not provided

For one site (which has a tech audience), 54% of Google organic search terms are reported as not provided, although most sites (particularly the more consumer-oriented) have a much lower percentage of search terms showing up this way.

Should We Freak Out? Can We Get This Data Back?

You should not freak out. Yes, it’s great data and tough to lose, and yes, the percentage obscured is likely to increase, but we will continue to have enough data to make the same decisions and determine the same actions.

One of the most important things we’re losing is the ability to see trends. I always recommend looking at categories of search queries, rather than just at individual terms, but if you look at, say, branded searches, or “how to” searches over time, the trend will likely look as though it’s going down, when in reality, a substantial amount of that traffic has simply moved into the not provided bucket.

Using Google Webmaster Tools Data

One way to get the data back is to look beyond your Web analytics platform. The key issue here is that the search term is no longer in the referrer, but Google still stores the search terms in aggregate in their query logs. They make this data available to you in Google Webmaster Tools. (Disclosure: I helped build webmaster tools originally when I worked at Google.)

You can see the top 2,000 search terms for your site for the last 90 days. Google points out that for 98% of sites, webmaster tools shows 100% of search terms. Of course, the remaining 2% are large sites which are likely the most interested in this data, but even for those sites that don’t see all the search data in webmaster tools, the trends tend to be accurate. So, you can still get a very good sense of what your audience is looking for, and in what topic areas you’re beginning to falter.

I think webmaster tools data is so valuable (see disclosure above about my bias; heh) that my company has built search analytics software that provides insights on top of this data. Blueprint (Disclosure: I sell this product) segments webmaster tools query data into logical topical categories and stores it over time (so you aren’t limited by Google’s 90 day restriction).

Webmaster tools gives you a different view of this data than your Web analytics program does, showing you impression data, ranking, and click through rate for each query. By looking at search queries through this lens, you can tell not only what your audience is interested in, but if traffic drops are due to ranking issues or simply are due to seasonality.

Below, for instance, you can compare these data points for 2012 for branded traffic. Ranking stayed fairly consistent throughout the year, but traffic changes were due in part to search volume changes (because of seasonality) and changes in click through rate from the search results (the site added rel=author markup in September).

Blueprint

Using Paid Search Data

As I mentioned above, it’s only the referral that no longer includes the search term. So, not only do Google’s query logs still have the data available (and used as the source for webmaster tools), but Adwords has this data available as well. If you use paid search, you can get similar audience insights as you can from organic search data. This doesn’t help you if you don’t use paid search, of course.

Using Data From Other Search Engines

Yes, I realize Google likely brings in most of your search traffic. But if what you’re looking for is better understanding of your audience — what they are looking for and if you are providing it — don’t forget that Bing (and Yahoo and the rest) are still sending the referrer in all cases.

Using Web Analytics Data In New Ways

How to Turn (Not Provided) Into Useful, Actionable Data talks about different views of Web analtytics data beyond lists of search terms.

What Does the Future Hold?

Since clearly, more sources are moving to secure search (and it’s possible other search engines may one day follow), it’s important to start making plans now. Take a look at your processes.

Can you get to the same conclusions and action plans with a subset of query data? Can you use webmaster tools and other data to approximate what you really need to make decisions? You can augment audience data with keyword research. You can augment on site behavioral data by looking at a page-based, rather than query-based view. The key is to really think through what you need to accomplish and look at what other pieces of data will help you accomplish that rather than focus on the specific data that’s lost.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Analytics | Features: Analysis | Google: Analytics | Google: Chrome | Google: Privacy | Google: SEO | Google: Webmaster Central | Top News

Sponsored


About The Author: is a Contributing Editor at Search Engine Land. She built Google Webmaster Central and went on to found software and consulting company Nine By Blue and create Blueprint Search Analytics< which she later sold. Her book, Marketing in the Age of Google, (updated edition, May 2012) provides a foundation for incorporating search strategy into organizations of all levels. Follow her on Twitter at @vanessafox.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • RyanMJones

    I predict that not only will Google’s [not provided] eventually hit 100%, but I think within 5 to 10 years, newer browsers will ditch the HTTP_REFERER altogether. It constantly comes up as a discussion in places like the FF developer forums, and there’s a LOT of support on both sides. It’s an interesting issue to watch.

  • http://twitter.com/bwhalley Brian Whalley

    I think it’s not that far off for some sites. My site, which receives a pretty solid amount of traffic from Google, is at 90% of google-provided search traffic being (not provided) for the last 30 days. It’s incredibly frustrating. Our audience is extremely technical and Google-saavy, and as a result we receive almost no keyword data at this point.

  • http://twitter.com/YoungbloodJoe Joe Youngblood

    Stop using Google+, stop promoting it, stop telling clients to use it, uninstall author rank tags. done.

  • http://raventools.com Jon Henshaw

    I predict it will hit 101%. You heard it here first!

  • David Rekuc

    I hate this double standard. Either its a privacy concern or its not. If you’re going to hide referring queries for natural search, it ought to be the same for paid.

    I understand this is Google trying to self regulate and avoid privacy issues. Self regulation seems to be fine as long as its convenient and doesn’t negatively effect an advertiser’s ad spend.

    Not to mention this effects search retargeting data, which can be seen as a competitor in the display space.

    I hate being cynical about these kinds of things, but if we’re going to regulate ourselves it ought to be standardized. Natural and paid get the same treatment. I haven’t heard a good argument about why paid search data is any more secure or anonymous than natural data.

  • http://twitter.com/anilopez Ani Lopez

    Unless Google decides to do a better integration between WMT and Analytics, and I mean all the way through to conversions or dollar value for the WMT data (unlikely to happen), it seems pretty absurd to say “there is no reason to freak out” for having a high volume of (not provided) and every SEO knows that.

    “We still have WMT data!” yes, a great argument when you are promoting something like Blueprint that seems, for what I see at the product tour, a WMT nothing more than a nice interface on steroids for WMT but it is going to suffer from the exact same issue that WMT suffers now, isolation from conversion metrics and such.

    I don’t recall WMT to have any API for traffic data extraction. Or Vanessa has figured out a better way to do it or her tool is using same old https://code.google.com/p/php-webmaster-tools-downloads/ or such.

  • http://twitter.com/seocharlie Carlos Chacón l SEO

    Data is importante, but user experience more. So, once we create unique landing pages with unique content, everything else will be less and less relevant.

    GWT is a great place to still seeing some data, at least now. Don´t forget about Google AdWords. The data right there still untouchable.

  • http://lostpr.es/ David Iwanow

    I thought 58% from my site and 47% from my blog was bad enough currently…

  • http://lostpr.es/ David Iwanow

    I think the one issue that is getting harder to track how branded traffic is changing against last year as even if you listen to Google and use GWT data that only goes back 90 days… so if they are serious about working to improve transparency they would expand GWT data beyond 90 days and offer the ability to filter out Google Products and see just Clicks or Impressions data….

  • http://lostpr.es/ David Iwanow

    i’m going with 102% due to recalculations due to multi-device use….

  • http://about.me/jemindesai jemin

    Google is an Evil and Google+ is big Ash…h.. Is Google have any rights to talk about Privacy? And if it’s really about privacy then why it’s still gathering our data, Google Map, our location and other things. Its time to avoid Google Now totally use other search engines so Bluddy Matt cutts and other Google Management people come to on road then we will see that how they react about all stupid things.

  • http://www.moonfiller.com/ Taran

    Data is important, but user experience more. So, once we create unique landing pages with unique content, everything else will be less and less relevant.GWT is a great place to still seeing some data.

  • http://twitter.com/localoptimizer Dave

    Vanessa: I run local smb sites. I run extensive ppc on all of them. Its 24/7 and typically the radii extend somewhat beyond our reasonable catch areas.

    Adwords impression data and webmaster tool data do not sync. They aren’t even close. webmaster tools doesn’t seem to have any grasp of geography. For local smb sites I don’t care how many worldwide impressions there are for “deli” or bicycles, or pizza. I only care about the impressions in my geo territory for those phrases and phrases with geo modifiers. Even the webmaster tools for phrases w/ geo modifiers are way off. I’ve run some of those phrases both regionally and nationally and run them now w/ relevance for the geo phrase regardless of the location of the user’s IP.

    To me webmaster data is like google places dashboard data, manipulated by google to present “feel good” data …but absolutely worthless in substance and pretty unreliable in detail.

    any other suggestions.

  • Pat Grady

    Shitake, as a paid search guy, I’m going to have more people asking me for data, and how to pull it, and what it means. and reporting, and how to use regex, and how… yuck.

  • http://twitter.com/JSolis102 Jonathan

    This is definitely upsetting. Maybe they’ll offer it to people who pay. After all it is valuable data (not that I agree with they’re doing)

  • Guy Hadas

    I am wondering whether this is another try from Google to increase the use in AdWords… Even if you are doing a SEO campaign but would like to know where to emphasize your efforts you will need to conduct a PPC campaign in order to collect some data…
    I think this is something that we can see for the last year, the efforts of Google to send all advertisers into paid search….

  • Munaz Anjum

    The most predictable catastrophe may yet to come when % of ‘Not Provided’ in Google Analytics increases up to more than 60% or 80%. As of now, its 30%-40%,which least draws our attention, as we often claim that 80% data in GA is more than sufficient to analyze visitors’ search patterns. Sounds good but what if ratio inverses as 80:20? Summarizing my thoughts in agreement with Stephane that concealing search keywords should be actually a profile option rather than an arbitrary decision on the part of Google? Do you agree? Read here my full post on http://www.digitalgossips.com/

  • Abdul Wahab

    Yup, it seems like google trying to stop SEO or organc traffic.. I wonder in coming years google will not provide any kind of free traffic…. It will just allow adwords….

  • Unbound Marketing

    I miss it because i’m nosey, but looking at landing pages can tell you a lot of what’s going on. Unless you have all your keywords stuffed into your home page which is probably a bad idea anyway.

  • http://twitter.com/rankontop Troy Curfman

    Alot of info Depending on what tool used, there are still ways around this. Webmaster tools is still limiting your actual results. Data will be harder to get as se try to conceal and close data distribution to help thier own investments. Eventually it will leave public analytical only to the search engines.

  • http://www.v2interactive.net/ Josh

    I’ve been pushing “Search with Bing” on many of my clients internal search.

  • http://www.facebook.com/ranu.jain2 Ranu Jain Gupta

    My site is receiving half of its Google traffic via ‘Not provided’ and I won’t be surprised if it increases to 100%.

    It’s been more than an year now and I am still unable to understand that how hiding keyword data helps in meliorating the privacy for users. Even if it does, why has Google not applied it in Adwords too?

  • Trevor Fox

    I welcome this news.

    Working for a smaller agency who represents many boutique hotels with smaller data sets, I welcome anything that will provide parity with our clients’ larger chain competitors. Competition based on Quality of Analytics Rather Than Scale is key now and will become more so as more privacy issues arise.

    However, being fairly new to the industry, I just began reading Marketing in the Age of Google for some insight and its relevance is diminishing slightly. There is still a lot of valuable information in there. Anyway…

    Failure is not Fatal. Failure to Change might be. –John Wooden

  • http://www.ninebyblue.com Vanessa Fox

    Dave, You can filter the data in webmaster tools by country, but not to specific region. (Just choose Filters to change the property (such as web or images) and location). The data is definitely not manipulated by Google. It’s definitely accurate in terms of trends (although sometimes sampled, depending on site size and certainly bucketed).

    I wouldn’t expect your AdWords impression data to necessarily match exactly in any case, as not all searchers see all ads and all organic results. There’s some flux in organic due to personalization (and if the searcher went past the first page) but there’s definitely flux in how many searchers see your ads.

    I totally understand, though, that you would like regional data, and webmaster tools only enables you to drill down to the country level.

  • http://twitter.com/DerrickHicks3 Derrick Hicks

    Awwwwwwwweeeeeeeesssssssoooommmmeee!!!! I freakin love Google. SMH…

  • http://twitter.com/DerrickHicks3 Derrick Hicks

    That is exactly what I was thinking about doing now. Which is great! Now, instead of sending our sem traffic to a specially made landing page, we get to point it to the page ranking organically and burn through some more AdWords budget just to get some keyword level analytics data. I freakin love Google!

  • http://twitter.com/DerrickHicks3 Derrick Hicks

    Geez @twitter-10665432:disqus, sorry to hear that! @thelostagency:disqus, I was thinking the same thing.

  • http://twitter.com/DerrickHicks3 Derrick Hicks

    lol In their mind they already do. It’s called AdWords. :)

  • Mark

    Your statement that we can see the (not provided) keywords in Webmaster Tools is completely wrong. I was pretty sure it’s not possible; I have just extracted all keywords from Analytics and Webmaster Tools for December and both the number of keywords and visits per keyword are way lower and inaccurate in Google Webmaster Tools.

  • http://www.facebook.com/people/Ben-Guest/100002606955053 Ben Guest

    Let’s not forget about WIndows IE 8 InPrivate setting: http://windows.microsoft.com/en-US/windows-vista/What-is-InPrivate-Browsing

    Wasn’t that going to be the default? I don’t use IE so I don’t know if in fact it came enabled,

  • http://www.facebook.com/people/Ben-Guest/100002606955053 Ben Guest

    Also remember GWT data is always 2 days behind in Google Analytics.

  • RyanMJones

    I would like GWMT to do a better job of tying pages to keywords. It shows me the keyword, the impressions, rank, clicks – but not which page is the highest ranking one for that keyword. I can piece this together on my own, but if they did that it would help to better tie it into analytics data. I could then get conversion rates per page and know the top keywords for each page to focus on.

  • http://twitter.com/cstechjoel Joel K

    What exactly would this solve? The query data is made secure outside of Google+. All you’d accomplish is setting yourself back.

  • http://ChiefAlchemist.com/ Mark Simchock

    Re: “The searches people do that lead them to a site is valuable data to an
    organization. What other data source do we have that gives us direct
    access to our customers’ wants and needs? Sure, we can ask them, but
    customers can lie. Or choose not to answer. By looking at exactly what
    visitors have searched for, we can learn a lot about who our customers
    are, what they really want, and if we’re providing it to them.”

    Yup. It certainly helps.

    That being said, the number of reliable conclusions you can glean from a couple of keywords – typically coming off an auto-complete / suggestions list – is probably too often over-valued. Yes, it gives you a clue but not really the type of evidence you can use in court (so to speak), eh?

    For example, I enter a product model number. Am I looking to buy? For the manual? For tips & tricks? The possibilities are endless, yes? But I’d bet most e-comm sites are going to assume “buy” and a site that does repairs will think “spare parts”.

    Don’t get me wrong. It’s a shame to lose these. Just the same, it might just be a blessing in disguise for those willing to optimize their “machine” to bypass would could be deception. Aside from webmaster tools, now might be the time to add site search and/or specialty landing pages to back fill some of the holes the unstoppable force called Google is now creating.

  • http://www.examiner.com/internet-and-technology-in-national/david-frankk David Frankk

    I thought (not provided) was because of the searcher was logged into their account and not because of any other reason.

  • http://www.ninebyblue.com Vanessa Fox

    Originally, but now other types of searches are secured as well.

  • http://www.ninebyblue.com Vanessa Fox

    You can back into this (a feature we’re currently working on with our product), both by combining top pages data from webmaster tools and integrating analytics data.

  • http://www.facebook.com/farhan.asghar.54 Farhan Asghar

    What exactly solution is? The query data is made secure outside of Google plus.
    http://pakvideos.net/

  • Stephen Olson

    This makes on page SEO even more important as if we know what the content on the page is we can start to look at landing page hits instead of query searches. Also it is even more important to have a site search field on the site so that we can see what users are searching for once they get to the site.

  • http://twitter.com/robkingston Rob Kingston

    He means stop promoting it, so that fewer people remain logged into their Google accounts.

    Ultimately, if you want this data back, you need to convince people to use Bing :

  • http://www.facebook.com/edea.krammer.3 Edea Krammer

    Yep! Well said! I shared the same thought… XOXO:D
    DCLand.info

  • http://twitter.com/JC_Davies Jon Davies

    Google appears to have launched it’s Chrome 25 over the past 2 weeks. Version 25 is now coming out top for Chrome browser visits

  • http://twitter.com/doloreshark Dolores Hark

    Entered the field over a decade ago because the data was SO interesting and useful – a revolution in marketing, really. (We’ve always focused on good content and landing pages data AS WELL as keyword data to get the big picture – getting the whole picture was the revolution – not just the data side.) Now I’m training for another more stable job as a musician in preparation of no data and still no one wanting to pay much (small businesses) cause Google is free, and this general attitude that is it okay to sign up for expensive monthly service fees related to other marketing services or high pay per click budgets, but custom creative work is frequently foregone (i.e., reliance on commodified marketing strategies). Also, if you track the want ads, Google is beefing up it’s SMB bench. Google Analytics is free because it makes everyone Google’s little marketing lambs. It’s been painful to hear otherwise seemingly very intelligent people support such a marginally useful product, which I just assume is due to lack of exposure to other products. I’d advocate that people stop using it. Other programs are much more user friendly, with key insights right there on opening the program, without having to link this and that and set up experiments and hunt and peck for granular data. Also, I don’t think I’ve ever gained any great unique insights from Webmaster Tools, that wasn’t much better presented by my paid for tools, but now I will definitely investigate Blueprint as an overlay product – good work on coming up with that. Also, wanted to note – I don’t think the large third-party dataset tools I’ve always used are even worth the bother now – the data was always highly projected from a small sample size and I believe the projection errors are even worse now, and basically feel compelled to just rely on Google Adwords data as the key third party data set. Too bad no one in our government seems to understand the anti-trust needs of our modern economy. It seems odd to me that one company, with this incredible data power which it is now attempting to monopolize, can have it’s digits stuck in literally every pie in the information age pie shop.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide