Google recently released news that it is now moving to encrypt all organic search activity. This means that analytics programs will no longer be able to report on keyword terms that drive organic traffic from Google, instead noting the data as being “not provided.”

This could be devastating to marketing managers and webmasters who rely heavily on this search data to determine ROI on their organic campaigns. With the data obfuscated from analytics programs, many feel that the organic search marketing channel has been stricken from the record as a legitimate marketing channel.

This data obfuscation should serve as a clear indicator to marketing managers that they must begin using organic search to fuel a healthy, more holistic inbound marketing strategy. We took the indicators that way. But what if you could still get access to the “not provided” data?

Having an idea of keyword-specific organic traffic to landing pages is more invaluable than ever. Understanding this data helps a marketing manager correctly focus organic budget and efforts. So how would one go about calculating keyword-specific data if 100 percent of your traffic will show up as “not provided” moving forward?

Uncloaking Not Provided

Use Case: Reporting on last month’s keyword-specific organic traffic to a landing page.

Google Analytics NavigationStep 1: Pull traffic data for a landing page from Google Analytics for the previous month.

To do this, navigate to the “Behavior” reports, drilling down into “Site Content” and viewing the “Landing Pages” report. From here, set your date range to the previous month. You can see all landing pages that received traffic over the last month as well as how much traffic they received. Export the data, and step 1 is complete.

Step 2: Build an irrefutable list of keywords that drive traffic to a targeted landing page. Identify — and, moving forward, periodically catalog — keywords and keyword rankings for specific landing pages using a tool like SEMRush.

You should get a fairly accurate snapshot of keywords driving traffic to specific landing pages. SEMRush also provides rankings. SEMRush updates their data once a month; if you’d like more frequently updated data for a more detailed look, you could use another tool (easy to find) that pulls rankings more regularly like once a week or even once a day. Export the data, and step 2 is complete.

If we were doing this with existing pages, we’d start with analytics to build the list, supplement the list with paid search data, and compliment it with Bing/Yahoo search volume data using methods similar to Moz’s explanation. If we were doing this with new pages, we’d start with SEMRush data and supplement it with data from a host of other keyword ranking identification tools.

Step 3: Pull search volumes for the keywords ranking for the landing page from Google Keyword Planner (yes, you can pull exact and phrase).

Google Keyword Planner recently replaced Google Keyword Tool. Yes, the data is still available. Here’s how to find and use it: Go to Google Keyword Planner and click the “Get traffic estimates for a list of keywords” option. Drop in your list of keywords, which were pulled from SEMRush. Turn bid range all the way up. Click the last point on the bid range.

Google Keyword Planner Navigation

The table you’ve produced will show a list of your keywords with daily estimate data. For exact match search volumes, go to the “Edit match types” dropdown, hover over “All ad groups” and select “Exact match.” Make sure you select “Segment by” keyword. Download the data.

For phrase match search volumes, go to the “Edit match types” dropdown, hover over “All ad groups” and select “Phrase match.” Make sure “Segment by” keyword is selected. Download the data. To calculate monthly data, take the daily estimates for both your phrase and exact match, multiply each keyword data point by 365 (for days in a year) and divide by 12 (for months in a year). Then, to get exact percent of search volume, divide your exact monthly impressions by total of exact and phrase monthly impressions.

Do the same for phrase percent of search volume, divide your phrase monthly impressions by total of exact and phrase monthly impressions. Multiply each percentage by the search volume provided by the Google Estimator. This will give you monthly search volume for both phrase and exact for that keyword.

Step 4: Determine CTR ratios, using a CTR study like the ones from BrightEdge or DigitalRelevance as a benchmark.

Take the keywords and keyword rankings that you pulled from step 2, apply CTR to the rankings to get a CTR benchmark for exact and CTR benchmark for phrase. Multiply the monthly exact search volume you pulled from step 3 to the exact CTR benchmark for each keyword. Multiply the monthly phrase search volume you pulled from step 3 to the phrase CTR benchmark for each keyword. This will give you a traffic estimate benchmark for each keyword.

Let’s talk for a moment about one of the major benefits of using a CTR study that includes LT CDigitalRelevance CTR StudyTR. Long-tail CTR is extremely important when understanding total traffic to a landing page. Long-tail traffic generally makes up the majority of traffic to a landing page. In other words, 5 percent of phrase volume from a rank-1, page-1 search engine listing will be much more traffic than 18 percent of exact volume to the same listing. That’s why it’s essential to have long-tail CTRs included in calculations when uncloaking not provided data (and similarly, when using predictive analytics to provide traffic estimates).

But what about keywords that don’t fall into positions 1-10 because they’re covered by the CTR study? Here’s where regression analysis comes in. Take the CTR study curve, apply a regression trend line to the curve, and you have your CTRs for the positions that follow. From here, perform the same calculation mentioned above for your keywords that are in positions higher than 10.

Step 5: Perform calculation (yes, this is provided in the post) to determine keyword-specific organic search traffic.

Total all the traffic benchmarks from Step 4. Divide the traffic for each keyword by overall traffic to get a percent of overall traffic for that keyword. Compare your total traffic to the Google Analytics landing page traffic pulled in Step 1. There will most likely be a difference — and that’s fine. The CTR study is a benchmark. Your site may have under-performed or out-performed the benchmark.

To determine each keyword’s actual traffic from here, multiply the total landing page traffic by each keyword’s percent of overall traffic. Voila! You have successfully uncloaked keyword-specific organic “not provided” data with a degree of accuracy based on an industry standard.

Formulas

Below are the formulas for uncloaking not provided data versus provided data and data results.

Uncloaking Not Provided Data

  • Exact CTR: Use Current Rank to Pull Exact CTR from CTR results tool
  • Exact Monthly Search Impressions: (Exact Daily Impressions * 365)/12
  • Exact % of Search Volume: Exact Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
  • Search Volume Exact: Exact % of Search Volume * Google Provided Search Volume
  • Not Provided Exact Volume: Search Volume Exact * Exact CTR
  • Phrase CTR: Use Current Rank to Pull Long-Tail CTR from CTR tool
  • Phrase Monthly Search Impressions: (Phrase Daily Impressions * 365)/12
  • Phrase % of Search Volume: Phrase Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
  • Search Volume Phrase: Phrase % of Search Volume * Google Provided Search Volume
  • Not Provided Phrase Volume: Search Volume Phrase * Phrase CTR
  • Not Provided Total Traffic (driven by that keyword): Not Provided Exact Volume + Not Provided Phrase Volume
  • Not Provided % of Overall Traffic: Not Provided Total (for that keyword) / Sum of Not Provided Total (sum of all keywords)

Versus Provided Data

  • Actual Traffic (for each keyword): Pulled from Analytics
  • Not Provided Calculation (for each keyword): Actual Traffic * (1-Not Provided (for the month)/Sum of All Keywords Actual Traffic)
  • Total Traffic Per Keyword (better estimate for each keyword): Actual Traffic + Not Provided Calculation
  • % of Overall Traffic: Total Traffic Per Keyword / Sum of Total Traffic Per Keyword (sum of all keywords)

Data Results

  • Diffs: Absolute Value of (Not Provided – Actual)
  • Traffic for Not Provided Final Breakdown (for each keyword): Overall Landing Page Traffic * Not Provided % of Overall Traffic

Uncloaking Not Provided Next Steps & Considerations

We will work to refine this method for more accurate results as time progresses. For now, this should serve as a basic starting point for calculating not provided moving forward. We’ve worked on putting this uncloaking methodology together. While it still needs some work, our next steps are meant to provide real-world examples.

For more accurate results, build a stronger keyword list for the targeted landing page by using additional keyword research tools. In addition, the same analysis would be better by tracking daily SERP rankings for each keyword using a platform like BrightEdge.

In turn, CTRs would change daily and the overall calculation would provide a more detailed picture of the not provided organic search traffic by keyword breakdown. Regarding each data pull (keyword list, analytics data, keyword planner, etc), the goal is to build a tool to pull this data periodically and perform calculations automatically after providing only a targeted landing page.

Using this theory of uncloaking “not provided,” we’ve run some initial real-world tests and have found a positive correlation. Although there is a positive correlation to the theory compared to the actual data, the data does not conclusively support this method of uncloaking “not provided.” Simply put, we need more tests and more data to provide irrefutable evidence of using this method. More importantly, the goal here is to spur discussion around this topic and hopefully provide a starting point for keyword-specific reporting and decision-making.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: All Things SEO Column | Analytics | Channel: SEO | Google: Analytics | Google: SEO | How To: Analytics | Search & Analytics | SEO - Search Engine Optimization | SEO: General

Sponsored


About The Author: is an Inbound Marketing Consultant at DigitalRelevance. He graduated from the Indiana University Kelley School of Business with a B.S. in Business Management.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Scott Pittman

    The search volume for phrase and exact (and broad) is the same – it’s just the exact match search volume that is shown. If anyone actually finds a way to get search volume for different match types then please share. . unless I’ve missed something?

    Thanks

  • Scott Pittman

    The search volume for phrase and exact (and broad) is the same – it’s just the exact match search volume that is shown. If anyone actually finds a way to get search volume for different match types then please share. . unless I’ve missed something?

    Thanks

  • http://www.inmediaconcepts.co.nz/ William John

    A long process to uncover not provided data, but still it makes sense putting the effort to pull the data when it’s available and fresh.

  • http://www.inmediaconcepts.co.nz/ William John

    A long process to uncover not provided data, but still it makes sense putting the effort to pull the data when it’s available and fresh.

  • Scotts Contracting

    This makes me really question how #google is really trying to help the small business industry.

  • Scotts Contracting

    This makes me really question how #google is really trying to help the small business industry.

  • https://plus.google.com/117626320669881683246/ Bradley G. Smith

    Scott, I’m glad you asked this question. I can see you’re actively using Google Keyword Planner. I covered how to calculate phrase and exact search volume in the post. I’ll probably try to write a blog post on this in the near future so you can see the reasoning behind the methodology. The trick is to utilize daily impressions. When you download the data from Google Keyword Planner, ‘Estimated Impressions’ (vs provided search volume) are unique to phrase and exact (and broad) match terms. Step #3 from the blog post covers the calculation (refer to this step in the post for the process). The formulas for calculating phrase and exact are below (they are also in the Formulas section of the post).

    Formulas:
    Search Volume Exact
    1) Exact Monthly Search Impressions: (Exact Daily Impressions * 365)/12
    2) Exact % of Search Volume: Exact Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
    3) Search Volume Exact: Exact % of Search Volume * Google Provided Search Volume

    Search Volume Phrase
    1) Phrase Monthly Search Impressions: (Phrase Daily Impressions * 365)/12
    2) Phrase % of Search Volume: Phrase Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
    3) Search Volume Phrase: Phrase % of Search Volume * Google Provided Search Volume

    Again, I’ll try to write a post on this in the near future so I can share my detailed reasoning on the methodology provided here.

  • https://plus.google.com/117626320669881683246/ Bradley G. Smith

    Scott, I’m glad you asked this question. I can see you’re actively using Google Keyword Planner. I covered how to calculate phrase and exact search volume in the post. I’ll probably try to write a blog post on this in the near future so you can see the reasoning behind the methodology. The trick is to utilize daily impressions. When you download the data from Google Keyword Planner, ‘Estimated Impressions’ (vs provided search volume) are unique to phrase and exact (and broad) match terms. Step #3 from the blog post covers the calculation (refer to this step in the post for the process). The formulas for calculating phrase and exact are below (they are also in the Formulas section of the post).

    Formulas:
    Search Volume Exact
    1) Exact Monthly Search Impressions: (Exact Daily Impressions * 365)/12
    2) Exact % of Search Volume: Exact Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
    3) Search Volume Exact: Exact % of Search Volume * Google Provided Search Volume

    Search Volume Phrase
    1) Phrase Monthly Search Impressions: (Phrase Daily Impressions * 365)/12
    2) Phrase % of Search Volume: Phrase Monthly Search Impressions/(Exact Monthly Search Impressions + Phrase Monthly Search Impressions)
    3) Search Volume Phrase: Phrase % of Search Volume * Google Provided Search Volume

    Again, I’ll try to write a post on this in the near future so I can share my detailed reasoning on the methodology provided here.

  • http://www.zldoty.com/ Zach Doty

    Scott, good question. I’d like to follow up to Brad & expound on his response bit. From my understanding, the question is a bit unclear – my understanding and corresponding answer assumes you’re talking about match types in Google Keyword planner.

    In Google Keyword Planner, you can edit the match types in the “Your Plan” section on the right of the screen. Click the pencil and you can change match types as desired.

    If you’re referring to the end result of getting broad match data, I would encourage you to consider the principles of conservatism in data estimation / forecasting. From your perspective, would you want to reflect the highest amount of data or the most accurate data? I think it touches a bit too on managing expectations in your client or stakeholder relationships.

  • http://www.zldoty.com/ Zach Doty

    William, you’re absolutely right. Consider this, all of these updates are a knell for shortsighted marketers and a windfall for above-board practitioners/strategists.There are a number of SEO automation platforms which are performing this (for a fee, of course) and you could probably string together this data automatically if you have Adwords API access and strung together Excel templates or Macros. Hope that helps.

  • http://www.LeadDiscovery.com/ Jerry Nordstrom

    Why does keyword data have to be encrypted? NSA or NASDAQ?

    Can’t KW data be extracted from log files – Should we not go old-school?

    How can Google Glass data be less invasive than keyword data?

    Many will proclaim that the future of Internet marketing is based upon analysis and insertion of our companies into the real-time consumer data stream or “holistic marketing”. This is true, but the ugly red-headed step child at Thanksgiving here is data privacy. If consumers are given control of their personal data what happens to this model that so many tech companies have bet their bean bag laden gaming rooms on?

  • http://www.zldoty.com/ Zach Doty

    Jerry, it’s probably somewhere between privacy from the NSA, Adwords & Google Analytics Premium sales. Keyword data cannot be extracted from log files. Consider this, the Google keyword data was water coming from a well, (in G Analytics) via a faucet in the kitchen. Google has encrypted the URL and data, so they’ve effectively sandbagged our metaphorical well. :-) Moving to another room of the house won’t help when the well is closed. Does that make sense?

    The question of consumers and data privacy another beast. Also, if you’re interested in the Google sales aspect, I believe Slingshot SEO (aka Digital Relevance) published an interesting post on Google’s efforts to extract one last year of YOY revenue percentage increases, or something to that effect. Perhaps Brad could furnish you the link.

  • http://tweetphiladelphia.com/ Eric Van Buskirk

    I work at SEMrush. Great peice, but please note we update data daily and at the MOST weekly. It’s not a month old.

  • https://plus.google.com/117626320669881683246/ Bradley G. Smith

    Eric, thanks for the update. I just spoke with another employee of SEMRush and was coming to make a note on the post. Thanks for the correction and clarification.

    I’m told that in the case of some keywords the data is updated even more than once a day. And, this underscores my point even more, SEMRush can be utilized to build a relevant, fresh, up-to-date irrefutable keyword list.

  • rituraj

    why not just use the webmaster tools instead to get the KWs and the CTR

  • https://plus.google.com/117626320669881683246/ Bradley G. Smith

    rituraj, good question. As I’m sure you’ve already seen from this post, http://googlewebmastercentral.blogspot.com/2011/10/webmaster-tools-search-queries-data-is.html, those reports from GWT are now in Google Analytics. In these reports, Google does not directly correlate keyword data with landing page data. As of right now, Google keeps those reports separate. So, reporting on keyword-specific organic traffic to a landing page from those reports is not currently possible.

  • http://dannybrown.me/ Danny Brown

    Would something like Woopra counter this, if used as a main analytics platform? Or are they based around the same “limitations” (for want of a better word) that the new GA set-up dictates?

 

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide