• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • Live
  • More
  • Events
    • Follow
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • More
  • Newsletters
  • Home
Analytics & Conversion

Googlebot Makes An Appearance In Web Analytics Reports

A few days days ago, I noticed some strange Google Analytics data: Googlebot appeared as a browser in the reports. Although this might sound like a not-so-important fact when it comes to SEO, it is a major change in the Web Analytics field. As Avinash Kaushik and I wrote in the SEMJ journal article Web […]

Daniel Waisberg on August 3, 2009 at 1:12 pm
  • More

A few days days ago, I noticed some strange Google Analytics data: Googlebot appeared as a browser in the reports. Although this might sound like a not-so-important fact when it comes to SEO, it is a major change in the Web Analytics field. As Avinash Kaushik and I wrote in the SEMJ journal article Web Analytics 2.0: Empowering Customer Centricity, an important advantage of all JavaScript based solutions (Google Analytics, Omniture, Yahoo Web Analytics…) is:

The JavaScript is not read by crawlers, which generates high amounts of traffic and are not representative of customers’ behavior. Crawlers can be excluded from the analysis; however, it is a time consuming task, and many of them are not recognizable.

To check whether this bot is really from Google, and not some kind of user agent switcher, I drilled down on the data and here is what I found.

Googlebot appears in Google Analytics reports

First of all, as we can see below, the Googlebot is recognized as a browser (version 2.1):

Googlebot Browser on Google Analytics

Second, when we drill down to the network location report we find the following:

Googlebot Network properties

How does it affect the data?

If we look at the behavior of this bot, we see a very low time on site, very low pages/visit, and very high percentage of new visits. This might be due to the fact that the bot does not fetch cookies, which is essential to accurate analytics tracking. Below are some numbers:

Googlebot Behavior

Statistically speaking, this means that the Googlebot is an outlier, which is a data point that lies outside of the overall pattern of a distribution. It means that it can distort the numbers. In the example above, just a few visits with very low time on site and percentage of new visits can significantly decrease the overall average time on site andpercentage of new visitors, which is clearly bad for someone looking at the overall behavior of visitors.

How to exclude Googlebots from your Google Analytics data

Here is a filter that can be applied to Google Analytics profiles to exclude this Googlebot from messing with your data.

Exclude Googlebot Filter on Google Analytics

What lies ahead?

Google has been officialy scanning JavaScript since 2008. So maybe this has been a low priority or low usage technique untill now, used only in very specific cases. But recently we have seen an increase in this practice, so the big question is whether this is a trend that will increase as time passes or is it just a few specific tests run by Google? Editor’s note: Google declined to comment when asked for more information.

For now, we can only hope that this kind of data is not being collected by analytics packages from the back door. If it has been this might have been skewing the data quite a bit given Googlebot’s low time on site and percentage of new visits stats.

Disclosure: The data used on the screenshots above was extracted from the Web Analytics Association website. If you would like to take a look at this data, it is currently available to all members as part of the Web Analytics Championship.

Postscript: Google Analytics posted a response in the comments:

“The official Google bot does not execute Google Analytics JavaScript. We’re not sure what it is exactly, it could be anyone’s bot, some intern’s experiment, or other such traffic.”

I agree with this comment in that the official Googlebot reads JavaScript but does not execute it. Besides, it does not store and send cookies, which means that Paves/Visit would be exactly 1 and time on site exactly 0. Lastly, If the officiall Googlebot did execute JavaScript, we would have seen massive ammounts of visits.

It is also important to note that although we used Google Analytics as an example, we mean all JavaScript based solutions, including Omniture, Yahoo Web Analytics, WebTrends and others.

Please note that this issue requires additional investigation both in regards to Google Analytics and to how Google Search uses the Googlebot.


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.



About The Author

Daniel Waisberg
Daniel Waisberg is the Principal og Conversion Journeyand the founder of Online Behavior, a Marketing Measurement & Optimization website. He holds a M.Sc. in Operations Research and Decisions from Tel Aviv University, where he developed a statistical model that helps to optimize websites using Markov Chains. Daniel is a frequent speaker & member of the Advisory Council of the eMetrics Marketing Optimization Summit. You can follow him on Twitter or Google+.

Related Topics

Channel: Analytics & ConversionFeatures: AnalysisGoogle: Analytics

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter search marketers rely on.
See terms.

ATTEND OUR EVENTS

Lorem ipsum doler this is promo text about SMX events.

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December 17, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

Discover actionable tactics that can help you overcome crucial marketing challenges. Our next conference will be held:

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • The State of Local Marketing Report 2020-2021
  • Quality CRM Data: The Key to Delivering Great Customer Experiences
  • How the Microsoft Search Network Can Maximize Your Search Campaigns
  • The Marketer’s Playbook for Customer Acquisition
  • How To Optimize SEO With UGC
See More Whitepapers

Webinars

  • How to Avoid the Digital Transformation Trap
  • How to Build a Marketing System of Record
  • Meet BIMI: The brand-boosting email security marketers must have for 2021
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily search news and analysis.
Search Engine Land
Download the Search Engine Land App on iTunes Download the Search Engine Land App on Google Play

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX
  • MarTech

Resources

  • White Papers
  • Research
  • Webinars
  • Search Marketing Expo
  • MarTech Conference

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok