How To Use Fetch As Googlebot Like An SEO Samurai

For those of you not familiar with this feature in Google Webmaster Tools, contributor John Lincoln explores its capabilities in depth.

Chat with SearchBot

seo-samurai-ss-1920

Do you know about Fetch as Googlebot? It is a feature of Webmaster Tools that allows you to input a URL, evaluate it, and (should you choose) submit it to the Google index for a fresh crawl. In this post, I am going to show you how to use the Fetch as Googlebot feature like an SEO samurai.

What Is Fetch As Googlebot

Fetch as Googlebot has two modes.

Fetch – Quick Check

In this mode, Googlebot will crawl a URL based on a path you specify, and you can then view the response your site sent to Googlebot. This can be used to check connectivity, basic errors, redirects or security issues with your website.

Fetch & Render – Deeper View

In this mode, Google crawls the URL and displays the page the way a browser would.

First, Googlebot gets all the resources referenced by your URL such as picture, CSS, and JavaScript files, running any code. to render or capture the visual layout of your page as an image. You can use the rendered image to detect differences between how Googlebot sees your page, and how your browser renders it.

How To Use Fetch As GoogleBot

Fetch as Googlebot

Here are the basic steps:

  1. On the Webmaster Tools home page, select your site.
  2. In the left-hand navigation, click Crawl and then select Fetch as Google.
  3. In the textbox, enter the path component of a URL on your site that you want Googlebot to retrieve.
  4. From the grey drop-down, choose the type of Googlebot with which you wish to perform a fetch (or fetch and render).

The Different Types Of Googlebot Requests

There are four different Googlebot requests you can make in Fetch as Google.

If you would like to see how a desktop browser sees your site, select Desktop. To see how smartphone mobile crawlers see your site, choose Mobile Smartphone.

If you want to see how the mobile crawler for a feature phone sees the site, select Mobile cHTML or Mobile XHTML/WML. (Google notes that the Mobile cHTML is mainly for Japanese websites.)

Dropdown Fetch as Google

If you do a basic Fetch, you can see the HTTP response that Google got from your page. When you do Fetch and Render, you get the response and get to see how your page looks in the specified browser.

(Note: The “Fetch And Render” option is not available for the WML and cHTML at this time, so unfortunately you won’t be able to see how your page renders for feature phones.)

Once you’ve entered your URL and fetched it, you will have the option of submitting it to Google for indexation. You’ll see a button allowing you to do so. (This button will also appear on the upper right-hand corner of the Fetch details page.)

google-webmaster-tools-fetch-render-submit

After you submit a URL to be fetched, you can click on the path once it’s completed. This will bring you to a page with further details. At the top, you’ll see the URL that was fetched, the Googlebot type, and when it was completed. If you chose to Fetch and Render the page, you’ll see a “Fetching” tab and a “Rendering” tab.

webmaster-tools-fetch-details

The Fetching tab shows you the downloaded HTTP response, which can be very helpful if you are ensuring a page status, and it also shows you the code of the page. Finally, it shows you the download time of the page at the bottom.

The Rendering tab will give you an idea of how your page renders in the browser selected (desktop or smartphone). It will also list any page resources that Googlebot was not able to access.

How Many Fetches & Submissions Can You Do?

So, there actually is a quota on fetches and submissions, but most of us will never exceed it. According to Google, you are limited to 500 fetches per week. (You’ll get a notification that you’re approaching your limit once you have hit 400.)

When it comes to actual submissions to the index, they give you 500 individual URL submissions and 10 submissions where Google will crawl that URL and all of the direct links as well.

Fetch as Google Submit

Diagnosing Your Fetch As GoogleBot Results

With this tool, you can submit a site a to get it indexed quicker, see if a page is blocked by robots.txt, see if a page is being redirected, make sure a page works on mobile, see if your server is taking too long to respond and so much more. But you can’t do any of that if you don’t know what the fetch statuses mean. So here they are.

Fetch Statuses

What are the Different Fetch as Googlebot Status

Fetch And Render Statuses 

What are the Different Fetch and Render Status

Tricks Of The Trade For Fetch As Googlebot

Now that we have covered the basics, I want to give you a few ideas on how you can use this tool.

  • Did you update an important page with new optimization and copy? Fetch that page and submit to the index. You will get a fresh, quick crawl and index.
  • Did you launch a new section of your site? Fetch the main page and submit it, along with all the linked pages. It will get that section indexed quicker.
  • Did you launch a new mobile experience (mobile site, responsive design, etc.)? Fetch as the mobile smartphone Googlebot and ensure that Google can access it.
  • Did you add a new robots.txt file? Fetch a page that you might be worried about and make sure both Google and Google mobile can access it. There is also the robots.txt tester tool for this, mind you.
  • Do you have rel=canonical or 301 redirects on certain URLs? Do you have multiple redirects? Want to make sure Google is interpreting them correctly? This is a great tool to test it.
  • Do you have an https version of your site live, but you don’t want it in the index? Set up webmaster tools for that version and test that is its acting correctly with this tool.

As you can see, there are many applications for this tool. One thing that got me really excited about it and made me want to write this article was the indexing. I made some significant changes to a page, used this tool to submit it to the index, and within a few hours, I was visible for a competitive keyword. So, as you can see, this tool can help in a variety of areas.

Summing Up Fetch As Google

This tool is a useful one to have in your arsenal. For me, it is right up there with the mobile testing tool, page speed tool, robots.txt tester and disavow tool in its ability to deliver useful technical SEO insights.

Do you use Fetch as Googlebot in a way I didn’t mention? Have a good story? Comment below.


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.


About the author

John E Lincoln
Contributor
John Lincoln is CEO of Ignite Visibility, a digital marketing agency and an Inc. 5000 company. Lincoln is consistently named one of the top marketing experts in the industry. He has been a recipient of the Search Engine Land "Search Marketer of the Year" award, named the #1 SEO consultant in the US by Clutch.co, most admired CEO and 40 under 40. Lincoln has written two books (The Forecaster Method and Digital Influencer) and made two movies (SEO: The Movie and Social Media Marketing: The Movie) on digital marketing.

Get the must-read newsletter for search marketers.