Search Engine Land
  • SEO
    • > All SEO
    • > What Is SEO?
    • > SEO Periodic Table
    • > Google: SEO
    • > Bing SEO
    • > Google Algorithm Updates
  • PPC
    • > All PPC
    • > What is PPC?
    • > Google Ads
    • > Microsoft Ads
    • > The Periodic Tables of PPC
  • Focuses
    • > Local
    • > Commerce
    • > Shopify SEO Guide
    • > Content
    • > Email Marketing Periodic Table
    • > Social Media Marketing
    • > Analytics
    • > Search Engine Land Awards
    • > All Focuses
  • SMX
  • Webinars
  • Intelligence Reports
  • White Papers
  • About
    • > About Search Engine Land
    • > Newsletter
    • > Third Door Media
    • > Advertise

Processing...Please wait.

Search Engine Land » Google » Google: SEO » Google On How They Know When To Slow Or Stop Crawling Your Web Site

Google On How They Know When To Slow Or Stop Crawling Your Web Site

Today at SMX East, Google’s Webmaster Trends Analyst, Gary Illyes shared with the audience two technical ways Google determines when GoogleBot, its crawler, should slow down or stop crawling your website. One of the more important factors with SEO is to ensure the search engine crawlers can access your web pages. If they cannot access your […]

Barry Schwartz on October 1, 2014 at 10:54 am

google-tools1-ss-1920

Today at SMX East, Google’s Webmaster Trends Analyst, Gary Illyes shared with the audience two technical ways Google determines when GoogleBot, its crawler, should slow down or stop crawling your website.

One of the more important factors with SEO is to ensure the search engine crawlers can access your web pages. If they cannot access your web pages, then you will most likely have a hard time ranking in the search results.

Google said that it uses many signals for determining if it should stop crawling your website, outside of the obvious one like the disavow, robots.txt and nofollow tags.

Gary from Google said the following two signals are important crawl signals for Google:

Connect Time

Google will look at how much time it takes for them to connect to the server and web page. If that connect time is getting longer and longer, Google will back off and slow or stop crawling your web pages. Google doesn’t want to end up taking down your server, so it uses connect time as a crawling factor.

HTTP Status Codes

Google will also stop or slow their crawling when it gets server status codes in the 5xx range. The 5xx Server Error status codes often mean there are issues with the server responding. You can find a whole list of them on Wikipedia.

Google says when it sees these codes, it backs off, as to not cause more issues for your server.

In both cases, GoogleBot will come back later, but it backs off when it sees these two signals are causing issues as to not cause more issues for your users when it tries to access your website.


New on Search Engine Land

    More FAQ rich results being displayed in Google Search

    Webinar: Benchmark your social media performance for a competitive edge

    Google releases May 2022 broad core update

    Spotify, Meta update political ad offerings for 2022 election cycle

    Take web hosting to the (NVMe) extreme

About The Author

Barry Schwartz
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Related Topics

Google: SEOSEO

Get the daily newsletter search marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

The SMX Conference logo.

Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.

March 8-9, 2022: Master Classes (virtual)

June 14-15, 2022: SMX Advanced (virtual)

November 15-16, 2022: SMX Next (virtual)

Learn More About Our SMX Events

The MarTech Conference logo.

Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.

Start Discovering Now: Spring (virtual)

September 28-29, 2022: Fall (virtual)

Learn More About Our MarTech Events

Webinars

Benchmark Your Social Media Performance For a Competitive Edge

Take a Crawl, Walk, Run Approach to Multi-Channel ABM

Content Comes First: Transform Your Operations With DAM

See More Webinars

Intelligence Reports

Enterprise SEO Platforms: A Marketer’s Guide

Enterprise Identity Resolution Platforms

Email Marketing Platforms: A Marketer’s Guide

Enterprise Sales Enablement Platforms: A Marketer’s Guide

Enterprise Digital Experience Platforms: A Marketer’s Guide

Enterprise Call Analytics Platforms: A Marketer’s Guide

See More Intelligence Reports

White Papers

The State of Influencer Pricing

How to Measure Influencer Performance

Reputation Management For Healthcare Organizations

Unlock the App Marketing Potential of QR Codes

Realising the power of virtual events for demand generation

See More Whitepapers

Receive daily search news and analysis.

Processing...Please wait.

Topics

  • SEO
  • PPC

Our Events

  • Search Marketing Expo - SMX
  • MarTech

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2022 Third Door Media, Inc. All rights reserved.