Google explains what “crawl budget” means for webmasters

Google says crawl demand and crawl rate make up GoogleBot's crawl budget for your website.

Chat with SearchBot

tech-arrows-direction-forward-movement-ss-1920

Gary Illyes from Google has written a blog post titled What Crawl Budget Means for Googlebot. In it, he explains what crawl budget is, how crawl rate limits work, what crawl demand is and what factors impact a site’s crawl budget.

First, Gary explained that for most sites, crawl budget is not something to worry about. For really large sites, it becomes something  to consider looking at.

“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters,” Gary said.

Here is a short summary of what was published, but I recommend reading the full post.

  • Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server.
  • Crawl demand is how much Google wants to crawl your pages. This is based on how popular your pages are and how stale the content is in the Google index.
  • Crawl budget is “taking crawl rate and crawl demand together.” Google defines crawl budget as “the number of URLs Googlebot can and wants to crawl.”

The blog post goes on to explain what factors impact crawl budget and then provides some FAQs on the topic.


About the author

Barry Schwartz
Staff
Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Get the must-read newsletter for search marketers.