Google confirms AJAX (i.e. XHR) calls consume crawl budget

In January 2017, Gary Illyes from Google published a document on the Google blog explaining how crawl budget works to webmasters and SEOs. He has now updated the document to make clear that AJAX (i.e. XHR) calls, can and will consume a site’s crawl budget.

JFYI: based on @ajkohn 's recommendation, i added AJAX/XHR calls explicitly to the list of things that will consume your site's crawl budget.https://t.co/kVYj6XaNn0

— Gary "鯨理" Illyes (@methode) April 23, 2019

The clarification. The update (in italics added here) reads, “Generally, any URL that Googlebot crawls will count towards a site’s crawl budget. Alternate URLs, like AMP or hreflang, as well as embedded content, such as CSS and JavaScript, including AJAX (i.e. XHR) calls, may have to be crawled and will consume a site’s crawl budget. Similarly, long redirect chains may have a negative effect on crawling.”

What is crawl budget? Google defines crawl budget as “the number of URLs Googlebot can and wants to crawl.” In short, crawl budget is “taking crawl rate and crawl demand together.”  You can read more about it here.

Did anything change? No, Google just clarified that AJAX/XHR calls do consume crawl budget and that has always been the case.

Why we should care. Generally Google has said time and time again that crawl budget is not something to worry about for most webmasters and SEOs. Google will crawl most sites without an issue. Really large web sites that need to prioritize what Google crawls may want to double check their AJAX/XHR calls to verify if Google is crawling them and more importantly, if they want those calls to take up their site’s crawl budget.