Crawl budget: What you need to know in 2027

Technical SEO in 2027 doesn’t mean just caring about traditional search engines. AI agents and bots are now part of the picture, too. Crawl budget issues don’t just impact how easily Googlebot can navigate a site. We also have to consider how AI agents and bots navigate our sites.

Search bots can limit how much they crawl, while AI bots introduce new considerations for how sites are accessed. Understanding what affects crawling — and how to help bots navigate your site efficiently — is an important part of managing your site’s technical SEO.

What is crawl budget?

Crawl budget refers to the amount of time and resources a bot allocates to crawling a website. For Googlebot, it’s determined by two key factors: the crawl capacity limit and crawl demand.

The crawl capacity limit reflects how much crawling a site can handle without performance issues.

Crawl demand is based on Googlebot’s assessment of the website’s content, including individual URLs, and the need to update its understanding of those pages.

More popular pages are crawled more frequently to ensure the index remains up to date.

Google calculates this budget to balance the resources it can afford to spend on crawling with the need to protect both the website and its own infrastructure.

Why would search bots limit crawling?

Google’s Gary Illyes provides an excellent explanation of crawl budget, describing how Googlebot strives to be a “good citizen of the web.” This principle is key to understanding the concept of a crawl budget and why it exists.

Think of the last time tickets to your favorite band went on sale.

If it were anything like my last time trying to buy tickets, too many users likely flooded the website, overwhelming the server and causing it not to respond as intended. This is frustrating and often prevents users from buying tickets. Remember the Taylor Swift debacle?

This same phenomenon can also happen with crawlers. I mean, have you ever forgotten to adjust the crawling speed or number of simultaneous connections allowed on your favorite site crawler and accidentally brought down the website you were crawling on? Yeah, me, too.

Googlebot isn’t any different from other crawlers and could also cause this kind of website collateral damage. For example, it could hit a website too frequently or through too many parallel connections and cause the same effect, essentially overwhelming the server.

As a “good citizen,” it’s designed to avoid that happening. Google sets its crawl capacity limit for a site based on what the site can handle. If the site responds well to the crawl, it’ll continue at that pace and increase the volume of connections. If, instead, it responds poorly, the fetch speed and the number of connections used will be reduced.

Crawling, parsing, and rendering consume resources, so there are also financial considerations involved. That’s another major reason Google and other search engines and LLM companies may adjust how they crawl a site to benefit it: cost efficiency.

What causes issues with crawl budget

Not every site will have issues with its crawl budget. Google explicitly states that only three types of websites need to actively manage their crawl budget. These are:

• Large sites (i.e., those with over 1 million unique pages).
• Medium or large sites with frequently updating content.
• Sites with a high volume of “Discovered – currently not indexed” — essentially uncrawled pages — as detailed in Google Search Console’s Page Indexing report.

I’d advise you to be cautious before automatically assessing your website as none of the above. Instead, take the time to crawl your site and see what’s really going on.

For example, you may believe that your small ecommerce store only has a couple of thousand SKUs and a handful of informational pages. In reality, though, with faceted navigation and pagination, you could have 10 times the volume of URLs you thought you would have.

Don’t forget that having more than one language or location targeted at your domain may yield multiples of each page.

Set your crawling tool to crawl as Googlebot or Bingbot and let it loose on all pages that these search bots would be able to access. This will give you a more accurate picture of the size of your website as they know it.

If your crawl budget is too low to allow the bots to discover all the new URLs you’ve added to your site or to revisit URLs that have changed, then they won’t know about the content on them.

That means the pages may not be indexed or, if they are, they may not rank as high as they could if the bots could crawl them.

How crawl budget issues happen

Three main factors can cause crawl budget issues:

• The quality of URLs.
• The volume of URLs.
• The accessibility of your pages.

Let’s look at what I mean by that.

Quality of URLs

We know that Google considers the other pages on a website when deciding whether to crawl newly discovered pages.

Googlebot may decide a page isn’t worth the resources to crawl if it anticipates — because of this other content — its content will not be of high enough value to index. This can be due to:

• High volumes of on-site duplicate content.
• Hacked pages with poor-quality content.
• Internally created low-quality and spam content.

Poor-quality pages may have been intentionally created, either internally or by external bad actors. They may also be an unintended side effect of poor design and copy. And this can negatively impact your website’s crawl budget.

Volume of URLs

Your site may have more URLs than you realize, often due to common technical issues like faceted navigation and infinite URL creation.

Faceted navigation

Faceted navigation is usually found on ecommerce websites.

If you have a category page like www.example.com/cats/toys, you may have a filtering system to help users narrow down the products on that page.

If you want to narrow down the cat toy products in this fictitious pet store, you may select the “contains cat nip” filter. That may then yield a URL that looks something like this:

• www.example.com/cats/toys?contains=catnip

This is faceted navigation.

Now, consider what happens if the user wants to narrow the search down even further to toys that have feathers.

They might end up on a URL like this one:

• www.example.com/cats/toys?contains=catnip&design=feathers

What about if they want to sort the list by price?

Clicking the sort button may take them to yet another new URL:

• www.example.com/cats/toys?contains=catnip&design=feathers&sort=low

You can see how quickly additional URLs are created stemming from one category page. If Googlebot can find these pages, either through internal or external links, or perhaps they have been included in the XML sitemap, it may crawl them.

Pretty soon, instead of crawling your site’s 200 category pages and individual product pages, Googlebot might be aware of thousands of variants of the category pages. As these filtering systems lead to new URLs being created, they can all be crawled unless you stop the bots from doing so or they deem the pages too low-value to do so.

You can see how crawling the wrong pages due to a large volume of pages can quickly become an issue for your crawl budget.

Infinite URL creation

Events calendar. Book a table. Reserve a space. These types of date-based systems on websites that allow users to click through to future days or months can cause “bot traps.”

Picture an events calendar. It shows the whole month with a highlight on the days with events. It sits on the URL /events-calendar, and if you’re looking at the month of January 2027, the URL will contain /events-calendar/january-2027. This is pretty common practice.

If that calendar also has a button at the top that allows users to click through to the next month’s events, that wouldn’t be abnormal either.

Clicking once to view the next month’s events might take you to a URL containing /events-calendar/February. Clicking again, and you might end up on /events-calendar/march-2026. However, the real fun comes when there’s no limit to how far into the future you can click.

For example, you could click on “view next month’s events” enough times that you end up on /events-calendar/december-2086. If the calendar is set up in such a way that the “view next month’s events” link changes on each page to be the next URL in the sequence of months, then the search bots could also end up following the links all the way through to /events-calendar/december-2086 and beyond.

There’s almost certainly no useful content on the /events-calendar/december-2086 page. There probably haven’t been any events organized yet.

All of the resources wasted on those empty calendar pages could have been utilized by the bots on new products just uploaded to the site.

Accessibility of pages

Search bots may reduce the frequency of crawling a URL if it returns a server response code other than 200.

For example, a 4XX code indicates that the page cannot or should not be found, leading to less frequent crawling of that page. Similarly, if multiple URLs return codes like 429 or 500, bots may reduce the crawling of those pages and eventually drop them from the index altogether.

Redirects can also impact crawling, albeit to a smaller extent. However, excessive use, such as long chains of redirects, can have a cumulative effect over time.

Dig deeper: The ultimate guide to HTTP status codes for SEO

How to identify crawl budget problems

It’s impossible to determine whether your site is suffering from crawl budget issues just by clicking around on it. Instead, you’re going to have to roll up your sleeves and do a bit of investigative work to figure out how your site is functioning.

See what the search engines are reporting

The first step to identifying if search bots are having issues crawling your site is to use their webmaster tools. For example, look at the “Crawl stats” report in Google Search Console. This will help you identify whether a problem on your site may have caused Googlebot to increase or decrease its crawling activity.

Also, have a look at the “Page indexing” report. Here, you’ll see the ratio between your site’s indexed and unindexed pages. When looking through the reasons for not indexing pages, you may also see crawl issues reported, such as “Discovered – currently not indexed.”

This can be your first indication that pages on your site don’t meet Google’s crawling criteria.

Log files

Another way to tell if the search bots are struggling to crawl your pages as much as they’d like to is to analyze your log files.

Log files report any human users or bots that have “hit” your website. By reviewing your site’s log files, you can understand which pages haven’t been crawled by the search bots for a while. If these are pages that are new or updated regularly, this can indicate that there may be a crawl budget problem.

The industry’s trusted source for essential news and analysis.

Sign up for our newsletter to receive exclusive insights and updates directly to your inbox.

• Daily breaking news
• Expert analysis from industry veterans
• Join over 200k subscribers
• Covering SEO, PPC and AIO

---

How to fix crawl budget problems

Before trying to fix a crawl budget issue, you’ll want to make sure you have one.

That’s because some of the fixes I’m about to suggest are generally good practices for helping search bots focus on the pages you want them to crawl, but others are more serious and could have a negative impact on your crawling if not applied carefully. In other words, make sure you need the fix before you take your site apart to repair it.

Another word of warning

Carefully consider whether you’re addressing a crawling or indexing issue before making changes. I’ve seen many cases where pages are already in the index, and someone wants them removed, so they block crawling of those pages.

This approach won’t remove the pages from the index — at least not quickly. Worse, they sometimes double down by adding a noindex meta tag to the pages they’ve already blocked in the robots.txt file.

The problem? If crawling is blocked, search bots can’t access the page to see the noindex tag, rendering the effort ineffective.

To avoid such issues, don’t mix crawling and indexing solutions. Determine whether your primary concern is with crawling or indexing, and address that issue directly.

Fixing crawl budget issues through the robots.txt

The robots.txt is a very valid way of helping the search bots determine which pages you don’t want them crawling. The “disallow” command essentially prevents good bots from crawling any URLs that match the disallow command.

Bad bots can and do ignore the disallow command, so if you find your site is getting overwhelmed by bots of another nature, such as competitors scraping it, they may need to be blocked in another way.

Check if your robots.txt file is blocking URLs that you want search bots to crawl. I’ve used the robots.txt tester from Dentsu to help with this.

Fixing crawl budget issues at a server level

As I mentioned above, not all bots respect the robots.txt commands. Although the mainstream search engines do respect it, the newer LLM bots don’t always obey it. This means that if you’re having issues with bots overwhelming your site and impacting how easily human visitors and desired bots can access your site, you may need to be proactive.

There are three main ways of blocking bots at a server level: at the server level itself or even earlier in the tech stack — at the web application framework (WAF) or content distribution network (CDN).

 The WAF and the CDN both allow your website to stop bots before they hit your server. This means that if a particularly pesky bot is taking up too much of your server bandwidth, you can block it before it hits the site. This leaves more capacity for the legitimate search engine bots or human users to access the site without it pulling the server down.

Improving the quality and load speed of pages

If search bots struggle to navigate your site, speeding up page loading can help. Load speed is important for crawling because of both the time it takes for the server to respond to a search bot’s request and the time it takes to render a page.

You should test the templates used on URLs that aren’t being crawled regularly and see if they’re slow-loading. If they are, here’s a handy guide to everything you need to know to make them faster.

Another reason you may not see pages being crawled, even for the first time, is because of quality. To check if this is an issue, you should routinely audit the pages not being crawled and those that perhaps share the same subfolder but have been crawled. Make sure that the content on those pages isn’t too thin, duplicated elsewhere on the site or spammy.

Control crawling through robots.txt

You can stop search bots from crawling single pages and entire folders through robots.txt. Using the “disallow” command can help you decide which parts of your website you want bots to visit.

For example, you may not want the search bots wasting crawl budget on your filtered category page results. You could disallow the bots from crawling any page with the sorting or filtering parameters in the URL, like ?sort= or ?content=.

Consider nofollow links on internal links

Another way to prevent bots from crawling certain pages is to add the “nofollow” attribute to the link tag.

For example, in the case of the events calendar example earlier, each “View next month’s events” link could, and probably should, have the “nofollow” attribute added. That way, human visitors could still click the link, but bots wouldn’t be able to follow it.

Remember to add the “nofollow” attribute to the links wherever they appear on your site. If you don’t do this, or someone adds a link to a deeper page in the events calendar system from their own site, the bots could still crawl that page.

Navigating crawl budget for SEO success in 2027

Although most sites don’t need to be actively concerned with their crawl budget, it’s always wise to monitor it. You want to know for certain whether your site is affected by a reduced crawl budget. The issues that cause crawl budget issues may crop up unexpectedly.

With agentic AI and LLM bots accessing your site, it’s important to know which bots are consuming your server bandwidth and why. If you don’t keep an eye on which bots are successfully crawling your site, you may be giving away information to bots you don’t trust. This, in turn, may impact your visitors’ or search engines’ abilities to access your site if your server becomes overwhelmed.

Even if you’re not running out of crawl budget, there could also be issues preventing search bots from crawling pages as you intend them to. Or you could be allowing them to access pages you don’t want them to.

A core part of monitoring a website’s technical health is knowing how and when bots attempt to access it. Being aware of this through regular reporting is key to maintaining a healthy, accessible website.