Crawl Budget: Who Needs to Care and What to Fix
Crawl budget is the set of URLs on your site that Google can and wants to crawl, determined by how much crawling your server handles comfortably (crawl capacity) and how much Google wants to fetch (crawl demand). For most sites it is not a problem worth solving. For very large or fast-changing sites, it decides how quickly new and updated pages reach the index, and every wasted fetch on a duplicate or filter URL delays a page that matters.
This guide is for owners of large ecommerce, publishing, marketplace and listings sites, and for anyone told by an audit tool that they have a “crawl budget issue”. It is based on Google’s crawl budget documentation, which Google substantially updated in July 2026. It covers who should care, how capacity and demand work, what wastes crawling, how AI crawlers add load, how to read the Crawl Stats report, and how to earn more crawling.
On this page
- Do you need to worry about crawl budget?
- Crawl capacity and crawl demand
- What wastes crawl budget
- Faceted navigation: the biggest drain
- How do AI crawlers affect crawl budget?
- How to read the Crawl Stats report
- How do you get more crawl budget?
- Keeping crawl budget focused on pages that matter
Do you need to worry about crawl budget?
Probably not, unless your site is very large or changes very fast. Google’s crawl budget guide opens by saying that if your site “doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide”. It names three types of site the guide is really for:
- Large sites with more than 1 million unique pages whose content changes about once a week.
- Medium or larger sites with more than 10,000 unique pages whose content changes daily.
- Sites with a large share of URLs listed in Search Console as “Discovered, currently not indexed”.
Google adds that these numbers are “a rough estimate”, not exact thresholds. For everyone else, it says keeping your sitemap up to date and checking the Page indexing report regularly “is adequate”.
The third group is the useful test. Search Console’s help for the Page indexing report explains that “Discovered, currently not indexed” typically means Google wanted to crawl the URL “but this was expected to overload the site”, so it rescheduled. A growing pile of those URLs is the clearest sign crawl capacity is the constraint. A pile of “Crawled, currently not indexed” is a different problem: Google fetched the page and chose not to index it, which is about quality, not budget.
Crawl capacity and crawl demand
Crawl budget is the overlap of two limits: what your server can take and what Google wants. Google’s guide defines each, and treats each hostname as a separate site with its own budget, so www.example.com and shop.example.com are budgeted separately.
Scroll the diagram sideways to see all of it.
Crawl capacity limit, which Google also calls hostload, caps “the total amount of time your server spends holding connections open for Google”. The July 2026 version of the guide states that “every site starts with the same default, conservative crawl capacity limit” and that Google raises it over time if there is demand and the site stays healthy. Health means consistent responses and stable or improving latency and time to first byte. Slower responses, 5xx errors or 429 rate-limit responses push the limit down.
Crawl demand for Googlebot depends on “a site’s size, update frequency, page quality, and relevance, compared to other sites”. The factors you can influence are perceived inventory (how many URLs Google knows about, which Google calls “the factor that you can positively control the most”), popularity and staleness. Site moves temporarily raise demand while Google reprocesses the new URLs.
The two interact in one direction that matters: “Even if the crawl capacity limit isn’t reached, if crawl demand is low, Google will crawl your site less.” A fast server doesn’t earn crawling on its own.
What wastes crawl budget
Crawl budget is wasted on URLs that shouldn’t be crawled at all, and the biggest source is almost always duplication. Google’s best practices map directly onto the common problems:
| Waste | Why it costs crawling | Google’s recommended fix |
|---|---|---|
| Duplicate URLs | Each copy must be fetched before it can be grouped | Consolidate duplicates; focus crawling on unique content |
| Faceted and sorted URLs | Near-infinite combinations of the same items | robots.txt Disallow on the parameter patterns |
| Soft 404 pages | They “will continue to be crawled” | Return a real 404 or 410 |
| Removed pages returning 200 | Stay in the crawl queue | 404 or 410; a 404 is “a strong signal not to crawl that URL again” |
noindex pages you never want | Google still requests them, then drops them | robots.txt block instead |
| Long redirect chains | “A negative effect on crawling” | One hop to the final URL |
| Slow pages | Less content read per visit | Faster server responses; support 304 Not Modified |
Two points in the guide contradict common advice. First, noindex doesn’t save crawling: “Google will still request, but then drop the page when it sees a noindex meta tag or header”. Second, don’t use robots.txt to temporarily shift budget to other pages, because “Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s crawl capacity limit”.
The wider cleanup of duplicate URLs is covered in our guide to duplicate content SEO, and chains in redirects and SEO. A clean XML sitemap that lists only canonical, indexable URLs helps too: the guide says Google reads sitemaps regularly and recommends the <lastmod> tag for updated content.
Faceted navigation: the biggest drain
Faceted navigation is the most common reason a large site has a genuine crawl budget problem. Google’s guide to managing faceted navigation URLs says parameter-based filters “can generate infinite URL spaces”, which leads to overcrawling, because crawlers can’t tell filter URLs are useless until they have fetched them, and slower discovery of new pages.
Google’s recommendations, in order of effectiveness:
- Block filter URLs in robots.txt if you don’t need them indexed. Its example disallows each filter parameter and allows only the unfiltered listing.
- Use URL fragments (
#color=green) for filters, since Google generally ignores fragments when crawling. - Canonical tags and
rel="nofollow"on filter links are “generally less effective in the long term”. Canonicals may reduce crawl of filtered URLs “over time”; nofollow only works if every link to the filtered URL carries it.
If some filtered pages deserve to rank, such as a category plus one popular attribute, give those a clean static URL and keep the rest blocked. Google also asks that filter combinations with no results return a 404 rather than an empty page. Crawl rate spikes from calendars and dynamic ad targets follow the same logic, according to Google’s page on reducing crawl rate.
How do AI crawlers affect crawl budget?
AI crawlers don’t count against Google’s crawl budget, but they compete for the same server capacity that Google’s budget depends on. Google’s guide says the crawl capacity limit “is shared across all crawlers”, meaning all of Google’s own crawlers, so “high demand from one crawler can reduce the capacity available for others”. Other companies’ bots are outside that accounting. Their load still lands on your server, and Google lowers its capacity limit when response times rise or errors appear.
The volume has grown quickly. Cloudflare’s analysis From Googlebot to GPTBot found AI and search crawler traffic across its network grew 18% from May 2024 to May 2025, with Googlebot requests up 96%, GPTBot up 305% and ChatGPT-User up 2,825%. The chart shows each crawler’s share of that traffic in May 2025.
Scroll the diagram sideways to see all of it.
For most sites this load is harmless. For large sites on tight hosting, it is worth watching. Check your server logs by user agent, verify that bots claiming to be Googlebot really are, using the reverse DNS and IP range methods in Google’s guide to verifying Google’s crawlers, and decide deliberately which AI crawlers to allow. Our guides to robots.txt for AI crawlers and whether to block AI crawlers cover that decision.
How to read the Crawl Stats report
The Crawl Stats report in Search Console is the closest thing to a crawl budget dashboard, and it is only available for root-level properties. Read it in this order:
Scroll the diagram sideways to see all of it.
A few details from Google’s help page make the numbers easier to interpret:
- Host status covers the past 90 days. A red status means a significant availability issue in the last week; the charts mark a problem when a metric crosses a threshold, for example DNS resolution failing for more than 5% of requests on a given day.
- robots.txt availability can stop crawling. If Google can’t get a successful robots.txt response, it stops crawling for the first 12 hours, then uses the last good copy for up to 30 days. A 404 for robots.txt counts as success; a 429 or 5xx doesn’t.
- Requests are not pages. Page resources such as CSS and images count, redirects count per hop, and duplicate requests count individually.
- Discovery versus refresh shows whether Google is finding new URLs or rechecking known ones. After publishing a batch of content, you should see a bump in discovery.
Server logs complete the picture, because they show every bot, not just Google’s. Our Google Search Console guide covers the rest of the reports.
How do you get more crawl budget?
You get more crawl budget by raising both limits: more server capacity and more reasons for Google to want your pages. Google’s guide names exactly two routes:
- Add server resources if Google can’t crawl because of capacity on your side, for example if the URL Inspection tool reports “Hostload exceeded”.
- Improve content quality for the product you’re targeting. For Google Search, that includes “popularity, overall user value, content uniqueness, and serving capacity”.
Speed feeds the first route. Google says faster loading and rendering mean “we might be able to read more content from your site”, and that supporting 304 Not Modified lets Google reuse cached copies. Our guide to site speed optimization covers the server side.
The opposite lever exists for emergencies. If Google’s crawling is overloading your infrastructure, its crawl rate guide says you can return 500, 503 or 429 for a short period, such as “a couple of hours, or 1-2 days”, and Google will slow down across the whole hostname. Use it sparingly: Google warns that new pages will be discovered more slowly and removed pages may linger in the index.
Keeping crawl budget focused on pages that matter
Crawl budget work is mostly subtraction. Stop Google finding URLs that shouldn’t exist, return honest status codes for pages that are gone, keep one clean URL per piece of content, and make the server fast enough that Google’s crawlers never have a reason to back off. For the few sites where it matters, those changes speed up indexing of new pages more than any submission tool.
For everyone else, the right move is to confirm you don’t have a problem and then stop worrying: new pages indexed within days, few “Discovered, currently not indexed” URLs, and a green host status. The broader set of checks is in our technical SEO checklist for AI search. If you run a large catalog or listings site and want the crawl data, logs and index coverage analyzed together, that analysis is part of our AI visibility audit.
Frequently asked questions
What is a crawl budget?
Crawl budget is the set of URLs on a site that Google can and wants to crawl. It combines the crawl capacity limit, how much crawling your server can handle without slowing down, with crawl demand, how much Google wants to crawl based on the site's size, freshness, quality and popularity.
How do I check my crawl budget?
There is no single crawl budget number to look up. Use Search Console's Crawl Stats report, available for root-level properties, to see total crawl requests, average response time, host status over the past 90 days, and breakdowns by response code, file type, purpose and Googlebot type. Server logs give the complete picture.
Does crawl budget matter for small websites?
Usually not. Google says its crawl budget guide is intended mainly for sites with more than a million unique pages that change weekly, sites with more than 10,000 pages that change daily, or sites with many URLs stuck as Discovered, currently not indexed. If new pages are crawled the day they are published, you don't need to worry.
What are crawl requests?
Crawl requests are the individual fetches Google's crawlers make to your server, including pages, images, CSS and JavaScript files, and each hop of a redirect chain. Search Console counts duplicate requests for the same URL separately, so crawl requests are not the same as unique pages crawled.
Does noindex save crawl budget?
No. Google still has to request a page to see its noindex rule, so the crawl is spent anyway. Google's guidance is to use robots.txt to block URLs you don't want crawled, to return 404 or 410 for removed pages, and to consolidate duplicates.
Do AI crawlers use up my Google crawl budget?
Not directly: Google's crawl budget covers Google's own crawlers, which share one crawl capacity limit per host. But AI crawlers from other companies hit the same server, and Google lowers its crawl capacity when your site slows down or returns server errors, so heavy bot traffic can reduce Google's crawling indirectly.