Crawl Budget Optimization Checklist
For owners of large or frequently updated sites, such as ecommerce, news or listings. Smaller sites rarely have a crawl budget problem, so use this when new pages are slow to be discovered or crawled.
- critical
Why and how
- Why it matters
- Most sites under a few thousand pages do not need crawl budget work, and effort is better spent elsewhere.
- How to check or fix it
- Compare how long new pages take to appear in Search Console with your publishing rate. Review the Pages report for Discovered, currently not indexed.
- critical
Why and how
- Why it matters
- It shows how Googlebot spends requests and whether the server slows it down.
- How to check or fix it
- Open Settings > Crawl stats in Search Console. Look at average response time, host status and the breakdown by response and file type.
- critical
Why and how
- Why it matters
- Logs show crawling waste that no crawler simulation can, such as parameters and old URLs.
- How to check or fix it
- Filter logs for verified Googlebot using reverse DNS. Group requests by directory, status code and parameter.
- important
Why and how
- Why it matters
- Many URL variants of the same content consume requests and dilute signals.
- How to check or fix it
- Use canonical tags, consistent internal links and robots.txt rules for pure tracking or sort parameters.
- important
Why and how
- Why it matters
- Pages that look empty but return 200 keep getting crawled and tested for indexing.
- How to check or fix it
- Review Soft 404 in Search Console. Return a proper 404 or 410 or redirect to a genuinely relevant page.
- important
Why and how
- Why it matters
- Each hop is another request, and chains slow crawling.
- How to check or fix it
- Use the Redirect Chains report in Screaming Frog. Update templates and navigation to link directly to the 200 URL.
- important
Why and how
- Why it matters
- Unbounded URL spaces can absorb a large share of crawling activity.
- How to check or fix it
- Identify patterns in logs and disallow them in robots.txt, or stop generating the links.
- important
Why and how
- Why it matters
- Google reduces crawl rate when a server responds slowly or errors, so fast responses allow more crawling.
- How to check or fix it
- Check response time in the Crawl Stats report, add caching or a CDN, and fix recurring server errors.
- nice
Why and how
- Why it matters
- Crawl demand follows perceived importance and freshness, which links and sitemaps help signal.
- How to check or fix it
- Include only canonical indexable URLs, and link new important pages from high-traffic pages and hubs.
- nice
Why and how
- Why it matters
- Without before and after data you cannot tell which fix helped.
- How to check or fix it
- Record weekly Googlebot requests per section from logs or Crawl Stats and note each deployment.
Progress is saved in this browser only. Nothing is sent to a server.
Common questions
Does crawl budget matter for a small website?
Usually not. Google says crawl budget concerns mainly apply to very large sites or those with many frequently changing pages. Small sites are normally crawled fully.
Does noindex save crawl budget?
Not directly, because Google still has to crawl a page to see the noindex. Over time it may crawl such pages less often, but robots.txt or removing links is more effective for crawl control.
Do these next
Robots.txt Checklist
Check that your robots.txt is reachable, correctly formatted and blocking only what it should, without accidentally hiding pages from Google.
XML Sitemap Checklist
Build and check an XML sitemap that lists only indexable canonical URLs, stays within limits and is submitted and monitored in Search Console.