Robots.txt Checklist

For site owners and SEOs who maintain a robots.txt file or are about to edit one. Use it after a launch or migration, and whenever crawling or indexing looks wrong.

11 checks ~30 min beginner 3 critical
0% 0 of 11 done
  1. critical
    Why and how
    Why it matters
    Crawlers only look at /robots.txt on the exact host. A 5xx response can make Google pause crawling, while a 404 is treated as no restrictions.
    How to check or fix it
    Open yourdomain.com/robots.txt in a browser and check the status in DevTools Network. Do this for every subdomain separately.
  2. critical
    Why and how
    Why it matters
    A staging rule that reaches production can stop all crawling and drop the site from results over time.
    How to check or fix it
    Read the User-agent: * group and look for a bare Disallow: /. Test key URLs in Search Console and a crawler set to respect robots.txt.
  3. critical
    Why and how
    Why it matters
    Google renders pages, and blocked resources can make it see a broken or empty page.
    How to check or fix it
    Use the URL Inspection live test and check Page resources for blocked files. Remove Disallow rules covering /assets/ or similar folders.
  4. important
    Why and how
    Why it matters
    Google parses only the first 500 KiB and ignores the rest. Bad encoding can break rules.
    How to check or fix it
    Check the Content-Type header is text/plain and check the file size. Trim redundant rules if it is large.
  5. important
    Why and how
    Why it matters
    Typos and misplaced directives are ignored silently, so a rule you think is active may do nothing.
    How to check or fix it
    Use only User-agent, Allow, Disallow and Sitemap. Review the robots.txt report in Search Console for parse errors and warnings.
  6. important
    Why and how
    Why it matters
    A blocked URL can still be indexed from links, without a snippet, and Google cannot see a noindex on a blocked page.
    How to check or fix it
    To remove a page from results, allow crawling and use a noindex meta robots tag or X-Robots-Tag. Use Disallow for crawl waste only.
  7. important
    Why and how
    Why it matters
    Rules are prefix matches, so Disallow: /shop can also block /shop-sale. Wildcards can overreach.
    How to check or fix it
    Use * for wildcards and $ for end of URL. Test sample URLs in a parser or a crawler set to use your robots.txt.
  8. important
    Why and how
    Why it matters
    It helps crawlers find your sitemap without relying on Search Console alone, including Bing and others.
    How to check or fix it
    Add Sitemap: https://www.example.com/sitemap.xml on its own line. Multiple Sitemap lines are allowed.
  9. important
    Why and how
    Why it matters
    AI bots have their own user-agent tokens, and a blanket rule may allow or block them unintentionally.
    How to check or fix it
    List tokens such as GPTBot, ClaudeBot and PerplexityBot and set Allow or Disallow per bot. Remember robots.txt is voluntary.
  10. nice
    Why and how
    Why it matters
    An accidental overwrite during a release is a common cause of sudden deindexing.
    How to check or fix it
    Store the file in your repo and add a deployment check that fails if Disallow: / appears on production.
  11. nice
    Why and how
    Why it matters
    It shows fetch status and when Google last read the file, so you can confirm changes were picked up.
    How to check or fix it
    Open Settings > Crawl stats > robots.txt report. Request a recrawl after significant edits.

Common questions

Does robots.txt stop a page from appearing in Google?

No. It stops crawling, but the URL can still be indexed if other pages link to it. To keep a page out of results, use a noindex directive on a crawlable page or password protection.

Should I block my admin and search result pages in robots.txt?

Blocking low-value areas like internal search results or cart pages can save crawling effort. Never rely on it to hide private content, since the file is public.

Do these next

XML Sitemap Checklist

Build and check an XML sitemap that lists only indexable canonical URLs, stays within limits and is submitted and monitored in Search Console.

10 checks~40 min

Crawl Budget Optimization Checklist

Find where Googlebot wastes time on your site and fix duplicate, low-value and slow URLs so important pages are crawled promptly.

10 checks~90 min

AI Crawler Access Checklist

Decide which AI bots may crawl your site, set robots.txt rules per user agent, and verify that firewalls are not silently blocking them.

11 checks~45 min

Browse all 33 checklists →

Rather have it done for you?

We run this checklist against your site, fix what fails and send a plain-language report.

Start a project