Robots.txt Tester: 4 Ways to Check Your File

A robots.txt tester checks whether a given crawler may fetch a given URL under the rules in your robots.txt file, and which rule made the call. Four methods cover almost every case: the Search Console report, the URL Inspection live test, a web tester that accepts a custom file, and Google’s open-source parser run on your own machine.

This guide is for site owners and marketers who edit robots.txt now and then and want to be sure the edit did what they meant. It compares the four methods, shows the mistakes that testers catch (and the one they cannot), and ends with a short routine. For what to put in the file in the first place, see robots.txt for AI crawlers.

On this page

• What a robots.txt tester can and cannot tell you
• The four ways to test a robots.txt file
• How do you test a draft before you publish it?
• How to catch rules that silently block CSS and JavaScript
• How do you test AI crawler rules?
• Why a passing test can still be wrong
• How Google reads a broken or missing robots.txt
• A five-minute robots.txt testing routine
• Getting your robots.txt testing right

What a robots.txt tester can and cannot tell you

A tester answers one question: given these rules, is this URL allowed for this user-agent? It runs the same matching logic a crawler would, so it settles arguments about wildcards, rule order and which of two conflicting lines wins.

It does not tell you whether the page is indexed. Google’s robots.txt introduction is blunt that the file is “not a mechanism for keeping a web page out of Google” and that Google might still index a disallowed URL if it is linked from other places on the web. Blocking crawling and blocking indexing are separate jobs. The RFC 9309 standard adds that the rules are “not a form of access authorization”, so a green result is not proof a page is private.

Testers also read only the file. They cannot see your CDN, your firewall or your server’s response to a particular bot. We come back to that gap below.

The four ways to test a robots.txt file

Each method answers a different question, so use more than one. The diagram below shows what each is good for. The sections after it go through them one at a time.

Scroll the diagram sideways to see all of it.

1. The Search Console robots.txt report

The robots.txt report lists the robots.txt files Google found for your site, when each was last crawled, and any warnings or errors. It covers the top 20 hosts and is available for domain-level properties. A file is shown as Fetched, or as Not Fetched with a reason such as 404. The help page notes that having no robots.txt file is fine.

Its useful trick is the recrawl button. After you fix an error or make a critical change, you can request a recrawl of the file from the report, though Google says this does not guarantee an immediate recrawl of the unblocked URLs. If you have never set up the property, start with our Google Search Console guide.

2. The URL Inspection live test

For one page, the URL Inspection tool has a “Crawl allowed?” field. In a live test it reports whether your site will allow Google to crawl the page or block it with a robots.txt rule. Google’s URL Inspection help also warns that this is not the same as allowing indexing, and that a robots.txt block means “Indexing allowed?” always shows Yes, because Google cannot see a noindex rule on a page it cannot fetch.

Use it on your money pages and your templates, one of each type.

3. A web tester with a custom file

This is the method most people mean by “robots.txt tester.” The Tame the Bots checker, for example, runs a list of URLs against either the live file or a custom one, lets you pick Googlebot, Bingbot, GPTBot or your own agent, and reports which rule blocked each URL. It says it uses Google’s open-source parser, which matters because tools with their own matching code can disagree with Google on edge cases.

I have not tested every tool in the results for this keyword. The ones I would trust share three traits: they accept pasted rules, they name the matching rule, and they say which parser they run.

4. Google’s open-source parser, run locally

Google publishes its robots.txt parser and matcher as a C++ library, described in the README as slightly modified production code used by Googlebot. Build the robots_main tool and run it as robots_main <robots.txt path> YourBot <URL>. It exits with 0 for allowed and 1 for disallowed, which makes it easy to wire into a deploy check that fails if your homepage or key assets become blocked.

It is the least friendly option and the only one that can run unattended. If your robots.txt changes more than once a year, or several people edit it, it earns its keep.

How do you test a draft before you publish it?

Paste the draft into a tester that accepts a custom file, then run a fixed list of URLs against it. A robots.txt edit goes live the moment you upload it, and Google generally caches the file for up to 24 hours, so a mistake can cost you a day of crawling even after you fix it.

Build the URL list once and keep it:

• your homepage and one URL from each template (service, post, category, product)
• one URL you intend to block, to prove the block works
• your main CSS and JavaScript files
• your XML sitemap, which should never be blocked (see our XML sitemap guide)
• any URL with a query string or a file extension, if you use $ or * in rules

Google’s robots.txt specification is where the matching rules live, and testers are only as good as your understanding of them. The three that cause most surprises:

1. The most specific rule wins, judged by path length. When rules conflict, Google applies the least restrictive one.
2. Paths are case-sensitive. /fish does not match /Fish.asp. Field names such as Disallow are not.
3. Only one group applies to a crawler. If a User-agent: Googlebot group exists, Googlebot ignores the User-agent: * group entirely. Groups are not merged.

That third rule bites often. Say your * group disallows /private/ and someone later adds a Googlebot group for an unrelated reason. Googlebot now follows only its own group, and /private/ is crawlable again. A tester run as Googlebot shows it at once. A tester run as * does not.

The diagram below shows rule matching on a small file.

Scroll the diagram sideways to see all of it.

How to catch rules that silently block CSS and JavaScript

Test the asset URLs your pages load, not just the pages. A rule like Disallow: /assets/ is easy to write to hide a folder of internal files, and it also blocks the stylesheet and scripts that live in the same folder. The page still loads for people. A crawler that cannot fetch those files may see a different, emptier page.

Google’s guidance is to avoid blocking resources when their absence makes the page harder for Google’s crawler to understand, though you can block unimportant images, scripts or styles if the page works without them. The call is yours. The point is to make it on purpose.

A quick way to find the problem:

1. Open a key page in your browser, view source or the network panel, and note the CSS and JS file paths it loads.
2. Paste those paths, as full URLs, into a tester that accepts a list.
3. Run them as Googlebot and as your other important crawlers.
4. For any blocked asset, find the matching rule. Narrow the Disallow, or add a longer Allow for the file, as in the diagram above.
5. Run the URL Inspection live test on the page. If crawling is not allowed, the live test’s page fetch shows a failure.

This matters more for sites that render content with JavaScript, because a blocked script can blank the page for a crawler. We cover that side in JavaScript rendering and AI crawlers. Blocked assets also show up in the broader list of technical SEO issues worth auditing.

How do you test AI crawler rules?

Enter each AI user-agent separately and test your important URLs as that agent. OpenAI’s crawler documentation describes three independent agents. GPTBot crawls content that may train generative AI models, and disallowing it signals that your content should not be used for that training. OAI-SearchBot is used to surface websites in ChatGPT’s search features. ChatGPT-User fetches a page when a user asks, and OpenAI says that because these actions are user-initiated, robots.txt rules may not apply to it.

Two practical consequences for testing:

• A rule for one agent says nothing about another. Blocking GPTBot does not block OAI-SearchBot, so test both and read the result for each.
• Do not expect a test to cover ChatGPT-User. A tester can tell you what your file says to it. It cannot make OpenAI’s fetcher obey the file.

OpenAI also says it can take about 24 hours after a robots.txt update for its systems to adjust, so a change you make today is not a change you can verify in ChatGPT today. The decision of whether to allow training crawlers at all is a business one, covered in should you block AI crawlers.

The group rule from earlier applies here too. If you add a User-agent: GPTBot group and expect it to inherit your * rules, it will not. Repeat the rules you want in the specific group, then test as that agent.

Why a passing test can still be wrong

A tester proves what your file says, not what the bot experiences. Three gaps are common.

1. The CDN or firewall decides first. A bot-management rule can return a 403 before robots.txt is ever consulted. Your file can be perfectly open while the crawler is turned away at the edge. Check your CDN’s bot settings, then your server logs for 200 responses to the crawler’s user-agent.
2. The tool’s parser differs from Google’s. and *. Prefer tools that say they use Google’s parser, and confirm anything important with the URL Inspection live test.
3. You tested the wrong file. The report in Search Console lists files per host, so www, a bare domain and a subdomain can each have their own. Test the exact host the pages are served from.

User-agent strings can be spoofed, so a log line proves less than it looks like. For the whole picture of crawler access beyond robots.txt, see the technical SEO checklist for AI search.

How Google reads a broken or missing robots.txt

Google treats a missing file as permission and a failing server as a reason to pause. The diagram below summarizes what Google’s specification says for each response type.

Scroll the diagram sideways to see all of it.

Two details are worth knowing. First, a deleted file is not a block. If your robots.txt starts returning a 404, Google behaves as if the file never existed, which could expose paths you meant to keep out of crawling. Second, a 5xx is the worrying one. The cached copy can keep working for weeks, so a broken server may not show up in crawling right away, and the standard itself says a crawler must assume complete disallow while the file is unreachable. Google’s own docs describe the pause and fallback above, so the behavior differs by crawler. Do not assume every bot is as forgiving as Googlebot.

Google also enforces a size limit of 500 KiB and ignores content after it. RFC 9309 says parsers must handle at least 500 kibibytes. A file that large is unusual, but generated files with thousands of rules can reach it. crawl-delay is not supported by Google, and Sitemap lines are not used when rules are processed, so do not rely on either for control. If crawl waste is your actual worry, read our guide to crawl budget before you add more rules.

A five-minute robots.txt testing routine

Run these steps after every change, and quarterly even if nothing changed.

1. Open /robots.txt in a browser and read it top to bottom. It should be plain text, not an HTML error page.
2. Check the Search Console robots.txt report for a Fetched status and no new warnings.
3. Run your fixed URL list in a tester as Googlebot, then as Bingbot, GPTBot and OAI-SearchBot.
4. Confirm your CSS, JavaScript and sitemap URLs are allowed.
5. Run the URL Inspection live test on your homepage and one deep page.
6. Check your CDN bot settings and logs if any AI crawler is meant to be allowed.
7. Note the date and what you changed, so the next person can undo it.

A printable version of the file-level checks lives in our robots.txt checklist.

Getting your robots.txt testing right

The reliable pattern is simple: test the draft, publish, confirm what Google fetched, then spot-check real URLs. Most robots.txt disasters are not clever. They are a stray Disallow: / left from a staging site, a folder rule that catches the CSS, or a new user-agent group that quietly replaces the old one. A tester catches all three in seconds if you ask it the right question.

If robots.txt, rendering and crawl rules keep breaking when your site changes, the fix is usually in how the site is built and deployed, not in the file. Our web development service builds sites with crawl access, rendering and deploy-time checks designed in from the start.