robots.txt for AI Crawlers: GPTBot, PerplexityBot and Google-Extended
Yes, you can control AI crawlers with robots.txt — the same file that has guided search engines for decades. You add a User-agent block naming the AI crawler and a Disallow rule, and well-behaved bots honor it. The catch: robots.txt is voluntary, so it only works for crawlers that choose to obey it, and the AI-crawler landscape is new and changing.
Here’s exactly which user-agents exist, how to allow or block each, and the tradeoffs of doing so.
The AI crawler user-agents that matter
Different companies run different bots, and some companies run several with different jobs. That distinction is the whole game — one token may feed model training while another only fetches pages to cite in live answers.
| User-agent | Operator | What it’s for |
|---|---|---|
GPTBot | OpenAI | Fetching content that may be used to train models |
OAI-SearchBot | OpenAI | Surfacing and linking sites in ChatGPT search |
PerplexityBot | Perplexity | Indexing pages to cite in Perplexity answers |
Google-Extended | Opting in/out of Gemini and Vertex AI training (not classic Search) | |
ClaudeBot / anthropic-ai | Anthropic | Fetching content associated with Anthropic’s crawling |
A key nuance: Google-Extended is not a crawler with its own user-agent that fetches pages. It’s a token you use in robots.txt to control whether Google may use already-crawled content for its generative AI products. Blocking it does not affect how you rank in classic Google Search.
How to allow or block each crawler
The syntax is identical to standard robots.txt. You name a User-agent, then list Disallow or Allow rules.
To block OpenAI’s training crawler across your whole site:
User-agent: GPTBot
Disallow: /To allow ChatGPT search to cite you while still blocking training:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /To opt out of Google’s generative AI training but keep normal Search indexing:
User-agent: Google-Extended
Disallow: /To block Perplexity’s crawler:
User-agent: PerplexityBot
Disallow: /A sample robots.txt for AI visibility
Most sites that want to be found and cited by AI engines should welcome the search-oriented bots and think carefully before blocking anything. A permissive, GEO-friendly file looks like this:
# Allow standard search engines
User-agent: *
Allow: /
# AI answer engines — allow citation
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Point crawlers to your sitemap
Sitemap: https://yourdomain.com/sitemap.xmlIf you also publish an llms.txt, it lives alongside this file. The two are complementary — one governs access, the other curates content. We cover the second one in what llms.txt is and whether it’s worth building.
The tradeoffs of blocking
Blocking AI crawlers feels protective, but it cuts both ways. Think through what you actually want:
- Block training, keep citations. A common, sensible stance: disallow
GPTBotandGoogle-Extended, but allowOAI-SearchBotandPerplexityBotso you can still be cited in live answers. - Block everything. You protect content from training corpora, but you also disappear from AI answers entirely. If getting cited by ChatGPT and Perplexity is a goal, full blocking works against you.
- Allow everything. Maximum visibility in AI answers, at the cost of your content potentially feeding training. For most marketing sites, this is the pragmatic default.
There is no universally correct answer. A publisher protecting premium journalism will choose differently than a services business that wants to be quoted.
Honest limits of robots.txt
Three things to keep in mind so you don’t over-trust this file:
- It’s voluntary.
robots.txtis a convention documented in Google’s Search Central robots.txt guide, not an enforcement mechanism. Bots that ignore it face no technical barrier. - The space is new. User-agent tokens and their exact behaviors change. Verify current tokens against each operator’s official docs (for example OpenAI’s GPTBot documentation) before relying on them.
- It controls crawling, not content already out there. Blocking a bot today doesn’t retract content already trained on or cached.
For anything you truly must protect, use authentication or server-level rules — not robots.txt alone.
Knowing the syntax is the easy half. Deciding which bots to allow is the commercial call, and we work through it in should you block AI crawlers.
Where this fits in your stack
robots.txt is the front door of technical GEO: it decides who gets in. But being allowed in only matters if your pages are fast, server-rendered, and structured so crawlers can actually understand them. That’s the work behind our technical GEO services and web development — access is step one, comprehension is the rest.
Not sure how AI engines currently see — or don’t see — your site? Get an AI visibility audit and we’ll map exactly which crawlers you’re allowing, blocking, and losing citations to.
Frequently asked questions
What is the user-agent for OpenAI's crawlers?
OpenAI operates more than one. GPTBot is used to fetch content that may train models, and OAI-SearchBot is used to surface and link sites in ChatGPT search results. They are separate tokens, so you can allow one and block the other in robots.txt.
How do I block AI crawlers with robots.txt?
Add a User-agent block naming the crawler (for example GPTBot) followed by Disallow: / to block your whole site, or a specific path to block part of it. robots.txt is a voluntary standard, so it works only for crawlers that choose to honor it.
Does blocking GPTBot remove me from ChatGPT?
Not necessarily. GPTBot governs OpenAI's training and general fetching; ChatGPT's live search surfacing uses OAI-SearchBot. If you want to appear in ChatGPT search but not be used for training, allow OAI-SearchBot and disallow GPTBot.
Is robots.txt legally binding on AI crawlers?
No. robots.txt is a widely respected convention, not a legal contract or a technical barrier. Well-behaved crawlers follow it; it does not physically prevent access. For hard control, use authentication or server-level blocking.
Where does the robots.txt file go?
At the root of your domain, reachable at https://yourdomain.com/robots.txt. On a static site, place it in your public/ folder so it deploys unchanged.