robots.txt for AI Crawlers: GPTBot, PerplexityBot and Google-Extended

robots.txt is how you tell AI crawlers what they may fetch, and it is only half the story. The other half is your CDN, which in many cases is now blocking those crawlers regardless of what your robots file says.

This guide lists the user-agents that matter, what each actually controls, a sample file, and how to verify that any of it is working.

The AI crawler user-agents that matter

The critical distinction is between training crawlers, which collect data that may be used to train future models, and retrieval crawlers, which fetch pages to answer a question right now. Blocking the first does not affect whether you can be cited today. Blocking the second does.

Two of these deserve emphasis.

Google-Extended does not affect your Google rankings. It is a separate token governing Gemini training and grounding only. Blocking it does not remove you from Search, and blocking Googlebot would remove you from Search entirely, which nobody should do by accident.

Perplexity-User generally does not apply robots.txt rules, because it acts on a live human request rather than crawling. That is by design and worth knowing before you assume a blanket block is airtight.

How to allow or block each crawler

The syntax is ordinary robots.txt. Each block names a user-agent and then allows or disallows paths.

# Allow retrieval so you can be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block training collection
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Rules are per user-agent and do not inherit from User-agent: * when a specific block exists for that agent, so be explicit rather than relying on fallthrough.

To protect only part of a site, disallow the path rather than the domain:

User-agent: GPTBot
Disallow: /clients/
Disallow: /research/

A sample robots.txt for AI visibility

This is the shape most service businesses want: fully open to retrieval, deliberate about training, with the sitemap declared.

User-agent: *
Allow: /

# Retrieval crawlers: allowed, because citation depends on them
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: a deliberate decision, documented in your notes
User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap-index.xml

Whether the training blocks say Allow or Disallow is a business decision, not a technical default. We work through it in should you block AI crawlers.

The part most guides miss: your CDN overrides this

Cloudflare began blocking known AI crawlers by default for new domains in July 2025, and other providers offer similar toggles. A site can have a perfectly permissive robots.txt and still be unreachable, with nothing in the robots file to indicate it.

So the check is three layers deep:

1. robots.txt rules per user-agent.
2. CDN or WAF bot-management settings, including any “block AI scrapers” switch.
3. Server logs, to confirm the fetches actually happen.

If you only ever check layer one, you can be invisible for months without knowing.

How to verify it is working

Look for the user-agent strings in your server logs, then verify the requesting IPs against the ranges the operators publish. User-agent strings are trivial to spoof, so unverified log lines prove less than they appear to.

What you want to see monthly: successful 200 responses to OAI-SearchBot and PerplexityBot on your important URLs. What should worry you: 403s, 429s, or complete absence.

The tradeoffs of blocking

Blocking training crawlers protects content from being absorbed into future models. Blocking retrieval crawlers removes you from the answers where buyers are now shortlisting vendors. Those are very different decisions with very different costs, which is why a blanket Disallow for anything AI-shaped is usually the wrong call for a business that needs to be found.

For publishers whose product is the content itself, the calculus differs. The full argument is in should you block AI crawlers.

Honest limits of robots.txt

• It is voluntary. Well-behaved crawlers honour it; others do not.
• It is not access control. Anything reachable without authentication is reachable.
• It does not retroactively remove content already collected.
• Some agents ignore it by design, notably live user-requested fetchers.
• It says nothing about your CDN, which may be the actual gatekeeper.

For anything you genuinely must protect, use authentication or server-level rules, not robots.txt alone.

Where robots.txt fits in your stack

robots.txt is the front door of technical GEO: it decides who gets in. But being allowed in only matters if your pages are fast, server-rendered, and structured so crawlers can understand them. The rest of the list is in the technical SEO checklist for AI search, and the rendering half is in JavaScript rendering and AI crawlers.

Not sure how AI engines currently see, or fail to see, your site? Request an AI visibility audit and we will map exactly which crawlers you are allowing, blocking, and losing citations to, including at the CDN layer.