AI Crawler Access Checklist
For site owners and developers who need to control how GPTBot, ClaudeBot, PerplexityBot and similar crawlers use their content. Use it before changing robots.txt or bot protection, and again after any CDN or security change.
- critical
Why and how
- Why it matters
- Vendors use different bots for training and for live search, so one blanket rule blocks things you may want to allow.
- How to check or fix it
- Write a short policy: for example allow search bots so you can be cited, and decide on training bots by business risk.
- critical
Why and how
- Why it matters
- Current agents include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and Google-Extended.
- How to check or fix it
- Check each vendor's official crawler documentation for current names and purposes, since they change.
- critical
Why and how
- Why it matters
- A broad Disallow under User-agent: * can block AI search bots you meant to allow.
- How to check or fix it
- Open /robots.txt and read each group. Note which user agents match which rules, remembering that a specific group overrides the wildcard group.
- important
Why and how
- Why it matters
- Explicit groups make your intent unambiguous and survive later edits to the wildcard section.
- How to check or fix it
- Add blocks such as User-agent: GPTBot with Disallow: / or Allow: /, one group per bot, and keep a comment explaining each choice.
- important
Why and how
- Why it matters
- It is a control token for use in some Google AI products, separate from Googlebot crawling and indexing for Search.
- How to check or fix it
- Add User-agent: Google-Extended only if you want to opt out of those uses. Leave Googlebot rules unchanged.
- important
Why and how
- Why it matters
- Robots.txt only states policy. A WAF or bot manager can still return 403 or a challenge page to crawlers.
- How to check or fix it
- Run curl -I -A with each bot string against your homepage and a deep URL and confirm 200 for allowed bots.
- important
Why and how
- Why it matters
- Some CDN and security products block AI bots by default or via a managed rule, overriding your robots.txt intent.
- How to check or fix it
- Review the bot management or firewall rules in your CDN dashboard and confirm allowed bots are not challenged.
- important
Why and how
- Why it matters
- Anyone can spoof a user agent string, so allowlists based on the string alone can be abused.
- How to check or fix it
- Use the IP range lists or reverse DNS methods that vendors publish and apply them in your firewall allowlist.
- important
Why and how
- Why it matters
- Even allowed crawlers cannot read content that needs a session or client-side rendering they do not run.
- How to check or fix it
- Fetch the page with curl and confirm your main text appears in the returned HTML.
- nice
Why and how
- Why it matters
- Logs show which bots actually visit, how often, and whether they hit errors or rate limits.
- How to check or fix it
- Filter access logs by the bot user agents and review status code counts and top requested paths.
- nice
Why and how
- Why it matters
- New agents appear and existing ones change purpose, so a static rule set drifts out of date.
- How to check or fix it
- Set a calendar reminder to re-read vendor crawler docs and compare against your robots.txt.
Progress is saved in this browser only. Nothing is sent to a server.
Common questions
Should I block GPTBot?
GPTBot is OpenAI's training crawler, and blocking it does not stop ChatGPT search from citing you, because that uses OAI-SearchBot. Block GPTBot if you object to training use, and keep OAI-SearchBot allowed if you want citations.
Does robots.txt actually stop AI crawlers?
Reputable crawlers from major vendors state they follow robots.txt, but it is a request, not an enforcement mechanism. For content you must protect, use authentication or server-side blocking as well.
Do these next
Robots.txt Checklist
Check that your robots.txt is reachable, correctly formatted and blocking only what it should, without accidentally hiding pages from Google.
llms.txt Checklist
Create, publish and maintain an llms.txt file correctly, and set honest expectations about what the proposed standard does today.
AI SEO (Generative Engine Optimization) Checklist
Step through the technical, content and brand work that makes your pages easier for AI search tools to find, understand and cite.