ClaudeBot: What It Crawls and How to Control It

ClaudeBot is the web crawler Anthropic uses to collect public web content that could help train its Claude models, and it is only one of three Anthropic bots. The other two, Claude-User and Claude-SearchBot, fetch pages for live answers and for search, and they are controlled separately.

This guide is for site owners and marketers who saw “ClaudeBot” in a log, a CDN dashboard or a security alert and want to know what it is and what to do about it. It covers what each bot does, the robots.txt rules, how to tell from your server logs which bot actually visited, and what robots.txt cannot do.

On this page

• What is ClaudeBot?
• ClaudeBot, Claude-User and Claude-SearchBot compared
• Is ClaudeBot the same as ClawdBot or Moltbot?
• How do you control ClaudeBot in robots.txt?
• How to confirm in your logs which Claude bot visited
• Why blocking ClaudeBot by IP address backfires
• How do you slow ClaudeBot down?
• Getting ClaudeBot access right for AI visibility

What is ClaudeBot?

ClaudeBot is an automated crawler that Anthropic runs to gather public web content for training its generative AI models. Anthropic’s help article on its crawlers describes it as collecting web content that could help train those models. Blocking it, per the same article, signals that your site’s future materials should be excluded from training datasets.

Three details from that article set expectations:

• The bots identify themselves, so you can see them in your logs.
• They honor “do not crawl” signals in robots.txt.
• They respect anti-circumvention technology. Anthropic says it will not try to get past CAPTCHAs.

What the article does not give you is a promise about the past. Opting out covers future collection, so it is a go-forward control, not a deletion request.

If you have already read our guide to robots.txt for AI crawlers, the ClaudeBot row there is the short version. This post goes deeper on the one vendor where the three bots are easiest to confuse.

ClaudeBot, Claude-User and Claude-SearchBot compared

Anthropic lists three bots and a different consequence for blocking each one, so “block Claude” is really three decisions. The diagram below shows them side by side, with the two that touch search visibility highlighted.

Scroll the diagram sideways to see all of it.

Cloudflare’s verified bot categories use similar vocabulary from the network side. It defines an AI Crawler as one that “crawls websites for content that is used for training AI models,” and an AI Assistant as an “automated AI bot driven by user action.” Cloudflare’s page does not mention the Claude bots by name, so treat that as shared language, not a mapping. In your own logs, ClaudeBot is the training-style visitor and Claude-User is the assistant-style one.

For how Claude web search itself decides to search and cite, our write-up of Claude web search covers the product side. This post stays on the crawler side.

Is ClaudeBot the same as ClawdBot or Moltbot?

No. ClaudeBot is Anthropic’s crawler, while “Clawdbot” was a separate, viral open-source AI agent that people ran themselves. When we checked the live Google results for “claudebot” in the United States, the page was dominated by videos and articles about that agent, not about the crawler. That is the main reason this query is confusing.

Business Insider reported on January 27, 2026 that the agent, created by developer Peter Steinberger, was renamed Moltbot after Anthropic asked for a change. It had been named after a character users see while Claude Code reloads. Anthropic trademarked the Claude name and the mascot logo, according to the same report. Search results since then refer to the agent as OpenClaw.

For a site owner, the practical point is simple. If your log shows a user-agent containing ClaudeBot, that is the Anthropic crawler family, and the robots.txt controls below apply. If you are reading about a “ClaudeBot” that books dinner reservations or reads email, that is the agent, and nothing you put in robots.txt will affect it. An agent running on someone’s own machine is not a crawler you can configure.

How do you control ClaudeBot in robots.txt?

You control ClaudeBot with a standard robots.txt group at the top level of your domain, and Anthropic’s own example for blocking it is two lines. Anthropic’s article says to add the rule to the robots.txt in your top-level directory and to do so for every subdomain you want to opt out:

User-agent: ClaudeBot
Disallow: /

That is the only example the article gives. It names ClaudeBot and does not show syntax for Claude-User or Claude-SearchBot. The same pattern should work for them, because they are listed as separate bots, but that is our inference from the standard, not a line from Anthropic, so test it (see the log section below).

The standard itself is RFC 9309. Three rules from it matter here:

1. User-agent matching is case-insensitive, so claudebot and ClaudeBot select the same group.
2. When several path rules match, the most specific one, the one with the longest matching path, wins. If an allow and a disallow rule are equivalent, the allow rule is used.
3. The standard states that “these rules are not a form of access authorization.”

The third rule is the one to remember. robots.txt is a request, and well-behaved crawlers follow it. It is not a lock.

Here is a file for the most common goal for a business that wants to be found: stay out of training, stay open to search and retrieval. It follows the pattern from our robots.txt for AI crawlers guide.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Whether you should opt out of training at all is a business call, covered in should you block AI crawlers. The diagram below turns the three common goals into rules.

Scroll the diagram sideways to see all of it.

OpenAI’s documentation takes a similar split approach. Its bots page says each setting is independent of the others, and it adds that for ChatGPT-User, “robots.txt rules may not apply” because a user initiated the action. Anthropic’s article makes no such exception for Claude-User, but we would still not assume a user-triggered fetcher behaves like a scheduled crawler. Check the logs.

How to confirm in your logs which Claude bot visited

To confirm which Claude bot visited, filter your access log by bot name, then check each visitor’s IP address against Anthropic’s published list. The name tells you which bot a visitor claims to be, and the IP check tells you whether to believe it.

The two steps do different jobs:

1. The user-agent token separates the three bots. Search for ClaudeBot, Claude-User and Claude-SearchBot as case-insensitive text.
2. The IP check separates real visits from spoofed ones. Anthropic publishes its crawler addresses at claude.com/crawling/bots.json. When we fetched it, the file held a creation time of 2026-10-07 and a single flat prefixes list of 38 IPv4 CIDR entries, mostly single addresses, with one /22 block. It has no per-bot labels.

That second point has a consequence worth stating plainly. Because the list is not split by bot, an IP match tells you “this came from Anthropic,” and only the user-agent tells you which of the three it was. Neither check alone is enough.

To see a count per bot from a standard access log:

grep -ioE 'claudebot|claude-user|claude-searchbot' access.log | sort | uniq -c

To add the IP check, this short Node script reads the log and Anthropic’s JSON file (saved locally), and tallies each bot as verified or unverified. We tested it on a small sample log with one spoofed address. It handles IPv4 only, which matches the file as we fetched it, and it assumes the visitor IP is the first field, as in common web server formats.

import { readFileSync } from 'node:fs';
const [logFile, ipFile] = process.argv.slice(2);
const toInt = ip => ip.split('.').reduce((n, o) => n * 256 + Number(o), 0);
const nets = JSON.parse(readFileSync(ipFile, 'utf8')).prefixes
  .filter(p => p.ipv4Prefix)
  .map(p => {
    const [ip, bits] = p.ipv4Prefix.split('/');
    return { base: toInt(ip), size: 2 ** (32 - Number(bits)) };
  });
const inRange = ip => nets.some(n => toInt(ip) >= n.base && toInt(ip) < n.base + n.size);
const BOTS = ['ClaudeBot', 'Claude-User', 'Claude-SearchBot'];
const tally = {};
for (const line of readFileSync(logFile, 'utf8').split('\n')) {
  const bot = BOTS.find(b => line.toLowerCase().includes(b.toLowerCase()));
  if (!bot) continue;
  const key = `${bot} ${inRange(line.split(' ')[0]) ? 'verified' : 'UNVERIFIED'}`;
  tally[key] = (tally[key] || 0) + 1;
}
console.table(tally);

The diagram below shows the sequence, and what each outcome means.

Scroll the diagram sideways to see all of it.

Step three is where a robots.txt change proves itself. After you edit the file, wait for the next visits and look for a request to /robots.txt from the bot, followed by no requests to the paths you disallowed. A bot that reads the file and then fetches a disallowed path is a real finding. A bot that never fetches it at all, because a firewall or CDN rule turned it away first, is a different one, and the fix is on the network side. Our technical SEO audit checklist has a crawlability section that covers that layer, and the older technical SEO checklist for AI search covers the rendering side.

Why blocking ClaudeBot by IP address backfires

Blocking ClaudeBot by IP address is the wrong tool because Anthropic says it may not give you a lasting opt-out. The article warns that IP blocking “may not work correctly or persistently guarantee an opt-out,” because it stops the bot from reading your robots.txt in the first place. A bot that cannot fetch the file cannot see that you said no.

There is a second problem. Anthropic’s list is one flat set of ranges for all its bots. If you block those ranges to stop training collection, you also block the two bots that matter for search visibility, and you cannot tell them apart by address.

Use IP data for verification and for firewall allow-listing, not as your opt-out. Your robots.txt file is the control Anthropic documents. For anything you truly cannot expose, put it behind a login, since robots.txt is, by the standard’s own wording, not access authorization.

How do you slow ClaudeBot down?

You slow ClaudeBot down with the Crawl-delay directive, which Anthropic supports as a non-standard extension to robots.txt. Its example is:

User-agent: ClaudeBot
Crawl-delay: 1

Two limits apply. Crawl-delay is not part of RFC 9309, so other crawlers may ignore it, and it only throttles; it does not opt you out. If the real problem is cost, because a bot is hammering a heavy dynamic site, this is the lighter fix. Anthropic’s article also says that to report a suspected malfunction you should write to the contact address in the article from an email on the affected domain, since reports from elsewhere are hard to verify.

Getting ClaudeBot access right for AI visibility

The safe setup for most businesses is to decide the three bots separately: opt in or out of training with ClaudeBot, and leave Claude-SearchBot and Claude-User open if you want to be found in Claude’s answers. Then prove it in your logs instead of trusting the file.

A quick checklist to finish with:

• Confirm robots.txt is served at the top level of each domain and subdomain you care about.
• Write one group per bot, so no bot depends on a fallthrough.
• Check your CDN and firewall, because a permissive robots.txt does nothing if the edge returns 403.
• Run the log check monthly, and treat any Claude bot name from an unlisted IP as unverified.
• Revisit the decision when Anthropic updates its article. It offers a form to subscribe to notices of substantial changes.

Getting retrieved is only the first gate. Pages that Claude-User or Claude-SearchBot can reach still have to be quotable, which we cover in retrieval vs citation and in how AI engines choose citations. If you want someone to audit the crawler layer, the CDN settings and the content together, that is part of our GEO service.