CCBot and Common Crawl: Allow or Block?

CCBot is the web crawler of Common Crawl, a nonprofit whose open archive of web pages is one of the raw sources AI builders use to assemble training data, so the allow-or-block decision is about future models, not about today’s AI search answers. Allowing it will not get you cited tomorrow. Blocking it will not remove you from ChatGPT today.

This page is for site owners and marketers who spotted CCBot in a log, a firewall alert or a “block these AI bots” list and want a straight answer. It covers how Common Crawl data reaches models, why that is a different path from live retrieval, who should block, and how to check your logs.

On this page

• What is CCBot and what is Common Crawl?
• How does Common Crawl data reach AI models?
• How is this different from live AI search?
• Should you allow or block CCBot?
• What does blocking CCBot actually change?
• How do you check CCBot in your server logs?
• Setting your CCBot policy in robots.txt

What is CCBot and what is Common Crawl?

CCBot is the crawler that collects pages for Common Crawl, which describes itself as “a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet” for researchers, companies and individuals at no cost. The Common Crawl FAQ is the primary source for most of what follows.

A few facts worth knowing before you decide anything:

• It is a sample, not a full copy. Common Crawl says it does not generally archive any entire website, only a randomly selected subset of each one.
• It has been running a long time. The Common Crawl overview says the corpus has been collected since 2008 and holds petabytes of data, stored on Amazon Web Services.
• It identifies itself clearly. The CCBot page gives the user-agent as CCBot/2.0 (https://commoncrawl.org/faq/).
• It reads robots.txt first. The FAQ says CCBot checks your robots.txt before fetching pages and rechecks it periodically.

Do not confuse it with other things called ccbot. When we ran the live Google check for this keyword in the United States, the results were mixed: a Google AI Overview describing the crawler, plus unrelated pages such as a robotics company and a developer tool. The crawler you see in server logs is Common Crawl’s, and only that one is covered here.

How does Common Crawl data reach AI models?

Common Crawl data reaches models indirectly: CCBot collects pages, Common Crawl publishes them, and AI builders download the archive, filter it and train on the result. Common Crawl itself does not train anything on your content.

The most detailed independent study is the Mozilla Foundation’s report on Common Crawl, written by Stefan Baack. Its findings, as of the report’s October 2023 cutoff:

• Wide use. Of 47 text-generation models published between 2019 and October 2023, at least 64 percent (30) used at least one filtered version of Common Crawl for pre-training.
• Builders filter first. The report says Common Crawl is not used directly because it contains content undesirable for training. Builders use filtered versions such as Google’s C4 and EleutherAI’s Pile-CC, built with language filtering, keyword rules and deduplication.
• Crawls are large and regular. Since 2017, crawls have been published monthly, each capturing between three and five billion URLs, and roughly half of each crawl is made up of URLs fetched before.
• Well-linked sites repeat. Domains with very high link-based scores, such as Wikipedia, are always included, so heavily linked sites are more likely to appear.

The diagram below shows the two routes by which your page can reach an AI answer. Only the first runs through CCBot.

Scroll the diagram sideways to see all of it.

If you want the wider picture of how pretraining differs from retrieval, LLM SEO covers both levers, and the AI search glossary defines the terms used here.

How is this different from live AI search?

Live AI search fetches pages at the moment someone asks a question, using bots built for that job, while Common Crawl data is collected in advance and baked into model weights during training. Whatever a model learned from Common Crawl comes with no link back to you. A search bot fetches the live page, so it can.

OpenAI’s bots documentation shows the split clearly. OAI-SearchBot surfaces sites in ChatGPT’s search features. GPTBot crawls content that may be used to train foundation models. ChatGPT-User visits pages when a person asks. The page says “Each setting is independent of the others,” and that a robots.txt change can take about 24 hours to take effect for search.

Two earlier guides go deeper on the neighbors. GPTBot vs OAI-SearchBot vs ChatGPT-User covers OpenAI’s three agents, and ClaudeBot and Anthropic’s other bots covers Anthropic’s. For how retrieved pages become citations, see retrieval vs citation.

Should you allow or block CCBot?

Allow CCBot if you sell services or products and want to be discoverable and known, and block it if your content itself is what you sell or if you hold something you could license. That is our judgment, not a rule from Common Crawl, and it rests on one honest gap: no source we found measures whether being in Common Crawl changes how often models mention a brand.

The reasoning is a trade between two small, uncertain effects:

• The upside of allowing is indirect and unproven. Your pages become eligible for future training sets. Nothing guarantees a given builder includes them, filters keep them, or a model remembers them.
• The upside of blocking is also limited. It stops future collection, not past collection, and it does not touch live search, which is where visible traffic comes from.

So the decision mostly turns on your business model, the same lens used in should you block AI crawlers.

Scroll the diagram sideways to see all of it.

A note on timing. CCBot rechecks robots.txt periodically, and crawls are published monthly, so allow time before a change shows up in your logs or in new crawls. The same applies in reverse if you unblock.

What does blocking CCBot actually change?

Blocking CCBot changes one thing: whether Common Crawl collects your pages in future crawls. It does not affect your Google rankings, your ChatGPT search visibility or your Perplexity citations, and it does not erase what was already collected.

Here is what each side of that sentence rests on.

What it stops. Common Crawl’s blog on opt-out protocols says CCBot observes the Robots Exclusion Protocol and that a Disallow: / rule for CCBot blocks use for both machine learning training and search indexing.

What it does not stop. The same post says Common Crawl has no technical implementation yet for detecting several newer opt-out signals, including ai.txt and tdmrep.json, and calls building that a top priority. Because its archive files also capture HTTP headers and HTML metadata, downstream users can choose to honor protocols based on them. Whether they do is up to them.

What happens to old data. Common Crawl’s Opt-Out Ledger announcement says it published, on September 17, 2025, the full list of legal opt-out requests it had received, so downstream users can see what owners asked to exclude. It does not say whether previously collected data is removed. Mozilla’s report mentions that The New York Times pushed Common Crawl to remove its content from the archive, which shows removal requests exist, but treat removal as something to negotiate, not a setting.

What robots.txt is. RFC 9309 states that these rules “are not a form of access authorization.” CCBot follows them. A crawler pretending to be CCBot, or a downstream dataset copied elsewhere, may not. For anything private, use authentication.

The practical consequence: blocking CCBot is a statement about the future, made now. If you published a lot before blocking, assume older crawls may still contain it.

How do you check CCBot in your server logs?

Check by finding CCBot requests in your access log, then verifying each IP with the reverse-then-forward DNS method Common Crawl documents. Do this before you block, so you know whether the traffic is real, and after, so you know the block worked.

Step 1: find the requests. On Apache or Nginx, search for the user-agent token and count requests per IP:

grep -i "CCBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Step 2: verify the top IPs. Common Crawl warns that other crawlers falsely identify themselves as CCBot and recommends verifying user-agent strings. Its method is to run a reverse lookup on the IP, which should return a hostname under crawl.commoncrawl.org, then a forward lookup on that hostname, which should return the original IP. Its worked example is 18.97.14.84, which resolves to 18-97-14-84.crawl.commoncrawl.org.

Scroll the diagram sideways to see all of it.

Step 3: handle IPv6. Common Crawl says reverse DNS is not yet supported over IPv6. It publishes its IPv4 and IPv6 ranges as JSON at index.commoncrawl.org/ccbot.json. When we fetched that file for this post, it was dated August 11, 2026 and listed five prefixes (four IPv4, one IPv6), so check the live file rather than copying numbers into a firewall rule.

Step 4: read the status codes. Look at what your server returned to the verified requests. A pile of 429 or 5xx responses means the crawl is straining your host, and the fix there is a Crawl-delay, not necessarily a block. If you blocked CCBot, verified requests should stop fetching real pages after Common Crawl rechecks robots.txt, and any that continue deserve a closer look at your CDN or firewall rules. Our robots.txt for AI crawlers guide covers why a CDN can disagree with your robots file.

Setting your CCBot policy in robots.txt

Common Crawl’s own guidance for blocking is two lines, and its FAQ says Crawl-delay can slow the crawler instead. Pick the version that matches the row you chose above.

Block everything:

User-agent: CCBot
Disallow: /

Block only private areas, and leave the rest open:

User-agent: CCBot
Disallow: /clients/
Disallow: /members/

Slow it down without blocking (the 2 seconds is an example, tune it to your server):

User-agent: CCBot
Crawl-delay: 2

Two details trip people up. First, a specific User-agent: CCBot group does not inherit from your * group, so repeat any rules you still want. Second, the file only governs what is fetched next. Expect the change to show up over the following crawls, and confirm it in your logs using the steps above.

Where your CCBot decision goes next

Treat CCBot as a long-term, low-visibility decision and keep your effort on the bots that decide whether you are cited this week. For most service businesses that means allowing the search and retrieval crawlers first, then deciding on training-side bots such as CCBot on purpose, with the reason written down. The technical SEO checklist for AI search puts all of these decisions in one place, and AI search engines compared shows which engines draw on which sources.

If you would like someone to audit which AI crawlers can reach your site, what your CDN is quietly blocking, and what your policy costs you in citations, that is part of our GEO service.