GPTBot vs OAI-SearchBot vs ChatGPT-User

GPTBot, OAI-SearchBot and ChatGPT-User are three separate OpenAI agents with three separate jobs: GPTBot collects content that may train models, OAI-SearchBot decides whether you appear in ChatGPT search, and ChatGPT-User fetches a page when a person asks for it. Only one of them controls your search visibility, and it is not the one with the famous name.

This page is for site owners and marketers who have seen “GPTBot” in a log file or a guide and need to know what to put in robots.txt. It covers what each agent does, the rules that follow from each goal, the syntax trap that blocks all three by accident, and how to confirm the traffic is real.

On this page

• What do GPTBot, OAI-SearchBot and ChatGPT-User each do?
• Which one affects ChatGPT search visibility?
• What does blocking GPTBot actually change?
• Why ChatGPT-User is different
• The robots.txt rules for each goal
• The wildcard trap that blocks all three
• How to confirm a request is really from OpenAI
• Where OpenAI crawler settings go next

What do GPTBot, OAI-SearchBot and ChatGPT-User each do?

Each agent has one stated purpose, and OpenAI documents them separately on its overview of OpenAI crawlers. The diagram below puts the three side by side.

Scroll the diagram sideways to see all of it.

The same facts in table form, with the identifiers you will see in logs:

Version numbers appear inside the full user-agent strings (GPTBot/1.4 and OAI-SearchBot/1.4 at the time of writing, ChatGPT-User/1.0), and OpenAI notes the version may change. Match on the token, not the version.

Which one affects ChatGPT search visibility?

OAI-SearchBot is the one that matters for search. OpenAI recommends allowing it if you want to be found, and says that sites which opt out will not appear in ChatGPT search answers, though they can still show up as navigational links.

That makes the common advice “block GPTBot to stay out of ChatGPT” wrong. GPTBot is not the gatekeeper for search. If your goal is to be recommended in answers, the robots.txt line that matters is the one for OAI-SearchBot. Our guide to how to rank in ChatGPT search covers what happens after the crawler lets you in: content structure, entities and citation patterns. This page stops at the door.

Two practical points follow:

• Allowing is not a ranking factor. Letting OAI-SearchBot in makes you eligible to be surfaced. It does not make you the answer. Plenty of allowed sites are never cited.
• Changes are not instant. OpenAI says that for search results it can take about 24 hours from a robots.txt update for its systems to adjust. Do not test a change at noon and judge it at 12:05.

What does blocking GPTBot actually change?

Blocking GPTBot changes whether your content may be used for model training, and nothing about whether ChatGPT search can show you. OpenAI’s own wording is that disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models, and that this is independent of OAI-SearchBot settings.

Three limits are worth stating plainly:

• It is a signal, not a lock. The robots.txt standard, RFC 9309, says outright that these rules are not a form of access authorization. Well-behaved crawlers follow them. Nothing technical stops a page that is publicly reachable.
• It is forward-looking. A Disallow tells OpenAI not to use content for training as documented. OpenAI’s page does not describe what happens to anything collected earlier, so do not assume a block erases it.
• It is a business decision. If you sell the content itself, blocking has an obvious logic. If you sell services and your content is how buyers find you, the trade is less clear. We lay out both sides in should you block AI crawlers, so we will not repeat the argument here.

If your only goal is “keep my content out of training but stay visible in search”, you disallow GPTBot and allow OAI-SearchBot. That is the single most useful combination to know, and it is the first row of the goal diagram in the rules section below.

Why ChatGPT-User is different

ChatGPT-User is the odd one out because a person, not a crawl schedule, triggers it. OpenAI says it visits pages when a user asks ChatGPT or a Custom GPT a question, supports GPT Actions, is not used for automatic crawling, and does not determine whether content appears in search.

The consequence is one sentence from OpenAI’s documentation: because these actions are initiated by a user, robots.txt rules may not apply. In practice:

• A Disallow for ChatGPT-User is a polite request. It may be honored or not, and you should not build a privacy plan on it.
• It does not affect search visibility, so blocking it will not help or hurt your standing in ChatGPT search answers.
• If a page truly must not be fetched, put it behind a login or block it at the server. Do not rely on robots.txt for that. Google’s robots.txt introduction makes the same point for its own search: robots.txt is not a mechanism for keeping a page out of Google, and password protection is the fix.

This matches how other assistants treat live user requests. Our robots.txt for AI crawlers guide covers the equivalent agents from Perplexity, Google and others in one table, and the ClaudeBot guide does the same for Anthropic.

The robots.txt rules for each goal

Start from the goal, then pick the rule. The diagram below maps four goals to a rule per agent.

Scroll the diagram sideways to see all of it.

For the highlighted goal, the file looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

To restrict only part of a site, scope the rule by path instead of the whole domain:

User-agent: GPTBot
Disallow: /clients/
Disallow: /research/

Anything not matched by a Disallow stays open. Give each agent its own group rather than stacking several User-agent lines over one rule block, so a later edit to one token cannot silently change another.

The wildcard trap that blocks all three

A site-wide User-agent: * block with no named groups blocks all three OpenAI agents at once, including the one you wanted. It often comes from a staging file that shipped to production.

How Google’s robots.txt documentation handles groups explains why. Per Google’s robots.txt specification, only one group is valid for a particular crawler, and the crawler uses the group with the most specific matching user agent. User-agent specific groups and the global * group are not combined. RFC 9309 adds that if more than one group matches a product token, the rules must be combined into one, and that product-token matching is case-insensitive.

That gives you two safe habits:

1. Name the agents you care about. A named OAI-SearchBot group is not overridden by a * group elsewhere in the file, so you can keep a restrictive default and still allow the search agent.
2. Check the whole file, not the group you edited. A blanket Disallow: / under * is easy to miss when your edit is forty lines above it.

Other file-level limits from the same Google document are worth a quick check:

These are Google’s rules. OpenAI’s page does not publish equivalents, so treat them as the sensible baseline, not a guarantee for every crawler. Our robots.txt checklist walks through the file-level checks in order.

How to confirm a request is really from OpenAI

Check the user-agent token, then check the IP. A user-agent string is just text, and anyone can send GPTBot in a header. OpenAI publishes a JSON list of IP ranges for each agent, which is the check that counts.

The flow is short, and the diagram below shows it.

Scroll the diagram sideways to see all of it.

Use the list that matches the token: openai.com/gptbot.json, openai.com/searchbot.json or openai.com/chatgpt-user.json. Then look at what your server answered. A steady run of 200s for OAI-SearchBot on your key pages is the result you want. A run of 403s or 429s with a permissive robots.txt means a firewall, CDN or rate limiter is overruling your file. That is a separate layer, and our robots.txt for AI crawlers guide explains it.

One more reason to check is rendering. A crawler that gets a 200 but an empty JavaScript shell has still not seen your content, which we cover in JavaScript rendering and AI crawlers.

Where OpenAI crawler settings go next

Decide the three tokens once, write them down, and recheck after any CDN, plugin or migration change. The settings are short, but they break quietly: a staging file, a security plugin or a bot-management toggle can change them without anyone editing robots.txt.

A sensible routine:

1. Set the three rules from the goal table and note why you chose them.
2. Wait about a day, then confirm OAI-SearchBot returns 200s from verified IPs.
3. Recheck after site launches, host moves and CDN changes. Our technical SEO checklist for AI search lists the other crawl checks, and the wider checklists library has the rest.
4. Keep content decisions separate. Whether to allow training is strategy, and getting cited is a content and entity problem that LLM SEO covers.

If you want someone to check your file, your CDN and your logs against what OpenAI actually fetches, that is part of our GEO service.