GPTBot, OAI-SearchBot and ChatGPT-User explained
By the Why Won't AI team · Published · Last updated
As of November 2026, OpenAI runs three separate crawlers. OAI-SearchBot decides whether your site can appear in ChatGPT search answers. GPTBot collects content that may be used to train OpenAI's models. ChatGPT-User fetches a page when a person in ChatGPT asks for it. You control each one separately in robots.txt.
What each crawler does
| Crawler | Its job | If you block it |
|---|---|---|
| OAI-SearchBot | Finds sites to show in ChatGPT search answers | Your site is not shown in ChatGPT search answers, but can still appear as a navigation link |
| GPTBot | Collects content that may be used to train OpenAI's models | Your content is not used for training. Search is controlled separately. |
| ChatGPT-User | Fetches a page for an action a person starts in ChatGPT | robots.txt rules may not apply, because a person started the request |
The common mistake
Many sites block GPTBot and think that keeps them out of ChatGPT, or block every crawler with one broad rule and drop out of ChatGPT search without knowing it. Each crawler is a separate decision. A rule for all user agents applies to all three unless you give one its own group.
Decide on purpose
| What you want | OAI-SearchBot | GPTBot |
|---|---|---|
| Appear in ChatGPT search, and allow training | Allow | Allow |
| Appear in ChatGPT search, but no training | Allow | Disallow |
| Stay out of both | Disallow | Disallow |
This example keeps a site in ChatGPT search and opts it out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /OpenAI says a robots.txt change can take about 24 hours to apply to search.
robots.txt is not the only gate
Your CDN or host can turn a crawler away before it ever reads robots.txt, and a page built only in the browser can reach a crawler empty. Allowing a crawler does not mean it gets your content. Check what each crawler actually receives (see the Cloudflare and JavaScript guides). You can test your robots.txt rules with our free AI crawler robots.txt checker.
Questions people ask
If I block GPTBot, do I disappear from ChatGPT?
No. ChatGPT search uses OAI-SearchBot. Blocking GPTBot opts you out of training.
Does ChatGPT-User follow robots.txt?
OpenAI says robots.txt rules may not apply, because a person starts the request.
How fast does a robots.txt change apply?
OpenAI says it can take about 24 hours to apply to search.
What this does not tell you
Whether your CDN and host let each of these crawlers through, and what each one actually receives from your pages.
Sources
- OpenAI: Overview of OpenAI crawlers: what OAI-SearchBot, GPTBot and ChatGPT-User each do