GPTBot: User Agent, robots.txt, and Whether Blocking It Costs You Anything

User-agent: GPTBot
Disallow: /

User-agent: GPTBot
Allow: /

GPTBot is OpenAI’s training crawler. Its declared job is collecting web content that may go into training foundation models, and it is the agent most people actually mean when they say they want to block AI.

Blocking it is a legitimate choice. What matters is blocking it precisely, because the crawler sitting next to it in OpenAI’s documentation is the one that decides whether ChatGPT can cite you at all.

What it is for, and what it is not

OpenAI’s bot documentation separates the agents by purpose.

User agent	Job
GPTBot	Crawls content for training foundation models
OAI-SearchBot	Surfaces and links sites in ChatGPT search results

These are two different decisions and they deserve two different answers. Blocking GPTBot costs you nothing in ChatGPT search, because GPTBot is not what builds that index. Blocking both costs you the citation surface, and that is what a lazy User-agent: * rule does.

The rest of the crawler landscape splits the same way. /bots/ has the per-operator breakdown.

Can it read JavaScript

Not documented, and the one public measurement suggests no.

I read OpenAI’s bot page in full. It publishes exact user agent strings and explains robots.txt control. It says nothing whatsoever about JavaScript, rendering, or headless browsing. No commitment, no hint, no last-updated note on the behaviour.

The empirical side comes from Vercel’s rise of the AI crawler study of 17 December 2024. Measuring real traffic, it found OpenAI’s crawler fetched JavaScript files on 11.50% of requests on its nextjs.org sample and did not execute them. Downloading a bundle is not running it.

That study has never been updated. It is the best public evidence available and it is twenty months old, which is a long time in this space. So the honest label is undocumented, not “no”. Collapsing an undocumented behaviour into a confident no is how the bad comparison tables on this topic got built.

For a training crawler the stakes of rendering are lower than for a search crawler, since being missed by GPTBot is not a visibility loss. The same architecture question applies to every other bot hitting your site though, and the argument is laid out in AI crawlers and JavaScript rendering.

How to control it with robots.txt

OpenAI documents robots.txt as the control surface and names GPTBot separately, so you can be exact.

Opt out of training and stay in search, which is what most site owners actually want:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Allow training on your public writing while keeping commercial and private areas out:

User-agent: GPTBot
Allow: /blog/
Disallow: /

Note the ordering habit worth keeping: be explicit per agent rather than relying on a wildcard, because a wildcard makes a decision about crawlers you have not thought about yet.

Three things to check afterwards

Confirm it took effect. Search your access logs for the GPTBot token after the change and see whether the pattern matches your rule. Code you have not observed in production is a hypothesis.

Confirm the requests are real. The user agent string is a header anyone can send, and scrapers borrow OpenAI’s name to get past naive filters. OpenAI publishes crawler IP ranges. Verify against the source IP.

Confirm you understand the scope. A robots.txt rule is forward-looking and domain-scoped. It does not retrieve data already collected, and it does not touch copies of your content living on sites you do not control. If being in a training set is genuinely unacceptable to you, the robots.txt line is the start of that work, not the end of it.