OAI-SearchBot: User Agent, robots.txt, and What It Can Actually Read

User-agent: OAI-SearchBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot is the crawler that decides whether ChatGPT can cite you. It is not the training crawler. OpenAI runs those as separate agents with separate tokens, and telling them apart is the single most valuable thing on this page.

Most people who block “the OpenAI bot” mean they do not want their writing used to train a model. Then they write one blanket rule, catch the search crawler in it, and remove themselves from the answer surface they were trying to get into.

What it is for

OpenAI’s bot documentation declares distinct agents with distinct jobs. Two matter most for visibility.

User agent	Job
OAI-SearchBot	Surfaces and links sites in ChatGPT search results
GPTBot	Crawls content to train foundation models

OpenAI documents a third, user-triggered agent on the same page, which fetches a page live because someone in a conversation asked for it. Three agents, three decisions. You are allowed to answer them differently, and you probably should.

If citation traffic matters to you, OAI-SearchBot is the one you want in. See /bots/ for the same breakdown across every other operator.

Can it read JavaScript

OpenAI has never said, and that silence is the honest answer.

I read the current bot documentation in full. It gives exact user agent strings, explains what each crawler does, and tells you how to control them with robots.txt. It does not contain the word JavaScript. Nothing about rendering, headless browsers, or how a page is fetched and parsed. There is no commitment either way.

The only public experiment worth citing is Vercel’s rise of the AI crawler study, published 17 December 2024 and never updated since. It instrumented real traffic and found that OpenAI’s crawler fetched JavaScript files on 11.50% of requests on its nextjs.org sample without executing them. Fetching a script is not running it. The bytes came down the wire and nothing happened with them.

So the position is: not documented, and the one public measurement points at no execution. That measurement is now twenty months old and carries no update note, so treat it as a strong lead rather than a guarantee.

The practical conclusion does not depend on resolving the question. If your headline, your pricing, and your product description only exist after a client-side framework hydrates, you are betting your presence in ChatGPT on a behaviour nobody has committed to. Server-render it and the question stops mattering. The longer version of that argument is in AI crawlers and JavaScript rendering.

How to control it with robots.txt

OpenAI documents robots.txt as the control surface for its crawlers, and names each agent separately so you can be precise.

Block the search crawler entirely, which is rarely what you want:

User-agent: OAI-SearchBot
Disallow: /

The far more common intent is: stay citable, keep training out, hide the private areas.

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

That is two rules doing two different jobs. A single User-agent: * block cannot express it.

What to check after you change it

Directives are requests until your logs confirm them. After a robots.txt change, grep your access logs for the OAI-SearchBot token and confirm the behaviour matches what you asked for.

While you are in there, verify the requests are genuine. The user agent header is free text, and scrapers wear OpenAI’s name because it gets them through soft filters. OpenAI publishes crawler IP ranges. Check the source IP, not the string.

The last check is the one people skip. Fetch your own key pages with JavaScript disabled and read what comes back. If the page is close to empty, no robots.txt rule is going to help you, because the crawler you just invited in has nothing to take away.