Open your server logs and you will find requests from names you do not recognise. GPTBot. ClaudeBot. OAI-SearchBot. Bytespider. meta-externalagent. Most advice about them starts with how to block them, which skips the two questions that actually matter: what is each one for, and is it even really who it says it is.
A user agent is a string the client sends with every HTTP request, saying what it is. That is all it is. It is self-reported, unverified by default, and trivially forged. Treating it as identity is the root of most bad crawler policy.
The three jobs, which matter more than the brand
Group crawlers by vendor and you get a confusing list. Group them by job and it collapses into three, and the right policy falls out immediately.
| Job | What it does | What blocking it costs you |
|---|---|---|
| Model training | Collects text to train future models | Affects models that do not exist yet. No effect on today’s answers |
| Search index | Builds the index that answers get drawn from | You become ineligible to be cited. Immediate and permanent while blocked |
| User-triggered fetch | Retrieves one page because a person asked something | The assistant cannot read your page for a user who explicitly asked about you |
OpenAI runs all three. Its documentation lists GPTBot for training foundation models, OAI-SearchBot for surfacing sites in ChatGPT’s search features, and ChatGPT-User for user-initiated actions. Anthropic mirrors the split with ClaudeBot, Claude-SearchBot and Claude-User. Perplexity runs PerplexityBot for indexing and Perplexity-User for live fetches.
The expensive mistake is writing one rule for a vendor. Blocking everything OpenAI operates because you object to training also removes you from ChatGPT’s search index, which is the surface that sends traffic and cites you by name. Those are opposite decisions wearing the same label.
How they differ from a browser’s user agent
Read a declared AI agent next to a browser and the difference is a convention, not a format.
A real Chrome string looks like Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36. It describes an operating system, an engine and a version, and nothing in it identifies a purpose or an owner.
A declared crawler adds two things inside the compatible; section: a named token with a version, and a URL pointing at documentation. OpenAI’s is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. Perplexity’s indexer carries PerplexityBot/1.0; +https://perplexity.ai/perplexitybot.
That + URL is the whole convention. It is a bot voluntarily telling you where to read about it and how to opt out. Three consequences follow.
First, the Mozilla/5.0 prefix is meaningless. Almost every agent carries it for historical compatibility reasons, so its presence tells you nothing about whether something is a browser.
Second, the useful part is the token, not the whole string. Match on GPTBot, not on the full string, because vendors bump versions. GPTBot is on 1.4 at time of writing.
Third, and this is the point: everything distinguishing a crawler from a browser is a courtesy. There is no field a browser has that a bot cannot copy.
The string proves nothing
Any client can send any user agent string. curl -A "GPTBot" https://yoursite.com costs nothing and produces a log line indistinguishable from the real thing.
This matters in both directions. Traffic claiming to be an AI crawler may be a scraper wearing the name. Traffic that is not declaring itself may still be an AI system reading your page.
Verify by network identity instead:
- Check published IP ranges. Several operators publish theirs as JSON specifically so you can verify. Anthropic points to a bots list at
claude.com/crawling/bots.json. Check each vendor’s own documentation rather than a third-party directory. - Do a forward-confirmed reverse DNS lookup. Take the requesting IP, resolve it to a hostname, then resolve that hostname back to an IP. If it does not match the original, the request is not what it claims. This is the same technique Google has long recommended for verifying Googlebot.
Anything failing both checks is an unverified client that chose to type a bot’s name into a header field.
Undeclared traffic is the gap
Robots.txt rules and user agent blocks only reach clients that declare themselves and choose to comply. That set is not the whole of AI traffic, and there is now a measured example of the gap.
On 4 August 2025 Cloudflare published an account of Perplexity using undeclared crawlers. Its claim is that Perplexity was “repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity”, and thereby appeared “to obscure their crawling identity in an attempt to circumvent the website’s preferences”. The numbers Cloudflare gives:
- Declared crawler traffic: 20 to 25 million daily requests
- Undeclared traffic attributed to the same operator: 3 to 6 million daily requests
- Observed “across tens of thousands of domains and millions of requests per day”
The stealth string Cloudflare reports is an ordinary macOS Chrome user agent, indistinguishable from a person’s browser by inspection alone. Cloudflare says it de-listed Perplexity as a verified bot as a result.
Two caveats belong with that. It is one infrastructure vendor’s account of one operator, published by a company that sells crawler control, and it is a dispute rather than a settled finding. Take the structural lesson rather than the ratio: roughly a fifth of that operator’s measured volume was invisible to any policy written against a token.
What that example cannot tell you is the size of the gap in aggregate. Nobody publishes a measurement of undeclared AI traffic across the web, so treat a stated trend with suspicion unless whoever states it shows how they counted.
There is a second, entirely legitimate category that no block catches either. Vendors exempt their own user-triggered fetchers. OpenAI’s documentation says of ChatGPT-User that “Because these actions are initiated by a user, robots.txt rules may not apply.” That is not evasion. It is a stated position that a fetch on a person’s behalf is not crawling, and it means your robots.txt is not a complete description of who reads your pages.
The practical consequence: your crawler policy governs the declared, compliant subset. Treat it as such, and never conclude from clean logs that no AI system is reading you.
What to actually do
Decide per job, not per vendor. Write separate rules for training, search and user-triggered fetching. Most sites should allow search crawlers unconditionally, because the alternative is opting out of citation entirely.
Verify before you act on a log line. Before you conclude a bot is misbehaving, confirm it is that bot. A large share of “GPTBot is hammering my server” turns out to be a scraper using the name.
Check what the crawlers can actually read. Allowing a crawler in achieves nothing if your content only appears after JavaScript runs, and most AI crawlers’ rendering behaviour is undocumented. See which AI crawlers render JavaScript for what each vendor has and has not committed to.
Look up the specific agent. We maintain a directory of AI crawlers with each one’s exact user agent string, what it is for, whether the operator documents JavaScript execution, and copy-paste robots.txt for allowing or blocking it.
For the wider decision about which crawlers to permit in the first place, and how those choices interact with Google rankings, see AI crawler access.
The short version
A user agent is a claim, not an identity. Group crawlers by what they do rather than who runs them, verify the ones you care about by IP rather than by string, and remember that your rules only bind the clients that declare themselves and choose to listen.