ClaudeBot: User Agent, robots.txt, and What Anthropic Actually Documents

User-agent: ClaudeBot
Disallow: /

User-agent: ClaudeBot
Allow: /

ClaudeBot is Anthropic’s training crawler. Its stated job is collecting web content that could potentially contribute to training generative AI models. It is one of three agents Anthropic declares, and the other two are the ones that carry your visibility, so the split is worth getting right before you write a rule.

Three agents, three decisions

Anthropic’s crawler article names each bot and what it does.

User agent	Job, in Anthropic’s words
ClaudeBot	Collects web content that could potentially contribute to model training
Claude-User	Accesses websites when individuals ask questions to Claude
Claude-SearchBot	Analyses content to improve the relevance and accuracy of search responses

Block all three and you are invisible to Claude in every mode: not in the training data, not fetchable when a user asks about you, not eligible as a search source. Block only ClaudeBot and you have opted out of training while staying reachable and citable. Those are very different outcomes from rules that look nearly identical in a text file.

The same three-way split shows up across most operators. /bots/ has the comparison.

Can it read JavaScript

Anthropic does not address it, and the empirical picture is more interesting here than for most crawlers.

I read the crawler article in full. It explains what each bot is for, states the robots.txt position clearly, and points to a published IP list. It contains nothing about JavaScript, rendering, headless browsing, or how page content is parsed after it is fetched.

The measurement comes from Vercel’s rise of the AI crawler study, published 17 December 2024. It found Claude’s crawler fetched JavaScript files on 23.84% of requests on its nextjs.org sample, roughly double OpenAI’s 11.50%, and did not execute any of them. A crawler pulling scripts down without running them is doing something, but it is not rendering your page.

So: undocumented by the vendor, and the one public experiment says the bytes arrive and nothing runs. That study has never been updated. Twenty months on it is the strongest evidence available and it is still one snapshot from one vantage point.

The design conclusion is the same one every time. If a page’s substance appears only after client-side hydration, you are relying on an undocumented behaviour to be visible. Put the content in the initial HTML response and the question is moot. AI crawlers and JavaScript rendering works through what that means for a modern framework build.

How to control it with robots.txt

Anthropic’s position is unusually explicit: its bots respect do-not-crawl signals by honoring industry standard directives in robots.txt, and it supports the non-standard Crawl-delay extension.

Opt out of training only, keeping the surfaces that can send you readers:

User-agent: ClaudeBot
Disallow: /

Opt out of everything Anthropic runs:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

Slow it down rather than shutting it out, if the concern is server load rather than principle:

User-agent: ClaudeBot
Crawl-delay: 10
Disallow: /admin/

Crawl-delay is not part of the original robots specification and most crawlers ignore it. Anthropic is one of the few that says it honours it, which makes this a real option rather than a decorative one.

Verifying it is actually ClaudeBot

Anthropic publishes source IP addresses at claude.com/crawling/bots.json. If a request claiming to be ClaudeBot comes from an IP on that list, it is genuinely Anthropic. If it does not, you are looking at something wearing the name.

That matters more than it sounds. The user agent header is free text. Impersonating a well-known crawler is the cheapest way for a scraper to get past a filter that only reads strings, and it means the “AI crawler traffic” in your analytics is not necessarily what it claims to be. Check the IP before you draw conclusions from the volume.