CCBot: The Crawler Behind Most LLM Training Data
CCBot builds the Common Crawl archive that trains most large language models. It is also the only crawler that documents, in writing, that it does not execute JavaScript.
At a glance
- User-agent token
CCBot- Full user-agent
CCBot/2.0 (https://commoncrawl.org/faq/)- Operated by
- Common Crawl
- What it does
- Builds the free, public Common Crawl web archive used as training data by most large language models
- Executes JavaScript
- No Common Crawl documents the negative directly in its FAQ: 'Currently, JavaScript is not executed and Cookies are not used.' It is the only major crawler operator to state this plainly.
- Obeys robots.txt
- Yes
- Documentation
- commoncrawl.org
robots.txt
Block CCBot
User-agent: CCBot
Disallow: /Allow CCBot
User-agent: CCBot
Allow: /CCBot is the crawler behind Common Crawl, a free and public archive of the web that has been running since 2007. If you are worried about your content training language models, this is the crawler with the widest reach, because Common Crawl is a shared dataset rather than one company’s private collection.
It is also, unexpectedly, the most honest crawler on the internet about what it can do.
Can it read JavaScript
No, and this is the only major crawler where the operator has said so plainly.
The Common Crawl FAQ states: “Currently, JavaScript is not executed and Cookies are not used.”
Compare that with the rest of the field. Google and Apple document that they render. Common Crawl documents that it does not. Every other major operator, including OpenAI, Anthropic and Perplexity, has published nothing at all on the subject. That is set out in full in which AI crawlers render JavaScript, and you can see the whole comparison in our crawler directory.
Note the hedge in Common Crawl’s wording. “Currently” is not a permanent commitment, and a statement written at one time can be overtaken without an announcement. It is still a far better position than inferring behaviour from someone else’s experiment.
The practical upshot is direct: if your content requires JavaScript to appear, it is not in Common Crawl. It is therefore not in the training data of every model built on Common Crawl, which is a large share of them.
Why this crawler is different from the rest
Blocking one company’s crawler affects one company. Blocking CCBot affects everyone downstream of a public dataset.
That cuts both ways. If your goal is to keep your content out of model training broadly, CCBot is the single highest-leverage block available. If your goal is more targeted, blocking CCBot is a blunt instrument that also removes you from academic research, independent projects, and small developers who cannot afford to crawl the web themselves.
There is also a timing point people get wrong. Robots.txt is forward-looking. Blocking CCBot today does nothing about archives already collected or models already trained. Common Crawl has been publishing since 2007, so for most established sites the horse left some years ago.
How to control it
Block it entirely:
User-agent: CCBot
Disallow: /Allow it but keep sections private:
User-agent: CCBot
Allow: /
Disallow: /members/Common Crawl documents that it obeys robots.txt, and it is a non-commercial organisation with a long track record of doing so.
The other reason to know about Common Crawl
Common Crawl is not only a crawler. It publishes the archive, which means anyone can query what the web looked like at a point in time, including which sites linked to which.
That makes it genuinely useful rather than only something to defend against. It is the free alternative to commercial backlink tools for certain questions, and it is how you can still study a website after it goes offline.
If you want your content in the public record of the web, and legible to the widest set of future systems, the requirement is the same as everywhere else: put it in the initial HTML response.
Common questions
Does CCBot execute JavaScript?
No, and Common Crawl says so directly in its FAQ: 'Currently, JavaScript is not executed and Cookies are not used.' It is the only major crawler operator to document the negative rather than staying silent. Note the word currently, which is a hedge rather than a permanent commitment.
Why does blocking CCBot matter more than blocking one AI company?
Common Crawl is a shared public dataset. Many model developers train on it rather than crawling the web themselves. Blocking CCBot therefore affects a wide set of downstream users at once, including researchers and small developers who have no crawler of their own.
Does blocking CCBot remove my content from models already trained?
No. Robots.txt is forward-looking. Archives already collected still contain your pages, and models already trained on them are unaffected. Blocking CCBot changes what future crawls collect.