Reference
AI crawler directory
Every declared AI crawler we have documentation for, with its user agent, what it is for, and the two questions that actually matter: can it execute JavaScript, and does it obey robots.txt. Where a vendor has not answered, this directory says so rather than guessing.
| Crawler | Operator | Purpose | Executes JS | robots.txt |
|---|---|---|---|---|
ia_archiver ia_archiver | Alexa Internet (not the Internet Archive) | archive | Not documented | Not documented |
ClaudeBot ClaudeBot | Anthropic | model training | Not documented | Yes |
Applebot Applebot | Apple | search index | Yes | Yes |
Bytespider Bytespider | ByteDance | other | Not documented | Not documented |
CCBot CCBot | Common Crawl | model training | No | Yes |
Googlebot Googlebot | search index | Yes | Yes | |
Heritrix Heritrix | Internet Archive (software); run by many independent archives, libraries and universities | archive | Not documented | Yes |
BingPreview BingPreview | Microsoft | other | Not documented | Not documented |
GPTBot GPTBot | OpenAI | model training | Not documented | Yes |
OAI-SearchBot OAI-SearchBot | OpenAI | search index | Not documented | Yes |
PerplexityBot PerplexityBot | Perplexity | search index | Not documented | Not documented |
Twitterbot Twitterbot | X (formerly Twitter) | user-triggered fetch | Not documented | Not documented |
YandexAdditional YandexAdditional | Yandex | other | Not documented | Yes |
YandexBot YandexBot | Yandex | search index | Not documented | Yes |
Why three states and not two
Most published crawler tables give you yes or no. That is misleading, because for most of these bots the vendor has never addressed rendering at all. A "no" you inferred from someone else's experiment is a different claim from a "no" the vendor wrote down, and only one of them is safe to build on. This directory keeps them apart. The reasoning is set out in which AI crawlers render JavaScript.
For the separate question of which crawlers you should allow in the first place, see AI crawler access. For how to read these user agents in your own server logs, see AI user agents.