ia_archiver: The Archival Bot That Is Not the Internet Archive

User-agent: ia_archiver
Disallow: /

User-agent: ia_archiver
Allow: /

ia_archiver is the most misattributed user agent in the whole crawler landscape, and the correction comes from the best possible source.

In April 2017 the Internet Archive published a post explaining why robots.txt, a file designed for search engines, is a poor fit for web archiving. In the comments underneath, Mark Graham, Director of the Wayback Machine, answered a question about this exact token. His words: the ia_archiver User Agent is used by Alexa Internet, not the Internet Archive.

That one sentence undoes most of what is written about this bot. Nearly every third-party bot directory files ia_archiver under the Internet Archive, because the ia prefix looks like an abbreviation and the association is decades old. The organisation the token is credited to has said in public that it is not theirs.

Where the confusion comes from

The link was real, historically. Alexa Internet ran a large web crawl and its data fed into what became the Wayback Machine. Both organisations were founded by Brewster Kahle, and for years the two operations were tangled together in a way that made “Alexa’s crawler” and “the archive’s crawler” feel like the same thing to a webmaster reading a log file.

They were not the same thing, and Alexa Internet is no longer a going concern as a crawling service. I could not re-verify the exact wind-down date from a primary source in this pass, so treat any date you see attached to it as unconfirmed until you check it yourself.

What that leaves is a legacy token with no living operator and no documentation. Which brings us to the honest part.

There is no first-party documentation

Alexa Internet published no crawler documentation that still stands. The Internet Archive has disclaimed the token. So there is nobody to have written down a rendering behaviour, a robots.txt policy, a crawl rate, or an IP range.

Every confident table you will find that lists JavaScript and robots.txt behaviour for ia_archiver is copying another table. We are not going to add a sixth generation of that. The fields on this page are undocumented because the documentation does not exist, and we would rather say so than cite a bot directory as if it were the operator.

If you are seeing this token in current logs at volume, the more likely explanation is something reusing an old, harmless-looking string. Treat it the way you would treat any unverifiable agent: judge it by request behaviour, not by what it calls itself.

Can it read JavaScript

Unknown, and unknowable from any authoritative source.

There is no operator statement to read. There is no published beacon test that isolated this token. Archival crawlers as a class have historically worked by fetching HTML and extracting links from the markup rather than driving a browser, but “as a class” is not evidence about a specific agent, and we are not going to dress up a category generalisation as a fact about this one.

Undocumented is the accurate answer. If you want a rendering answer that actually changes what you build, the useful reading is AI crawler access, where the rendering question has real consequences for whether you get cited.

How to control it with robots.txt

If you want to write the rule anyway, this is the form:

User-agent: ia_archiver
Disallow: /

Historically some site owners paired it with an explicit allow while excluding private areas:

User-agent: ia_archiver
Allow: /
Disallow: /account/
Disallow: /internal/

Understand what these rules can and cannot do. They address an operator that has been disclaimed by the organisation people think they are writing to. If your actual goal is to keep a site out of the Wayback Machine, the Internet Archive’s own documented route is to write to info@archive.org and request that content be removed from the Wayback Machine and from future crawling.

It is also worth knowing the Archive’s stated position before you decide. In the same 2017 post, Mark Graham wrote that they see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective. That is a deliberate move away from treating robots.txt as binding for archival purposes, and it is public.

What this means for AI visibility

Nothing. This is an archive token, or the ghost of one. It sits in a different category from the model training crawlers and the AI answer crawlers that people are usually auditing for when they end up reading pages like this.

If you arrived here from a log audit, the rest of the bot directory will tell you which of the other unfamiliar tokens on your list are worth a decision. Most of them are not. The ones that are, are the ones that determine whether an AI answer can name you.

For the crawler that genuinely does the Internet Archive’s work, see our page on Heritrix. That is the software behind the Wayback Machine, and it is a far more interesting story than this one.