Heritrix: The Crawler That Built the Wayback Machine
Heritrix is the open-source archival crawler behind the Internet Archive. What it is, why the user agent tells you almost nothing on its own, and how its robots.txt policy actually works.
At a glance
- User-agent token
Heritrix- Full user-agent
Operator-configured. Heritrix ships a template each operator fills in with their own contact URL, so there is no single fixed string. Read the contact URL in the header, not the token.- Operated by
- Internet Archive (software); run by many independent archives, libraries and universities
- What it does
- Open-source archival web crawler. It is the software the Internet Archive built and uses to capture the web for the Wayback Machine, and that national libraries and universities run for their own collections.
- Executes JavaScript
- Not documented The Heritrix README and the official job configuration documentation at heritrix.readthedocs.io make no mention of JavaScript execution or browser-based rendering anywhere. Since Heritrix is software rather than a service, the honest answer is that behaviour depends on how a given operator configured and extended it, and no default is documented either way.
- Obeys robots.txt
- Yes
- Documentation
- github.com
robots.txt
Block Heritrix
User-agent: Heritrix
Disallow: /Allow Heritrix
User-agent: Heritrix
Allow: /Heritrix is not a bot. It is the software that made the last twenty-five years of the web readable.
The Internet Archive built it, open-sourced it under the Apache License 2.0, and still develops it in public on GitHub. It is the crawler behind the Wayback Machine. When you follow a dead link and land on an archived capture from 2009, you are reading something Heritrix fetched and stored. That is a genuinely unusual piece of infrastructure to have running in the world, and almost nobody who blocks it has thought about what they are blocking.
Why the user agent tells you almost nothing
Every other entry in the bot directory has one operator behind one token. Read the token, you know who is crawling you and roughly why.
Heritrix breaks that. It is downloadable software. The Internet Archive runs it. So do national libraries running legal deposit programmes, university archives capturing research collections, government agencies preserving their own web estate, and any number of smaller efforts. All of that traffic can look similar in a log.
The project’s own guidance is the fix. The README tells operators to always identify their crawl with contact information in the User-Agent, precisely so that affected sites can reach them or adjust their own behaviour. The user agent ships as a template that each operator fills with their own contact URL.
So the identity is in the string, just not in the token. Pull the contact URL out of the header and go look at it. A national library is not the same visitor as an anonymous crawl by someone who left the template untouched, and treating those two the same is how people end up blocking their own preservation.
Can it read JavaScript
Undocumented, and for once the reason is interesting rather than evasive.
I read the Heritrix README and the official job configuration documentation. Neither mentions JavaScript execution, headless browsing, or browser-based capture anywhere. There is no default stated in either direction.
That silence means something different here than it does for a hosted crawler. Heritrix is a framework with a configurable processing chain, so what any given crawl does depends on how the operator assembled and extended it. Two Heritrix crawls hitting your server this week can behave differently, and the token will not distinguish them.
For your purposes the conclusion is the same as everywhere else: content that only exists after JavaScript runs is content that may not be captured. The difference is the stakes. With a search crawler you lose a ranking. With an archival crawler you lose the record. If your site went down tomorrow, the version that survives is the version Heritrix could see, and a client-side rendered page can archive as an empty shell.
How to control it with robots.txt
The README states Heritrix is designed to respect robots.txt exclusion directives and nofollow meta tags, and its configuration documentation makes the policy explicit. robotsPolicyName takes four documented values:
| Value | Documented behaviour |
|---|---|
obey | Obey robots.txt directives and nofollow robots meta tags. This is the default |
classic | Same as obey |
robotsTxtOnly | Obey robots.txt directives but ignore robots meta tags |
ignore | Ignore robots.txt directives and robots meta tags |
That table is worth sitting with. Compliance is the shipped default, which is why we list respectsRobotsTxt as yes. It is also a setting the operator can change, and ignore is a supported, documented option. An archive running a legal deposit mandate may well set it. So write the rule, then verify it held rather than assuming.
To exclude archival crawling:
User-agent: heritrix
Disallow: /To allow capture while keeping genuinely private paths out:
User-agent: heritrix
Allow: /
Disallow: /account/
Disallow: /checkout/Note that a rule aimed at heritrix does not address the Internet Archive specifically, because the software is not the operator. The Internet Archive’s own documented route for removal is to write to info@archive.org and ask that content be removed from the Wayback Machine and from future crawling. Worth knowing before you write a rule and assume it landed.
Should you block it
Almost certainly not, and the reason is not sentimental.
The Wayback Machine is the only reason anyone can read a website after it dies. Companies get acquired, domains lapse, CMS migrations eat a decade of URLs. When that happens, the archive is where your own history goes to survive, and it is where journalists, courts, customers, and acquirers go to check what you said and when. Blocking archival capture is opting out of your own evidence.
There is no AI visibility argument here in either direction. Heritrix does not feed model training and it does not decide whether an assistant cites you. If you are working through an access policy, the crawlers that actually change that outcome are covered in AI crawler access. This one is not one of them, and the right decision is usually to leave it alone.
Common questions
Is Heritrix an AI crawler?
No. Heritrix is an archival crawler. It captures pages so they can still be read after the original site changes or disappears. It has nothing to do with model training or AI answer surfaces, and blocking it will not affect either.
Who is crawling me if I see Heritrix in my logs?
You cannot tell from the token alone, and that is the important part. Heritrix is open-source software, not one company's bot. The Internet Archive runs it, and so do national libraries, university archives, and government web preservation programmes. The identity you want is the contact URL the operator put in the user agent string, which the project's own guidance tells operators to include.
Does Heritrix obey robots.txt?
By default, yes. Its documented default robots policy is 'obey', which honours robots.txt directives and nofollow robots meta tags. But the policy is a configuration setting, and 'ignore' is a documented option. So compliance is the shipped default and the operator's choice, not a guarantee.