Twitterbot: The X Link Preview Fetcher, Not an AI Crawler

User-agent: Twitterbot
Disallow: /

User-agent: Twitterbot
Allow: /

Twitterbot is a link preview fetcher. When someone posts your URL on X, the platform fetches the page so it can build the card that appears under the post: image, title, description.

That is the whole job. It is not indexing you, not ranking you, and not feeding a model. It shows up because a human shared your link.

This page exists because that distinction is invisible in a log file. You see an unfamiliar user agent, you search it, and half the internet is currently primed to assume every unfamiliar bot is an AI crawler harvesting content. Twitterbot is not one, and knowing which category a bot belongs to is the difference between a five-second decision and an afternoon of pointless firewall work.

What I could and could not verify

Being straight about this, because the alternative is copying someone else’s copy.

The token is Twitterbot. That part is unambiguous, it is what appears in server logs, and it is what robots.txt rules address.

The full user agent header is a different matter. X’s developer documentation on cards was not publicly retrievable at the time of writing. Requests to developer.x.com returned HTTP 402, and the equivalent paths on docs.x.com returned 404. So there is no first-party page to quote, and this page will not invent a header string, a version number, or a rendering claim to fill the gap.

If you need the exact header for a firewall allowlist, take it from your own logs rather than from any table on the internet, including this one. Your logs are the primary source you actually control.

Why it behaves the way it does

Twitterbot is user-triggered, which makes its traffic shape completely different from a scheduled crawler.

A search crawler arrives on its own schedule and spreads load. A preview fetcher arrives because someone posted a link, which means it can go from zero to a burst and back to zero. A post that takes off produces a cluster of requests in a short window. Nothing about that is hostile, but it does trip naive rate limiting, and it is a common cause of “why did this bot suddenly hammer us” tickets that resolve to “somebody’s post did well.”

The practical consequence: if you block or throttle it, you do not notice immediately. You notice weeks later when someone mentions that your links look bare on X while everyone else’s show an image.

Can it read JavaScript

Undocumented, because the documentation could not be read.

There is no first-party statement available to cite right now, and this field stays undocumented rather than being guessed at. What is safe to say is structural rather than empirical: preview cards are built from meta tags in the document head, and meta tags are the part of a page most likely to be server-rendered even on aggressively client-side sites.

So the guidance does not depend on the answer. Put your card tags in the initial HTML response. If your preview metadata is injected client-side by a framework after hydration, you are relying on undocumented behaviour for something you can trivially make deterministic. The same principle runs through every entry in the bot directory: make the content exist before any script runs and the rendering question stops being your problem.

You can verify your own case in about a minute. Post a link to a test URL from a private account, watch your access logs, and see what came through and what it fetched.

How to control it with robots.txt

To allow preview cards while keeping private areas out:

User-agent: Twitterbot
Allow: /
Disallow: /account/
Disallow: /checkout/

To stop link previews from your site entirely:

User-agent: Twitterbot
Disallow: /

Because X’s current documentation on robots.txt handling could not be retrieved, treat compliance as unverified. Write the rule, then check your logs after a real share to confirm it did what you asked. That is the correct habit for any crawler whose operator has not published a compliance statement you can point at.

Where this sits in an AI visibility audit

Nowhere, directly. Twitterbot has no bearing on whether an assistant cites you, and no bearing on training corpora.

There is one indirect line worth knowing. Content that gets shared and discussed accumulates the third-party mentions that AI answer engines tend to draw on, and a link that renders as a bare URL gets shared less than one that renders as a card. That is a distribution argument, not a crawler argument, and it is a reason not to block rather than a reason to optimise.

If your actual goal is deciding which crawlers to allow for AI visibility, the ones that matter are set out in AI crawler access. Twitterbot is not on that list, and the right move here is usually to leave it alone and make sure your meta tags are in the HTML.