Bytespider: User Agent, robots.txt, and Why ByteDance Documents Nothing

At a glance

User-agent token

Bytespider

Full user-agent

Bytespider (ByteDance publishes no first-party crawler documentation, so no exact user-agent string can be sourced. Match on the Bytespider token in your logs)

Operated by

ByteDance

What it does

Undocumented. ByteDance has published no statement of what this crawler collects or what it is used for

Executes JavaScript

Not documented ByteDance publishes no crawler documentation of any kind, so there is nothing to read. Vercel's December 2024 crawler study included ByteDance among the operators it measured and found that the AI crawlers it observed fetched JavaScript files without executing them.

Obeys robots.txt

Not documented

Bytespider is ByteDance’s web crawler. That sentence is close to the limit of what can be stated from a first-party source, and the gap between how little is documented and how confidently it is described elsewhere is the real story here.

What ByteDance publishes: nothing

ByteDance has published no crawler documentation. No policy page, no user-agent reference, no declared purpose, no IP range list, no robots.txt commitment. Every other operator in /bots/ has at least a page you can open and read. ByteDance does not.

That absence has consequences for anything you read about this crawler, including tables that give it a tidy yes or no in a rendering column. Those values are not sourced. They cannot be, because there is no source.

It also creates a vacuum that other people fill. One domain is widely cited as ByteDance’s official bot documentation and is in fact a parked domain listed for sale. If you are making a decision about your own infrastructure based on a page like that, you are taking configuration advice from a lander.

So this profile deliberately reads thinner than the others. Undocumented is the accurate answer, and inventing a confident one would make this page worse rather than more useful.

What is actually observable

Two things are real and checkable without ByteDance’s help.

The first is your own access log. The Bytespider token shows up in server logs across the web, at volume, and you can measure its behaviour on your own site directly: how often it comes, which paths it takes, whether it slowed down after you wrote a directive. That is first-hand evidence and it beats any table.

The second is Vercel’s rise of the AI crawler study, published 17 December 2024. It instrumented real traffic across OpenAI, Anthropic, Meta, ByteDance, Perplexity, Google, Apple, and Common Crawl. Its headline finding was that the AI crawlers it measured fetched JavaScript files without executing them. It remains the only public experiment of its kind and it has not been updated since.

Everything else circulating about Bytespider is inference, log anecdote, or repetition.

Can it read JavaScript

Undocumented, and it will stay undocumented until ByteDance publishes something.

There is no vendor statement to read. The only relevant measurement is Vercel’s, which grouped ByteDance among the operators it tested and reported non-execution of fetched JavaScript across the crawlers it observed. That is a reasonable lead and it is twenty months old.

The right response is architectural rather than investigative. If your pages carry their substance in the initial HTML, the answer to this question does not change anything you do. If they do not, then you have an undocumented crawler making an undocumented decision about content it may never see, and no amount of reading will resolve it. The reasoning is in AI crawlers and JavaScript rendering.

How to control it with robots.txt

Write the directive. Then verify it, because nothing has been promised to you.

Block it entirely:

User-agent: Bytespider
Disallow: /

Allow it while keeping private areas out:

User-agent: Bytespider
Allow: /
Disallow: /account/
Disallow: /checkout/

Because there is no published compliance commitment, treat both of these as a request rather than a control. The verification step is not optional here the way it might be with a documented crawler. Search your access logs for the Bytespider token before the change and after it, and compare. If the pattern does not change, you have learned something no article can tell you, and your next lever is at the network or firewall layer rather than in a text file.

One caution on log analysis. The user agent header is free text and anyone can send it. A hit claiming to be Bytespider may be an unrelated scraper borrowing the name, and with no published IP list there is no way to confirm which. Weigh conclusions about traffic volume accordingly.

Common questions

Does Bytespider obey robots.txt?

Unknown. ByteDance publishes no crawler documentation, so there is no commitment to point at either way. Write the directive, then check your own access logs for the Bytespider token and see whether the behaviour changed. Your logs are the only evidence available for this crawler.

Where are ByteDance's official crawler docs?

There are none. ByteDance has not published a crawler policy page, a user-agent reference, or an IP range list. Third-party pages that appear to be official ByteDance bot documentation should be treated with suspicion. One domain widely cited as the source is a parked listing for sale, not a ByteDance property.

Does Bytespider execute JavaScript?

Undocumented, and it cannot become documented while ByteDance publishes nothing. Vercel's December 2024 study included ByteDance in the operators it measured and found the AI crawlers it observed fetched JavaScript files without executing them. Assume your content needs to exist in the initial HTML.