PerplexityBot: User Agent, robots.txt, and What It Can Actually Read
What PerplexityBot is, its exact user agent, how to allow or block it, and the honest answer on whether it executes JavaScript.
At a glance
- User-agent token
PerplexityBot- Full user-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)- Operated by
- Perplexity
- What it does
- Crawls and indexes pages so they can be surfaced and linked in Perplexity search results
- Executes JavaScript
- Not documented Perplexity does not address rendering in its documentation. No published beacon test has observed PerplexityBot directly, so treat any yes or no you read as inference.
- Obeys robots.txt
- Not documented
- Documentation
- docs.perplexity.ai
robots.txt
Block PerplexityBot
User-agent: PerplexityBot
Disallow: /Allow PerplexityBot
User-agent: PerplexityBot
Allow: /Perplexity does not document that PerplexityBot obeys robots.txt. Treat these directives as a request rather than a control, and verify against your own server logs.
PerplexityBot is Perplexity’s indexing crawler. It exists so that pages can be surfaced and linked inside Perplexity’s answers, and Perplexity states plainly that it “is not used to crawl content for AI foundation models.”
That distinction matters more than it sounds. Most of the anxiety about AI crawlers is really anxiety about training. PerplexityBot is the other kind: it is the crawler that decides whether you are eligible to be cited at all.
What it is for
Perplexity runs two declared user agents and they do different jobs.
| User agent | Job |
|---|---|
PerplexityBot | Builds the index. Determines whether your pages can appear as a source |
Perplexity-User | Fetches a page live, because a user’s question required it |
If you block PerplexityBot, you are opting out of being indexed as a citable source. That is a much larger decision than opting out of training, and it is the one people most often make by accident with a blanket rule.
Can it read JavaScript
Perplexity has never said, and this is where most published tables overreach.
I read Perplexity’s current bot documentation in full. It covers what each crawler is for and how to control it. It does not mention JavaScript, rendering, headless browsing, or how page content is fetched. There is no last-updated date on the page.
The widely repeated answer of “no” comes from Vercel’s December 2024 crawler study, which found that the AI crawlers it measured fetched JavaScript files without executing them. That study is credible and it is the best public evidence available, but it is now twenty months old and carries no update note.
So the honest position is: not documented, and weakly evidenced even empirically. Build your pages so it does not matter. If your content only exists after JavaScript runs, you are betting your visibility on a behaviour nobody has committed to.
How to control it
Block it entirely:
User-agent: PerplexityBot
Disallow: /Allow it while keeping specific areas private:
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /checkout/Because Perplexity does not document robots.txt compliance for this crawler, verify rather than assume. Check your server logs for the PerplexityBot token after a change and confirm the behaviour matches what you asked for.
What to do instead of blocking
If the worry is content theft, blocking the indexing crawler is the wrong lever. It removes you from a channel that cites sources visibly and passes referral traffic, while doing nothing about training.
The more productive move is making sure that what PerplexityBot can reach is worth citing: content that exists in your initial HTML, states facts plainly, and answers the question in the first paragraph rather than the fifth.
Common questions
Does PerplexityBot execute JavaScript?
Perplexity does not document rendering behaviour anywhere in its crawler documentation, and no published experiment has observed PerplexityBot directly with an instrumented test. Any confident yes or no you find is inference. Assume it cannot render, because that assumption costs you nothing and the alternative risks invisibility.
Is PerplexityBot the same as Perplexity-User?
No. PerplexityBot crawls to build an index. Perplexity-User fetches a page live because someone asked a question that needs it. They are separate user agents and you can allow one and block the other.
Should I block PerplexityBot?
Usually not. Perplexity states this crawler is for surfacing and linking sites in results, not for training foundation models. Blocking it removes you from a surface that sends referral traffic and cites sources visibly.