YandexBot: User Agent, robots.txt, and Why It Is Not an AI Crawler

User-agent: YandexBot
Disallow: /

User-agent: YandexBot
Allow: /

YandexBot is the main indexing crawler for Yandex, the search engine that carries the majority of search volume in Russia and a meaningful share across Central Asia and the Caucasus. It is a conventional search bot. It fetches pages, builds an index, and decides what can rank.

Most people who end up reading a page like this got here from a server log. They saw an unfamiliar token, searched it, and half-expected to find out they had been quietly harvested for model training. That is not what happened here.

What it actually is

Yandex publishes a full robot roster with the exact header each one sends. The main indexing robot identifies itself as:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Related robots share the same shape with a different token: YandexImages/3.0 for image indexing, YandexComBot/3.0 for multilingual content. Yandex’s documentation notes that User-agent: YandexBot in robots.txt addresses the main indexing robot specifically, while User-agent: Yandex addresses the whole family.

That family distinction is the part people get wrong. A blanket User-agent: Yandex block takes out image indexing, video, and every other surface at once, which is almost never what someone actually intended when they were trying to stop one noisy crawler.

Verify before you act on it

YandexBot is one of the more commonly spoofed tokens, because it is trivial to copy a string into a scraper and it buys a soft pass on a lot of naive firewall rules.

Yandex documents the check. Pull the IP from your logs, run a reverse DNS lookup, and confirm the hostname ends in yandex.ru, yandex.net, or yandex.com. Then do a forward lookup on the hostname you got back and confirm it returns the original IP. Both directions have to hold. If they do not, you are looking at something wearing YandexBot’s name, and no robots.txt rule you write will affect it, because it was never going to read robots.txt in the first place.

This matters more than the blocking question. A lot of “aggressive YandexBot” complaints are actually scrapers, and treating them as a search engine problem means the real source keeps running.

Can it read JavaScript

Yandex does not say.

I read the current Yandex webmaster documentation on robot identification and user agents. It covers what each robot is for, the exact header each sends, how to verify authenticity by reverse DNS, and how robots.txt directives are applied. Nowhere does it address JavaScript execution, headless rendering, or how page content is retrieved.

So the answer here is undocumented, and it stays undocumented until Yandex writes something down. Tables that print a confident yes or no for this row are filling in a gap with inference.

The practical response is the same one that works for every crawler in the bot directory: put the content that matters in the initial HTML response. If your text only exists after a client-side fetch resolves, you are betting your visibility on undocumented behaviour, and the downside is silent.

How to control it with robots.txt

Block the main indexing robot only:

User-agent: YandexBot
Disallow: /

Block every Yandex robot at once:

User-agent: Yandex
Disallow: /

Allow indexing while keeping private areas out:

User-agent: YandexBot
Allow: /
Disallow: /account/
Disallow: /checkout/

One honest caveat from Yandex’s own documentation: it states that some Yandex robots may ignore robots.txt directives, including those written for User-agent: Yandex. The main indexing robot is documented as obeying, but the operator has told you in writing that the family is not uniform. Check your logs after a change rather than assuming the file did what you asked.

What this means for AI visibility

Nothing, directly. YandexBot is search infrastructure. It has no role in AI answer surfaces, training corpora, or citation in a chat assistant.

If you came here mid-audit, the useful move is to finish the audit against the crawlers that do matter for that. Our guide to AI crawler access covers which tokens control AI answer eligibility and which ones control training, because those are two different decisions and a single blanket rule collapses them into one bad outcome.

Yandex does have one AI-adjacent token, and it is YandexAdditional. It is not a crawler. It is a robots.txt control you use to keep already-indexed pages out of Yandex’s AI answers, which is a genuinely different lever from the one on this page.