Googlebot: The Only Major Crawler That Documents JavaScript Rendering

User-agent: Googlebot
Disallow: /

User-agent: Googlebot
Allow: /

Googlebot is the crawler almost everything else gets compared to, and for one question in particular it is the outlier: it is among the very few crawlers whose operator actually documents whether it executes JavaScript.

It does. Google’s JavaScript SEO documentation describes three phases: crawl, render, index. Googlebot fetches the HTML, queues the page for rendering, and then, in Google’s words, “Once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript.”

Can it read JavaScript

Yes, with one caveat worth taking seriously.

Rendering is queued, not immediate. Google says every page returning a 200 status gets queued for rendering unless a robots meta tag says otherwise, and that “the page may stay on this queue for a few seconds, but it can take longer than that.” Pages returning non-200 codes may skip rendering entirely.

So client-side content is in a genuinely better position with Googlebot than with any AI crawler. Better is not the same as safe. Rendered content can be indexed later than static content, and a page that fails to render leaves you with whatever was in the initial HTML.

This makes Googlebot the exception rather than the rule. Of the crawlers in our directory, only Googlebot and Applebot document that they render, and Common Crawl documents that it does not. Every other operator is silent. The full picture is in which AI crawlers render JavaScript.

Why it matters for AI search

Googlebot is not only a search crawler now. It is the supply line for Google’s AI features.

Google states that robots.txt directives for Googlebot control crawling for Search, and that this applies to AI features too. There is no separate AI Mode fetcher to allow or block. If Googlebot cannot reach a page, that page cannot appear in AI Overviews or AI Mode either.

That makes Googlebot the single highest-stakes crawler on your site. It governs classic search rankings and AI answer eligibility with one set of rules.

Do not confuse it with Google-Extended, which is not a crawler at all. Google-Extended is a control token for whether your content trains Gemini models. Blocking it leaves Search and AI Overviews untouched. People block it thinking they are opting out of AI, then wonder why they still appear in AI Overviews.

How to control it

Block it entirely, which also removes you from AI Overviews and AI Mode:

User-agent: Googlebot
Disallow: /

Allow it while keeping private areas out:

User-agent: Googlebot
Allow: /
Disallow: /account/
Disallow: /checkout/

Blocking Googlebot is close to leaving the web as far as discovery is concerned. Almost nobody should. The useful controls are narrower: noindex for pages you want crawled but not listed, and Disallow for areas that should never be fetched.

Verifying it is really Googlebot

The user agent string is trivially forged. Google’s long-standing advice is a forward-confirmed reverse DNS lookup: resolve the requesting IP to a hostname, then resolve that hostname back to an IP and confirm it matches. Genuine Googlebot resolves under googlebot.com or google.com.

This matters more than it used to. Scrapers routinely wear Googlebot’s name because so many sites give it privileged access.