llms.txt is a good idea with a marketing problem. The idea is a curated markdown index of your site, written for machines that have a limited context window and no patience for your navigation. The marketing problem is that it is named like robots.txt, which leads people to assume it has the same status: a file the crawlers check.
It does not have that status. Here is what I can verify and what I cannot.
What it is
Jeremy Howard proposed llms.txt, published 3 September 2024. The site now hosts v2 of the proposal, described there as “updated based on what I learned from two years of adoption.” The page carries a modified date of 10 August 2026, though it does not state a publication date for v2 specifically.
The format is markdown at /llms.txt, or at a subpath such as /docs/llms.txt. The spec calls for:
- An H1 with the project or site name. This is the only mandatory section.
- A blockquote with a short summary.
- Optional detail sections.
- H2-delimited file lists of markdown links, each with an optional note.
That is deliberately minimal. It is a table of contents a model can read in one pass, pointing at the pages that actually matter, with a sentence of context each.
Note what is not in there: no access rules, no allow or disallow, no crawl-delay, no enforcement of any kind. llms.txt is an index, not a policy. If you write “do not train on this” in it, nothing happens.
Who publishes one
Plenty of people, and the adoption story is genuinely real on the publishing side.
llmstxt.org states that “the AI labs themselves publish llms.txt files for their own developer docs: OpenAI, Anthropic, and Gemini.” It also states that “thousands of sites publish an llms.txt file, documentation platforms generate one automatically, and Chrome’s Lighthouse audits sites for one,” and names Mintlify, GitBook, Yoast SEO, AIOSEO and Wix as platforms that produce them. Directory sites exist to catalogue them.
I can independently confirm one piece of that. OpenAI’s crawler documentation opens by pointing readers at its own /llms.txt as the complete documentation index. So OpenAI does publish one.
The question that actually matters
Publishing an llms.txt for your own docs and reading other people’s llms.txt when you crawl are two completely different commitments. Almost every article about this file blurs them, and the blur is what makes readers think it works.
So: has any major AI vendor committed to reading your llms.txt?
Here is what the primary documentation shows.
Google. Its crawler documentation does not mention llms.txt anywhere. The whole page is user agent tokens and robots.txt directives. Separately, Google’s AI features documentation says there are “no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary,” and that “robots.txt directives for Googlebot is the control for site owners to manage access.” Both statements point at robots.txt and neither leaves room for a second file that changes eligibility.
OpenAI. Its bot documentation describes OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, and the controls it discusses are robots.txt directives. It links to its own llms.txt as a docs index. It does not say its crawlers fetch or use yours.
Perplexity. Its bots guide documents PerplexityBot and Perplexity-User, robots.txt behaviour and IP ranges. Nothing about llms.txt.
I could not find a public commitment from any major AI vendor to read llms.txt from third-party sites as an input to retrieval, ranking or citation. That is a statement about what I could verify from primary crawler documentation, not proof that no such commitment exists anywhere. If a vendor publishes one, this page should change. As of writing, the honest position is that the consumption side is unproven, and the frequency with which articles imply otherwise is the reason this page exists.
So is it worthless?
No, and this is where the discourse usually collapses into two wrong camps.
Where llms.txt genuinely works today: when something is pointed at it deliberately. Give a coding agent your docs URL and an llms.txt gives it a clean map instead of a scrape of your nav bar. Paste it into a context window and you have a compact index rather than 40,000 tokens of HTML. Internal RAG systems, developer tooling and agent workflows can be configured to fetch it, and many are. That is a real, working use case with a real payoff, and it explains why documentation platforms generate the file automatically.
Where it does not work: passive discovery. Nothing about publishing the file causes an AI search engine to find you, prefer you, or cite you. There is no evidence of a passive channel from your llms.txt into an engine’s retrieval pipeline.
The distinction is between a file that is useful when requested and a file that changes how you are indexed. llms.txt is well evidenced as the first and unevidenced as the second.
What to actually do
If you publish docs or an API, ship one. It is cheap, most platforms generate it, and it makes your documentation materially more usable to agents and developers who point tooling at it. Keep it current, because a stale index is worse than none.
If you run a marketing site and are hoping for AI citations, deprioritise it. Ship it if it costs you an hour, and expect nothing from it. Spend the effort on the things with a documented mechanism instead.
Never treat it as access control. It grants and denies nothing. Your crawler policy lives in robots.txt and at the network layer. Check yours against the AI crawler directory and make sure you are allowing search-index bots, since blocking those is the one action guaranteed to remove you from AI answers. See AI user agents for the job-by-job breakdown.
Fix the layers that are documented first. In order: is a search crawler reaching your pages, does your content exist in HTML rather than after JavaScript runs, and do your pages carry claims a model can ground an answer in. That last one is the whole of AI grounding, and most AI crawlers document nothing about JavaScript rendering, which makes the second item a much bigger risk than a missing markdown file.
How to spot bad advice about it
Three tells:
- It cites adoption numbers as evidence of effect. Thousands of sites publishing a file tells you the file is easy to publish. It tells you nothing about whether anything reads it.
- It says “AI labs support llms.txt” without saying which direction. Publishing one for your own docs is not supporting the standard as a consumer. Ask which crawler fetches it.
- It quotes a ranking improvement. Nobody can attribute a change in AI answers to a single file, because no engine exposes the counterfactual and the answers move on their own.
For our shorter reference on the format itself, see llms.txt.
The short version
llms.txt is a well-designed markdown index at your site root, proposed by Jeremy Howard, widely published, and genuinely useful when an agent or a developer is pointed at it.
What I cannot find is any major AI vendor committing to read yours. Google’s crawler docs do not mention it, and Google says robots.txt for Googlebot is the control for AI features. Publish one if it is free. Do not build a strategy on it, and be sceptical of every article that implies the citation benefit has been demonstrated, because as far as the primary sources go, it has not.