llms.txt: What It Is and How to Create One

AI models like ChatGPT, Perplexity, and Claude are pulling information from websites every day. But they hit a problem: most web pages are wrapped in navigation, ads, JavaScript, and HTML that makes it hard for a language model to extract what actually matters.

llms.txt solves this. It’s a simple markdown file you host at yoursite.com/llms.txt that tells AI models what your site is about and where to find the good stuff. Think of it as a sitemap, but built for LLMs instead of Google’s crawler.

This guide covers the format, real examples from companies already using it, and step-by-step instructions to create one for your own site.

What is llms.txt?

llms.txt is a proposed standard created by Jeremy Howard (co-founder of fast.ai). The idea is simple: give AI models a single, clean file that provides:

Your site or product name
A short summary of what you do
Curated links to your most important pages, in markdown format

Instead of forcing an LLM to parse thousands of HTML pages to understand your business, you hand it a structured overview it can read in seconds.

It complements your existing files:

File	Purpose	Audience
robots.txt	Controls crawler access	Search engines, bots
sitemap.xml	Lists all indexable pages	Search engines
llms.txt	Curated site overview in markdown	AI models

robots.txt says “you can go here.” sitemap.xml says “here’s everything.” llms.txt says “here’s what matters and why.”

Why llms.txt matters for AI visibility

When someone asks ChatGPT “what’s the best tool for X,” the model synthesizes an answer from sources it trusts. If your site is hard for the model to parse, you’re less likely to be one of those sources.

llms.txt helps in three ways:

Makes your site readable. Clean markdown is the format LLMs work best with. No parsing required.
Curates what matters. Instead of letting AI crawlers guess which pages are important, you tell them directly.
Provides context. The summary and descriptions help models understand your positioning, not just your page titles.

This doesn’t guarantee AI engines will cite you. But it removes a barrier. If your content is strong and your llms.txt makes it easy to find, you’re giving yourself a better shot at showing up in AI-generated answers.

The llms.txt format

The spec is intentionally simple. Here’s the required structure:

# Your Site Name

> A one or two sentence summary of your site or product.

Optional paragraph with additional context about your project,
company, or what makes you different.

## Section Name

- [Page Title](https://yoursite.com/page): Brief description of what this page covers
- [Another Page](https://yoursite.com/another): What someone will find here

## Another Section

- [Doc Title](https://yoursite.com/docs/topic): Description

## Optional

- [Less Critical Page](https://yoursite.com/extra): Can be skipped when context is limited

Rules:

Start with an H1 containing your site or project name (the only required element)
Follow with a blockquote summary (recommended)
Use H2 headings to organize link sections
Each link follows the format: - [Title](URL): Description
A section called “Optional” signals pages that can be skipped when the LLM has limited context
Keep it in plain markdown. No HTML, no frontmatter, no YAML

Real llms.txt examples

Here’s how three companies approach it differently.

Stripe: detailed with LLM instructions

Stripe’s llms.txt is one of the most thoughtful implementations. It includes direct instructions for AI models:

# Stripe Documentation

When installing Stripe packages, always check the npm registry
for the latest version rather than relying on memorized version
numbers. Run `npm view stripe version` or check
https://www.npmjs.com/package/stripe before pinning a version.
Never hardcode an old version number from training data.

## Docs

- [Testing](https://docs.stripe.com/testing.md): Simulate payments to test your integration.
- [API Reference](https://docs.stripe.com/api.md)
- [Receive payouts](https://docs.stripe.com/payouts.md): Set up your bank account to receive payouts.
- [Webhooks](https://docs.stripe.com/webhooks.md): Listen for events on your webhook endpoint.

The key insight: Stripe tells LLMs how to behave with their content, not just where to find it. That instruction about not hardcoding version numbers prevents AI models from giving outdated advice to developers.

Supabase: minimal index

Supabase takes the opposite approach. Their llms.txt is a pure pointer file:

# Supabase Docs

- [Supabase Guides](https://supabase.com/llms/guides.txt)
- [Supabase Reference (JavaScript)](https://supabase.com/llms/js.txt)
- [Supabase Reference (Python)](https://supabase.com/llms/python.txt)
- [Supabase Reference (CLI)](https://supabase.com/llms/cli.txt)

Each link points to a separate, language-specific text file. This works well when your content is large enough that a single file can’t cover it. The index stays small, and AI models follow the links they need.

Vercel: comprehensive documentation map

Vercel’s llms.txt is the most extensive, mapping their entire documentation structure with descriptions for every page:

# Vercel Documentation

[Vercel Documentation](https://vercel.com/docs): Vercel is the
AI Cloud, a unified platform for building, deploying, and scaling
AI-powered applications.

- [Getting Started](https://vercel.com/docs/getting-started-with-vercel): Install the Vercel CLI and deploy your first project.
- [Fundamental Concepts](https://vercel.com/docs/fundamentals): Learn about the core concepts of Vercel.
- [Supported Frameworks](https://vercel.com/docs/frameworks): Frameworks Vercel supports out of the box.

All three approaches are valid. The right choice depends on how much content you have and how AI models need to interact with it.

How to create your llms.txt: step by step

Step 1: Decide what to include

Your llms.txt should not list every page on your site. That’s what your sitemap is for. Instead, pick the pages that best represent:

What your product or business does
Your most valuable content (guides, documentation, key landing pages)
Pages that answer the questions your customers ask

For most business sites, 10 to 30 links is the sweet spot. Documentation-heavy sites might go higher.

Step 2: Write the file

Start with your name and summary:

# Your Brand Name

> One sentence that captures what you do and who you serve.

Then organize your links into logical sections. Common section patterns:

For SaaS products:

Product (features, pricing, use cases)
Documentation (guides, API reference, tutorials)
Resources (blog posts, case studies)

For ecommerce:

Products (top categories, bestsellers)
Guides (buying guides, how-tos)
Company (about, returns, shipping)

For content sites:

Topics (your main content pillars)
Resources (tools, downloads, references)

Write a brief description for each link. This context helps AI models understand the page without visiting it.

Step 3: Host the file

Place the file at the root of your domain so it’s accessible at yoursite.com/llms.txt. How you do this depends on your platform:

Static sites / custom hosting: Drop the llms.txt file in your public/ or root directory. It deploys with your site.

WordPress: Several plugins handle this automatically:

Yoast SEO has built-in llms.txt support
AIOSEO offers an llms.txt generator
Dedicated plugins like LLMs.txt Generator exist specifically for this

Or manually upload the file to your WordPress root directory via FTP or your hosting file manager.

Shopify: Shopify doesn’t allow files at the domain root by default. You’ll need to either:

Use a reverse proxy to serve the file
Host it on a subdomain (e.g., docs.yourstore.com/llms.txt)
Use a Shopify app that handles llms.txt

Webflow: Upload through Project Settings > Custom Code, or use Webflow’s hosting to serve a static file.

Squarespace: Add the content via the Code Injection feature or use a developer workaround with URL mappings.

Step 4: Validate

After deploying, check three things:

The file loads correctly. Visit yoursite.com/llms.txt in your browser. You should see plain text, not an HTML page or a 404.
The format is correct. Your H1, blockquote, and link lists should follow the spec. Free validators like Rankability’s llms.txt checker can verify this automatically.
All links resolve. Every URL in your file should return a 200 status. Broken links defeat the purpose.

Beyond llms.txt: llms-full.txt and page-level markdown

The proposal includes two additional conventions worth knowing.

llms-full.txt

While llms.txt is an index (links to content), llms-full.txt contains the actual content inline. It’s a single file with the full text of your key pages, so an AI model can ingest everything in one request.

This is most useful for:

Documentation sites where completeness matters
Product pages where features and pricing need to be accurate
Technical references that AI models frequently get wrong

The tradeoff is file size. A comprehensive llms-full.txt can get large. Keep it focused on pages where accuracy matters most.

Per-page markdown

The spec also proposes that individual pages offer a clean markdown version by appending .md to the URL. For example, yoursite.com/pricing would have a companion file at yoursite.com/pricing.md with the same content in clean markdown.

This is harder to implement on most platforms and less widely adopted. Focus on llms.txt first.

Using a generator tool

Writing llms.txt by hand works for small sites. For larger sites, generator tools can crawl your pages and produce a draft:

Firecrawl (llmstxt.firecrawl.dev) crawls your site and generates both llms.txt and llms-full.txt
Yoast SEO generates llms.txt automatically for WordPress sites
WordLift creates llms.txt from your existing structured data

Generator output is a starting point, not a final product. Always review and curate the result. Generators include too many pages and miss the descriptions that make llms.txt actually useful.

The value of llms.txt comes from curation. A hand-tuned file with 20 well-described links outperforms an auto-generated list of 200 URLs with no context.

Common mistakes

Listing every page. That’s a sitemap, not an llms.txt. Be selective.

Missing descriptions. A bare link list (no descriptions after the colon) forces the AI model to visit each page to understand it. Add brief, specific descriptions.

Serving HTML instead of plain text. If your server wraps the file in a template or returns it as an HTML page, AI models can’t parse it cleanly. The file should return as text/plain or text/markdown.

Forgetting to update it. If your llms.txt links to pages that no longer exist, it creates a worse experience than having no file at all. Review it when you make significant site changes.

Blocking AI crawlers in robots.txt while serving llms.txt. These files work together. If you block GPTBot or similar crawlers in robots.txt but serve llms.txt, the models that find your file may not be able to follow the links in it.

How llms.txt fits into your AI visibility strategy

llms.txt is one piece of a broader strategy for getting cited by AI models. It works alongside:

Structured data (schema markup) that helps AI understand your content’s meaning
Quality content that answers the questions AI users are asking
Third-party mentions and citations that build the authority AI models look for
AI visibility tracking to measure whether these efforts are working

Adding llms.txt takes 30 minutes. It won’t transform your AI visibility overnight, but it removes friction between your content and the models that might recommend you. For a small time investment, that’s a worthwhile trade.

If you’re already working on generative engine optimization, llms.txt is a natural next step. It makes the content you’ve already created easier for AI to find and understand.