MarketLensEvery prior web standard for machines answered the question “may you crawl this?” or “what URLs exist?” None answered the question language models actually have: what is this site about, and which pages matter? The llms.txt standard is the first serious attempt at that missing layer — and eighteen months after its proposal, it sits in a strange place: widely deployed, genuinely useful to agents, and still unrecognized by the biggest crawlers on the web.
This guide covers the standard end to end: where it came from, the exact format rules, the adoption landscape, the skeptic case against it, and a practical decision framework for whether your site should publish one.
Where llms.txt Came From
The proposal was published in September 2024 by Jeremy Howard, co-founder of Answer.AI and creator of fast.ai, at llmstxt.org. The motivating observation was concrete: context windows are finite, and HTML is a hostile format for them. A documentation page that reads as 800 words to a human often costs 10,000+ tokens once navigation, scripts, cookie banners, and markup are included. Howard’s answer borrowed the shape of robots.txt — a well-known file at a well-known path — but inverted its purpose. Instead of restricting machines, llms.txt briefs them.
The proposal also defined a companion convention that gets less attention but arguably matters more: appending .md to any page URL should return a clean Markdown version of that page. Together, the index file plus per-page Markdown mirrors form a parallel, machine-native view of a website.
The Format, Precisely
The specification is strict Markdown with a fixed skeleton, in this order:
- One H1 — the site or project name. This is the only mandatory element.
- A blockquote — a short summary of the site containing the key facts an agent needs to interpret everything else.
- Free-form paragraphs or lists — any additional context, with no headings.
- H2-delimited file lists — each H2 names a category (“Docs”, “Guides”, “Policies”); under it, a Markdown list where each item is
[title](url)optionally followed by: description. - An H2 named
Optional— a reserved section name. Agents short on context are explicitly permitted to skip everything under it.
A minimal valid file is seven lines:
# Acme Analytics
> Privacy-first web analytics for static sites. No cookies, EU-hosted, GDPR-compliant by default.
## Docs
- [Quickstart](https://acme.io/docs/quickstart.md): Install the snippet and see data in 5 minutes
- [API Reference](https://acme.io/docs/api.md): REST endpoints, auth, and rate limitsWhy Markdown rather than JSON or XML? Because the consumer is a language model. Markdown survives tokenization cheaply, needs no parser, and doubles as human-readable documentation. The structure is nevertheless regular enough that programmatic tools can parse it deterministically — the reference implementation converts the file to XML for injection into system prompts.
What Belongs in the File — and What Doesn’t
The standard’s failure mode is treating it as a second sitemap. A sitemap enumerates; llms.txt editorializes. Every line should answer “why would an agent fetch this?” Practical selection rules from MarketLens audit work:
- Include: cornerstone guides, product/service definition pages, pricing, API references, About pages that establish who you are. These are the pages agents cite.
- Summarize, don’t enumerate: archives, tag pages, category indexes. One link to the archive under
Optionalbeats 200 individual links. - Exclude: login pages, legal boilerplate, campaign landing pages, anything you would not want quoted as a definitive statement of what you do.
Descriptions carry more weight than titles. An agent deciding between forty links ranks them almost entirely on your one-line annotations, so write them like search snippets: front-load the differentiating noun phrase, keep them under ~140 characters, never repeat the title verbatim.
Adoption Landscape: Who Actually Ships It
Adoption clusters into three tiers. The first is documentation platforms: Mintlify generates llms.txt automatically for every hosted docs site, which alone accounts for a large share of all deployments; Anthropic, Cloudflare, and a long list of developer-tool companies publish the files for their own documentation. The second tier is agentic tooling — AI coding assistants and custom RAG pipelines that probe for the file because a pre-curated index is strictly cheaper than crawling. The third tier, conspicuously absent, is mainstream search: Googlebot, Bingbot, and GPTBot have not committed to consuming the file for ranking or answer generation.
That absence fuels the skeptic case, and it deserves a fair hearing. Google’s John Mueller has compared llms.txt to the keywords meta tag — a self-declared signal with no verification, historically an invitation to spam. The honest synthesis: llms.txt currently serves the agentic segment of AI traffic — coding assistants, custom retrieval pipelines, documentation readers — rather than the search segment. Nobody should ship the file expecting Google AI Overviews to change behavior next week.
Should Your Site Publish One? A Decision Framework
| Site type | Agent retrieval likelihood | Verdict |
|---|---|---|
| Developer docs / API reference | Very high — agents quote docs daily | Ship both llms.txt and llms-full.txt |
| SaaS marketing + blog | High for comparison and how-to queries | Ship llms.txt; add llms-full.txt if content is deep |
| Local service business | Moderate — assistants answer “who does X near me” | Ship a short llms.txt (services, pricing, area) |
| Brochure site, 5 pages | Low | Optional; cost is trivial either way |
| Paywalled / proprietary content | Negative — you may not want ingestion | Skip it, and tighten crawler policy instead |
Note the last row: llms.txt is an invitation, and access control is a different instrument. Decide your ingestion policy first via robots.txt directives for GPTBot, Perplexity, and Claude, then curate what you do expose.
If you publish one, automate it. A hand-maintained index rots within weeks, so generate it from your content collection at build time — the same template-driven approach used to inject JSON-LD schema from Hugo templates — and let a GitHub Actions deployment pipeline rebuild and republish it on every content change.
The Bottom Line
llms.txt is cheap, honest infrastructure: a few hours to implement, near-zero maintenance once automated, real value for the agent traffic that increasingly initiates first contact with your brand, and no downside beyond the effort. It is not a ranking signal and nobody credible should sell it as one. Publish it for what it is — the machine-readable front door of your site — and pair it with the access and performance layers that AI crawlers actually enforce. If you want a prioritized read on where your site stands across all three layers, that is precisely what a MarketLens audit delivers.