TL;DR
- Adoption of llms.txt is still niche — roughly one in ten AI-aware sites publishes one, and no major LLM vendor has publicly committed to reading it in production inference.
- The real consumers today are IDE agents (Cursor, Claude Code, GitHub Copilot, Windsurf), MCP servers, and in-product AI assistants that fetch it on demand.
- Shipping a good llms.txt is roughly a half-day of work — cheaper than almost any other piece of agent-readable infrastructure.
- Treat it as a low-cost option, not a ranking tactic. The upside is asymmetric if adoption spreads; the downside is a few hours of markdown.
- Skip it if your site has no docs, no API, and no reason for an agent to navigate it. Otherwise, ship it and forget it.
llms.txt is a proposed convention: a single markdown file at /llms.txt that gives language models a curated map of your site — key URLs, short descriptions, and optional full-text bundles at /llms-full.txt. It was proposed by Jeremy Howard in late 2024 and picked up quiet traction through 2025. Two years in, the question isn't "what is it" — it's whether the file earns its keep in a 2026 stack.
Who actually reads llms.txt in 2026
Let's separate wishful thinking from observed behavior.
Not reading it (as of writing): OpenAI's ChatGPT, Anthropic's Claude.ai, Google's Gemini and AI Overviews, and Perplexity have not publicly committed to consuming /llms.txt during retrieval or training. Their crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot) respect robots.txt, but there is no confirmed signal that llms.txt influences citation selection in production. Anyone claiming otherwise is guessing.
Actually reading it: the more interesting story is on the agent side.
- IDE coding agents — Cursor, Claude Code, GitHub Copilot's agent mode, Windsurf, Cline — routinely fetch llms.txt when a developer points them at a documentation domain. It gives the agent a low-token index of what's worth pulling into context.
- MCP servers built around documentation (Context7, Ref, various vendor-shipped servers) use llms.txt as a discovery manifest.
- In-product AI assistants — the "ask AI" widgets embedded in SaaS docs — increasingly bootstrap from llms.txt to answer questions about the product they're embedded in.
- Custom RAG pipelines at enterprises where a data team wants a curated seed list rather than a full crawl.
The pattern is consistent: llms.txt is being adopted bottom-up by tools that need a compact, human-curated site map, not top-down by the big inference providers. That's a meaningful distinction when you're deciding where to spend engineering hours.
The honest cost side
A minimum-viable llms.txt is not complicated. If you have decent docs and a clear IA, the work looks like this:
- Draft the file — one H1 (site name), a blockquote summary, and grouped H2 sections with linked bullets. Half a day is realistic for a mid-sized site; a couple of hours if you already maintain a sitemap.
- Decide whether to publish
/llms-full.txt. This concatenates the full markdown of key pages into one file. Useful for agents, but non-trivial to keep fresh — wire it into your build pipeline or don't bother. - Serve it at the root with
Content-Type: text/markdown; charset=utf-8. - Add a line to CI so it doesn't rot when your IA changes.
Ongoing maintenance is the real cost, not initial authorship. A stale llms.txt is worse than none because agents will confidently follow dead links. Budget for a quarterly review at minimum.
Compare that to the cost of, say, building a proper schema.org graph, migrating to SSR for JavaScript-heavy pages, or running a citation-monitoring stack. llms.txt is by a wide margin the cheapest agent-facing artifact you can ship.
The honest benefit side
Two categories of upside, ranked by likelihood.
Likely, near-term: better behavior from agents that already read it. If your users interact with your product or docs through Cursor, Claude Code, or an MCP-connected assistant, a curated llms.txt improves the quality of what those agents retrieve. Documentation vendors like Mintlify and Fern auto-generate the file for this reason — see Mintlify's llms.txt overview and the original llmstxt.org spec. This is a genuine developer-experience win for API and SaaS products.
Speculative, medium-term: LLM vendors adopt the convention. There is no confirmed roadmap for this. But the cost of preparing for it is a rounding error, and the analog to robots.txt (which took years to become universal) suggests conventions can crystallize quickly once one major vendor blinks. Publishing now costs almost nothing; retrofitting under time pressure later costs more.
What llms.txt is not: a citation-ranking signal. Publishing one will not, by itself, get you cited more often in ChatGPT or Perplexity. If that's your goal, the levers are content quality, third-party mentions, and structured data — not a markdown manifest almost no inference-time system currently reads.
When to skip it
Not every site needs an llms.txt. Reasonable reasons to pass:
- You're a marketing site with no docs, no API, no reference material. There's nothing for an agent to navigate. A good sitemap.xml and clean HTML are enough.
- Your content changes hourly (news, e-commerce inventory). The maintenance burden outweighs the benefit; agents will re-crawl anyway.
- You already ship excellent OpenAPI + sitemap + schema. llms.txt is redundant coverage; deprioritize it until adoption forces the issue.
If you're a docs-heavy product, a developer tool, a knowledge base, or a publisher with a stable content taxonomy, ship the file. If you're none of those, don't feel bad about waiting.
How to decide in ten minutes
Ask three questions:
- Do agents (IDE assistants, MCP tools, in-product AI) plausibly interact with my content? If yes, ship it.
- Can I automate the file from existing structured data (sitemap, docs frontmatter, OpenAPI spec)? If yes, the maintenance objection dies.
- Am I expecting llms.txt to drive citations in ChatGPT or Google AI Overviews? If yes, recalibrate — that's not what it does today.
The rational play in 2026 is to treat llms.txt as insurance with a half-day premium. Cheap enough to buy, not important enough to obsess over.
FAQ
Does Google or OpenAI read llms.txt?
Not in any capacity they've publicly confirmed. Both organizations respect robots.txt directives for their crawlers, but neither has documented llms.txt as an input to retrieval or ranking. Assume it is not read by the major consumer LLMs until proven otherwise.
Should I publish llms-full.txt as well as llms.txt?
Only if you can automate generation. /llms-full.txt is useful for agents that want a one-shot context dump, but a stale full-text file is a liability. Wire it into your docs build; otherwise ship only the index.
Will llms.txt help me rank in AI Overviews or get cited by Perplexity?
No direct evidence it does. Citations from major LLMs are driven by content quality, third-party mentions, structured data, and crawlability — not by a manifest file. Publish llms.txt for agent tooling, not for citation lift.
