TL;DR
- llms.txt is a Markdown file you place at the root of your domain —
/llms.txt— that points AI assistants to the parts of your site worth reading. - It was proposed in 2024 by Jeremy Howard of Answer.AI and the spec lives at llmstxt.org.
- The format is dead simple: an H1 site name, a blockquote summary, then sectioned lists of annotated links to your important pages.
- It is not a replacement for robots.txt or sitemap.xml. It is a curated reading list for LLMs — the way a thoughtful intern would brief a new hire.
- Adoption is still early but accelerating. Treat it as low-cost, high-leverage hygiene if you publish docs, guides, or product reference pages.
Where llms.txt fits
The web was indexed for crawlers, not for chatbots. When ChatGPT or Claude needs to ground an answer in your site, the model usually has seconds — not hours — to find the right page. Sitemaps list everything; robots.txt gates access; neither tells an LLM which page is the canonical answer to a specific question.
llms.txt fills that gap with a hand-curated shortlist. Per the llmstxt.org spec, the file is "a list of links to clean, useful Markdown content, with brief explanations, that LLMs can use to find relevant information quickly." Think of it as the table of contents for your site, written for a reader with no patience for marketing chrome.
The four required parts
A valid llms.txt has exactly four building blocks, in this order:
# Site Name— an H1 with your brand or site name.- A blockquote — one to three sentences describing what the site covers and who it serves.
- Section headings (
## Docs,## Guides,## API Reference) — group related pages. - Annotated links — under each section, a bulleted list of
[Page title](url): one-line description.
A minimal example for a SaaS:
# Acme Analytics
> Acme is a product analytics platform for B2B SaaS teams. We help product
> managers understand how customers use their software via session replay,
> funnels, and cohort analysis.
## Getting started
- [Installation](/docs/install): Add the Acme snippet to your site in 5 minutes.
- [First dashboard](/docs/first-dashboard): Build your first funnel and retention chart.
## API
- [REST reference](/docs/api): Full REST API surface for ingest and query.
- [Webhooks](/docs/webhooks): Subscribe to events for downstream automation.
The spec also defines an optional ## Optional section for "skim if time allows" content — useful for changelogs, blog archives, or deprecated pages.
How it relates to robots.txt and sitemap.xml
These three files do different jobs and should all coexist on a serious site in 2026:
| File | Audience | Job | Tone |
|---|---|---|---|
robots.txt | Crawlers | Gate access by user-agent | Permission |
sitemap.xml | Search engines | Enumerate every URL for indexing | Machine list |
llms.txt | LLMs / AI agents | Curated reading list for grounding | Editorial |
You still need robots.txt to permit (or deny) GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. You still need a sitemap so traditional search engines can crawl your full surface. llms.txt is the third leg — a deliberately small list that says "if you only read five pages on this site, read these."
When to ship one
Ship an llms.txt if any of the following describe your site:
- You publish technical docs, API references, or developer guides.
- You publish in-depth tutorials, knowledge bases, or product comparisons.
- Your top-of-funnel content has been duplicated, syndicated, or quoted out of context, and you'd like AI assistants to land on the canonical version.
- You have more than ~20 substantive pages — enough that an LLM picking blind would frequently land on a thin one.
Skip it if you only have a homepage and a contact form. It will not save a thin site.
Common mistakes
- Linking too much. llms.txt is a shortlist, not a sitemap. If every page on your site is "important," prune the list. Twenty curated links beats two hundred.
- Marketing voice in the descriptions. The blockquote summary and the link annotations should read like documentation, not a landing page. LLMs are extracting facts, not vibes.
- Pointing to gated content. If a link in llms.txt 404s, requires login, or is blocked by robots.txt, you've wasted a slot. Verify each URL is publicly reachable.
- Forgetting to maintain it. When you reorganize docs, update llms.txt the same week. Stale entries train the model on dead URLs.
Quick FAQ
Will llms.txt improve my AI search visibility on its own?
Not on its own. llms.txt is one signal among many — schema, brand mentions, off-page authority, robots.txt access for AI crawlers. Think of it as making your site easy to read once an LLM has decided to look. It does not summon traffic.
Is llms.txt an official W3C or IETF standard?
No. It is a community proposal hosted at llmstxt.org. OpenAI, Anthropic, and Google have not formally endorsed it, but several developer-tool vendors (Cloudflare, Vercel, Mintlify, Anthropic's own docs) ship one — which is the strongest soft signal of adoption you currently have.
Should I also publish an /llms-full.txt?
The spec defines an optional companion file that includes the full Markdown of your most important pages inline, for LLMs that prefer to fetch one file instead of crawling many. It's useful for small documentation sites where the full corpus fits in a single token budget. For larger sites, stick to plain llms.txt with links.
Sources
- llmstxt.org — the original specification
- Bluehost: What is llms.txt?
- Click Digital: llms.txt guide for the new AI standard
- Codersera: llms.txt honest guide to spec and adoption
Want to know if your site is ready for AI search? Run a free scan at CiteFlow — we check robots.txt, llms.txt, AI crawler access, and 30+ other GEO signals.
