CiteFlow
Sign Up
← / Blog

Technical

llms-txttechnical-seoai-searchgeo

How to Write Your llms.txt File: Step-by-Step Guide

A practical step-by-step for shipping a real llms.txt — file format, section structure, what to link, where to host, and the mistakes that quietly waste the file.

5 min read

TL;DR

  • llms.txt is a Markdown file at /llms.txt. Four building blocks: H1 site name, blockquote summary, optional context paragraph, then ## section headings with bulleted annotated links.
  • Aim for 10–30 curated links — your canonical answer pages, not your sitemap.
  • Serve it at the root, return text/plain or text/markdown, allow CDN caching for an hour, and version it in Git.
  • Common mistakes: too many links, marketing prose in the descriptions, linking to gated/404/JS-only pages, and forgetting to refresh it after a docs reorg.
  • It is not robots.txt. It does not gate access. It is the editorial reading list — written for an LLM that has ten seconds to find your best page.

File format, briefly

Per the llmstxt.org spec, the format is plain Markdown with a specific structure that any standard Markdown parser can read without extensions. Required parts:

  1. # Site Name — an H1.
  2. A blockquote > summary — one to three sentences.
  3. Optional context paragraphs.
  4. ## Section headings, each followed by bulleted lists of [Title](URL): one-line description.

That is the entire spec. The discipline is in what you choose to put inside it.

Section structure

Use H2 sections to group related pages by job. The most useful sections for a typical SaaS or content site:

  • ## Getting started — onboarding, install, first-run.
  • ## Docs — API reference, configuration, integration guides.
  • ## Guides — long-form tutorials and how-tos.
  • ## Reference — glossaries, comparison pages, cheat sheets.
  • ## Company — about, careers, security, contact (only if any of these are AI-relevant).
  • ## Optional — skim-if-time content like changelogs, blog archives, deprecated pages.

The ## Optional section has a specific role in the spec: pages an LLM should read only if budget allows. Put your blog archive, your changelog, and your "interesting but tangential" content here. The model is welcome to skip it.

Writing for different LLMs

Style notes that compound across Claude, GPT, Perplexity, and Gemini:

  • Short, declarative descriptions. The annotation after the colon should read like documentation, not marketing. "REST API for ingest and query" beats "powerful flexible API for all your data needs."
  • No marketing voice in the blockquote summary. State what the site is and who it serves. Two sentences max.
  • Stable URLs only. If a page redirects, link to the destination. If a URL changes monthly, do not link it.
  • Link to Markdown when you have it. A .md version of the page parses cleaner than HTML for most models. Spec convention: append .md to the URL if your site supports it, or publish a parallel docs tree.

A complete example

# Acme Analytics

> Acme is a product analytics platform for B2B SaaS teams. We help product
> managers understand how customers use their software via session replay,
> funnels, and cohort analysis.

This llms.txt points to canonical answers for our top product, integration,
and architectural questions. Updated monthly.

## Getting started
- [Installation](/docs/install): Add the Acme snippet to your site in 5 minutes.
- [First dashboard](/docs/first-dashboard): Build your first funnel and retention chart.
- [Concepts](/docs/concepts): Events, properties, identity, sessions — the core model.

## Reference
- [REST API](/docs/api): Full REST surface for ingest and query.
- [Webhooks](/docs/webhooks): Subscribe to events for downstream automation.
- [SDKs](/docs/sdks): Browser, iOS, Android, server.

## Guides
- [Funnels 101](/guides/funnels): When to use funnels, how to instrument them.
- [Retention math](/guides/retention): N-day, rolling, cohort retention explained.

## Optional
- [Changelog](/changelog): Monthly product release notes.
- [Blog archive](/blog): Editorial posts, less authoritative than docs.

Keep yours boring. The signal is the curation, not the prose.

Where to host it and how to cache it

Serve the file from the exact path /llms.txt at the apex of your domain. A few practical notes:

  • Content-Type. text/plain; charset=utf-8 is fine. text/markdown is also accepted. Anything that forces a download (octet-stream) is wrong.
  • Encoding. UTF-8. Not UTF-16, not Windows-1252.
  • HTTP cache. A modest Cache-Control: public, max-age=3600 (1 hour) works for most sites. The file is small; CDN edge caching is fine.
  • Subdomains. Each property gets its own. docs.example.com/llms.txt is separate from example.com/llms.txt.
  • Sitemaps. Do not list /llms.txt in your sitemap. It is not for general search.

Validating and monitoring

There is no equivalent of Search Console for llms.txt. What you can do:

  • Parse it locally. Run it through any Markdown parser. If it does not render a clean nested list, an LLM will not parse it cleanly either.
  • Check that every link resolves to a 200, indexable page. A scheduled script that walks the file weekly catches link rot. Even one 404 in llms.txt is a wasted slot.
  • Log server access. Filter for known AI user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) hitting /llms.txt and the linked pages. Volume tells you whether your file is being consumed.
  • Diff it in Git. Commit changes. Reviews catch the "we deprecated /docs/old-api but forgot to remove it from llms.txt" cases.

Common mistakes

  • Too many links. llms.txt is not your sitemap. Twenty curated links beats two hundred. If everything is important, nothing is.
  • Marketing voice in descriptions. The annotations should sound like a senior engineer briefing a new hire, not a landing page.
  • Linking to gated content. Login walls, paywalled docs, JavaScript-only routes that AI tools cannot render — each one is a wasted slot.
  • Wrong filename. It is llms.txt. Not llm.txt, not LLMs.txt. The spec is literal.
  • Forgetting to maintain it. A six-month-old llms.txt with three dead links is worse than no file. If you reorg docs, update llms.txt the same week.

FAQ

Should I add an llms-full.txt too?

The spec defines an optional llms-full.txt that inlines the full Markdown body of your top pages, for LLMs that prefer one fetch over many. It helps for small docs sites where the corpus fits in a single context window. For larger sites, plain /llms.txt with links is the sustainable shape.

Does llms.txt help with traditional SEO?

Not directly. It is not a Google ranking signal. It is a hint for AI assistants — particularly the ones building agentic browsing experiences. The SEO win, if any, is indirect: a curated reading list keeps the LLM focused on your canonical pages, which keeps the attribution clean when those pages get quoted.

How often should I update it?

Quarterly review is the baseline. Update sooner if you have a significant docs reorg, a new flagship product, or a major version cut that obsoletes prior reference pages.

Sources

Need to know if your llms.txt is doing the job? CiteFlow audits llms.txt presence, structure, link health, and AI crawler access in one scan.

Related reading