Short answer: llms.txt is a plain Markdown file at the root of your site (/llms.txt) that lists your most useful pages with one-line descriptions, so a language model can find them without parsing your whole navigation. It is a proposal, not a standard. It costs an hour to write, it helps some AI tools, and Google has said it does not use it.
What exactly is llms.txt?
llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI and is documented at llmstxt.org. The idea is simple: web pages are built for people and browsers, full of menus, scripts and layout. A language model reading your site has a limited context window, so a short, curated map of "this is what we are and these are the pages worth reading" is easier to use than raw HTML.
The file is ordinary Markdown with a fixed shape:
- an H1 with the name of the site or project,
- a short blockquote summarising what it is,
- optional paragraphs of context,
- H2 sections containing lists of links, each with a short note.
Some sites also publish llms-full.txt, a single file containing the full text of their key pages. That variant is common in developer documentation.
What does llms.txt not do?
This is where most confusion starts, so it is worth being precise.
- It does not control access. llms.txt cannot allow or block any crawler. Access is decided by robots.txt, which is a separate file with a different job. We compare the two in llms.txt vs robots.txt.
- It is not a ranking signal for Google. Google representatives have said publicly that Google Search does not use llms.txt. Treat it as positioning for AI tools, not as a Google SEO tactic.
- It is not guaranteed to be read. No major AI provider has committed to reading llms.txt for every site. Some developer tools and agents do read it when they are pointed at a site.
So llms.txt is a helpful hint, not a switch.
Who actually benefits from an llms.txt file?
Sites that are read by developers, agents and documentation tools benefit most: SaaS products, APIs, open-source projects and knowledge bases. When someone asks an AI coding assistant to "read the docs for X", a clean llms.txt saves it from guessing which of your 400 pages are the real reference.
For a local business or a small blog, the benefit is smaller. It rarely hurts, but it will not fix a site that answer engines cannot read in the first place. If your content is rendered only by JavaScript, or blocked in robots.txt, llms.txt will not rescue it.
How do you write a good llms.txt?
Keep it short and honest. A useful file for a SaaS site might look like this:
# ExampleCo
> ExampleCo is invoicing software for freelancers in the EU.
## Product
- [Features](https://example.com/features): what the product does
- [Pricing](https://example.com/pricing): plans and what each includes
## Help
- [Getting started](https://example.com/docs/start): set up in ten minutes
- [VAT rules](https://example.com/docs/vat): how EU VAT is calculated
Practical rules:
- Link only pages that answer real questions. A list of every URL is a sitemap, not a guide.
- Write each note as a plain statement of what the page contains.
- Keep the file in sync with your site. A stale llms.txt that links to 404 pages is worse than none.
- Serve it as plain text or Markdown with a 200 status. A common mistake is a site that returns its HTML 404 page with a 200 status at
/llms.txt, which tools then read as "the file exists".
What is the classic SEO counterpart?
llms.txt overlaps with things SEO already asks for: a clear site structure, descriptive titles, and an XML sitemap. The difference is the audience. A sitemap lists everything for search engine crawlers; llms.txt picks the few pages a model should read first. If you have neither, start with the sitemap and good internal linking, because those help Google and AI systems alike.
Should your website have one?
Add an llms.txt if:
- developers, agents or AI assistants are a meaningful part of your audience,
- you have documentation or reference pages that are hard to find through navigation,
- you can keep it up to date.
Skip it for now if your site has more basic problems, such as pages blocked from crawlers or content that only appears after JavaScript runs. Fix those first.
You can check whether your site publishes an llms.txt, and whether it is a real file rather than an HTML fallback page, with our free robots.txt and llms.txt checker. It also shows which of the AI crawlers in our crawler list your robots.txt allows.
Frequently asked questions
Is llms.txt an official standard?
No. It is a community proposal published at llmstxt.org. Unlike robots.txt, which was standardised as RFC 9309 in 2022, llms.txt has no formal standards body behind it.
Does llms.txt help with Google AI Overviews?
Google has said it does not use llms.txt. AI Overviews are part of Google Search, so the usual Search basics apply: the page must be indexed and eligible to show a snippet.
Where do I put the file?
At the root of your domain: https://yourdomain.com/llms.txt. It should return a 200 status with Markdown or plain text, not an HTML page.
Can llms.txt block AI crawlers?
No. Blocking and allowing crawlers is done in robots.txt. llms.txt only describes content.