Short answer: robots.txt is the one that decides whether AI systems can read your site, because it controls which crawlers may fetch your pages. llms.txt only describes which pages are worth reading. If robots.txt blocks an answer-engine crawler, no llms.txt will get you cited by that engine. Start with robots.txt; add llms.txt as a guide.
What does each file do?
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Allow or block crawlers | Point models to key pages |
| Status | Standard (RFC 9309, 2022) | Community proposal (2024) |
| Read by | Every well-behaved crawler | Some AI tools and agents |
| Used by Google Search | Yes | Google says no |
| Can block access | Yes | No |
| Format | User-agent and Disallow rules |
Markdown with links |
The two files are not competitors. robots.txt is a door; llms.txt is a map you leave on the table once people are inside.
Why does robots.txt matter more for AI visibility?
Because access comes first. Each AI answer engine uses its own crawler, and each crawler checks robots.txt before fetching:
- ChatGPT search uses OAI-SearchBot. OpenAI says sites that opt out of it will not appear in ChatGPT search answers.
- Perplexity uses PerplexityBot for its search results.
- Claude's search uses Claude-SearchBot.
- Google AI Overviews are part of Search and rely on Googlebot.
If any of these is disallowed, that engine cannot fetch your pages to quote them. A beautifully written llms.txt changes nothing, because the crawler never gets past the door.
Which bots should you block and which should you allow?
This is the decision that matters, and most robots.txt files get it wrong by treating every AI bot the same. There are two kinds:
- Answer-engine crawlers fetch pages to show and link them in answers. Blocking them removes you from that engine's results.
- Training-only crawlers collect text to train models. Blocking them is a licensing choice. It does not remove you from answers.
The names are easy to confuse. GPTBot is OpenAI's training crawler; OAI-SearchBot is the one that powers ChatGPT search. ClaudeBot is Anthropic's training crawler; Claude-SearchBot powers search. Google-Extended and Applebot-Extended are not crawlers at all but robots.txt tokens that control training use, and both Google and Apple say they do not affect search.
A common, reasonable policy for a business that wants to be cited but not used for training looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
This keeps answer-engine crawlers, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, allowed through the wildcard group, while opting out of the named training uses. Whether to opt out of training is your call; the point is to make it deliberately. Our AI crawler list shows every agent we track and which group it belongs to.
What mistakes do we see most often?
- Blocking with a wildcard.
User-agent: *withDisallow: /blocks every crawler without a group of its own, including answer engines. - Copying an "AI blocklist" from a forum. These lists often include OAI-SearchBot and PerplexityBot alongside training bots, and quietly remove the site from AI answers.
- Assuming robots.txt removes pages from indexes. It controls crawling. A blocked URL can still appear if other pages link to it. Use
noindexon a crawlable page to keep it out. - An llms.txt that is really an HTML page. Many sites return their 404 page with a 200 status at
/llms.txt. Tools that check only the status code think the file exists.
Where does llms.txt still help?
Once access is right, llms.txt can make a model's job easier on large or technical sites: it points to the pages that answer real questions instead of leaving the model to guess from navigation. It is most useful for documentation, APIs and SaaS products. We explain how to write one in What is llms.txt.
What is the classic SEO view?
robots.txt has been part of SEO for decades, and the same rules apply: do not block pages you want indexed, do not use it to hide sensitive content, and keep it short enough to audit. The new part is that there are now many more user-agents to think about, and the difference between a training bot and a search bot changes what a block costs you.
How do you check your own site?
Our free robots.txt and llms.txt checker reads your robots.txt, tells you which answer-engine and training crawlers you allow, and checks whether your llms.txt is a real file. It takes one request and no sign-up.
Frequently asked questions
If I block GPTBot, will ChatGPT stop citing me?
No. GPTBot is used for training. ChatGPT search uses OAI-SearchBot. Blocking GPTBot opts you out of training, not out of ChatGPT search.
Do I need llms.txt if my robots.txt is correct?
Not strictly. A correct robots.txt is the requirement; llms.txt is an optional guide that helps some AI tools on larger or technical sites.
Does Google read llms.txt?
Google has said Google Search does not use llms.txt. It does read robots.txt.
Can robots.txt protect private content?
No. robots.txt is public and only asks crawlers not to fetch. Anything private needs authentication.