AI crawlers
27 AI crawlers, named.
These are the user-agents our audit and free checker look for. Each is either an answer engine that can quote and link to you, or a training-only crawler that collects text for models and sends nothing back. That difference is what matters when you decide what to block.
Answer engines: 16 crawlers that can cite you
Blocking any of these removes you from a surface where a model could name you as the source.
| User-agent | Operator | What it does |
|---|---|---|
oai-searchbot | OpenAI | Powers ChatGPT's search index |
chatgpt-user | OpenAI | Fetches a page when a user asks ChatGPT about it |
claude-user | Anthropic | Fetches a page when a user asks Claude about it |
claude-searchbot | Anthropic | Powers Claude's search results |
perplexitybot | Perplexity | Builds the index Perplexity answers from |
perplexity-user | Perplexity | Fetches a page a user asked about |
google-cloudvertexbot | Vertex AI grounding | |
bingbot | Microsoft | Copilot answers are built on the Bing index |
msnbot-media | Microsoft | Media crawling for Bing/Copilot |
meta-externalagent | Meta | Crawls for Meta AI answers |
meta-externalfetcher | Meta | Fetches a page on user request |
amazonbot | Amazon | Feeds Alexa and Rufus answers |
youbot | You.com | Indexes pages You.com can cite |
cohere-ai | Cohere | Retrieval for Cohere-powered assistants |
diffbot | Diffbot | Knowledge-graph extraction used by several assistants |
timpibot | Timpi | Independent index used by AI search products |
Descriptions are kept in English, the language their operators publish them in.
Training-only: 11 crawlers that give nothing back
These collect content for model training. Blocking them is a licensing decision: it does not reduce your visibility.
| User-agent | Operator | What it does |
|---|---|---|
gptbot | OpenAI | Collects content that may be used to train OpenAI models; ChatGPT search uses OAI-SearchBot instead |
claudebot | Anthropic | Collects content that may be used to train Anthropic models; Claude search uses Claude-SearchBot |
google-extended | Not a crawler: a robots.txt token for Gemini training and grounding; Google says it does not affect Google Search | |
applebot-extended | Apple | Not a crawler: a robots.txt token for training Apple models; Apple says it does not affect Apple search |
claude-web | Anthropic | Legacy Anthropic agent no longer documented; current user fetches use Claude-User |
ccbot | Common Crawl | Public dataset many models train on |
bytespider | ByteDance | Training data collection |
omgili | Webz.io | Data resale and training corpora |
anthropic-ai | Anthropic | Legacy training-data agent (no live citation) |
facebookbot | Meta | Training data collection |
petalbot | Huawei | Petal search and training corpora |
Descriptions are kept in English, the language their operators publish them in.
Controlling one crawler
robots.txt works per user-agent. This group blocks one crawler and leaves everything else as it is:
User-agent: CCBot
Disallow: /Blocking an answer-engine crawler removes you from that engine's answers. Blocking a training-only crawler has no effect on whether you are cited.
Check your own robots.txt
The free checker reads your robots.txt and llms.txt and tells you which of these 27 crawlers you allow.
This page is generated from the same catalog the product uses, so it cannot drift from what the audit checks.