Should you block GPTBot in robots.txt?
Updated
Blocking GPTBot opts you out of OpenAI model training but doesn't control ChatGPT search. What each AI crawler does and how to choose which to allow.
Many sites added User-agent: GPTBot / Disallow: / to robots.txt in 2023 and 2024. Whether that's right for you depends on what you want: keeping your content out of AI training, or being found and cited by AI assistants. Those are now controlled by different crawlers.
What the main AI crawlers do
| User agent | Operator | Purpose (per the operator's documentation) |
|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train models |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search |
| ChatGPT-User | OpenAI | Fetches pages on behalf of a user in ChatGPT |
| ClaudeBot | Anthropic | Collects content for model training |
| Claude-User, Claude-SearchBot | Anthropic | User-initiated fetches and search |
| PerplexityBot | Perplexity | Indexes pages for Perplexity's answers |
| Google-Extended | A robots.txt token (not a separate crawler) that controls use of content for Gemini models; it does not affect Google Search | |
| Applebot-Extended | Apple | Controls use of content for Apple's AI models |
| CCBot | Common Crawl | Builds an open web archive that many AI models are trained on |
Sources: OpenAI, Google, and each operator's crawler documentation. Names and purposes change, so check the current docs before relying on a rule.
Three common choices
1. Allow everything. Best if you want maximum visibility in AI answers and don't mind your public pages being used for training. Most marketing sites, docs and SaaS landing pages are here.
2. Allow search, block training. Allow OAI-SearchBot, ChatGPT-User, PerplexityBot and the user-initiated agents; disallow GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Allow: /
3. Block everything AI. Reasonable for paywalled or licensed content, but expect AI assistants to describe you using other people's pages, or not at all.
Things to know
- robots.txt is a request, not an enforcement mechanism. Reputable crawlers honour it; to truly block traffic you need firewall rules.
- A
User-agent: *block applies to any crawler that doesn't have its own section. A blanketDisallow: /under*blocks AI search crawlers too. - Blocking a training crawler today doesn't remove content that was already collected.
BotVisible's free scan reads your robots.txt and shows exactly which AI crawlers are allowed or blocked on your page.