Should you block GPTBot in robots.txt?

Updated

Blocking GPTBot opts you out of OpenAI model training but doesn't control ChatGPT search. What each AI crawler does and how to choose which to allow.

Many sites added User-agent: GPTBot / Disallow: / to robots.txt in 2023 and 2024. Whether that's right for you depends on what you want: keeping your content out of AI training, or being found and cited by AI assistants. Those are now controlled by different crawlers.

What the main AI crawlers do

User agentOperatorPurpose (per the operator's documentation)
GPTBotOpenAICollects content that may be used to train models
OAI-SearchBotOpenAIIndexes pages for ChatGPT search
ChatGPT-UserOpenAIFetches pages on behalf of a user in ChatGPT
ClaudeBotAnthropicCollects content for model training
Claude-User, Claude-SearchBotAnthropicUser-initiated fetches and search
PerplexityBotPerplexityIndexes pages for Perplexity's answers
Google-ExtendedGoogleA robots.txt token (not a separate crawler) that controls use of content for Gemini models; it does not affect Google Search
Applebot-ExtendedAppleControls use of content for Apple's AI models
CCBotCommon CrawlBuilds an open web archive that many AI models are trained on

Sources: OpenAI, Google, and each operator's crawler documentation. Names and purposes change, so check the current docs before relying on a rule.

Three common choices

1. Allow everything. Best if you want maximum visibility in AI answers and don't mind your public pages being used for training. Most marketing sites, docs and SaaS landing pages are here.

2. Allow search, block training. Allow OAI-SearchBot, ChatGPT-User, PerplexityBot and the user-initiated agents; disallow GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

3. Block everything AI. Reasonable for paywalled or licensed content, but expect AI assistants to describe you using other people's pages, or not at all.

Things to know

BotVisible's free scan reads your robots.txt and shows exactly which AI crawlers are allowed or blocked on your page.

Check your site in 10 seconds

BotVisible fetches your page like an AI crawler and scores it from 0 to 100. Free, no signup.

Run a free scan

More guides