Should I Block AI Crawlers from My Website? (Training vs Search)

Block AI training crawlers to protect content from future models, but leave search crawlers allowed if you want citations in ChatGPT, Claude, and Perplexity answers. Training bots cost you nothing immediately. Search bots decide whether your business appears when someone asks a question you should win.

What blocking actually removes

AI training crawlers like GPTBot and CCBot scrape your site to train general-purpose models, giving you no citation, no traffic, and no way to track value (quattr.com). Blocking training bots is a defensible business choice, particularly if content represents a licensed commercial asset, with the only consequence being a slow degradation of the model's parametric knowledge of your category over successive training runs (playwire.com).

Blocking OAI-SearchBot removes you from ChatGPT's live search results completely, even though blocking GPTBot has no measurable effect on your regular Google rankings. Blocking GPTBot costs nothing in ChatGPT citations, but blocking OAI-SearchBot removes you entirely, and confusing the two is the expensive mistake (zenweb.my).

Google AI Overviews now appear in 60.32% of U.S. queries, doubling their presence since August 2024 and clearing the 60% mark for the first time (xponent21.com). AI Overviews are causing websites to lose between 17% and 79% of organic traffic depending on industry, reducing the click-through rate of the first organic result from 7.3% to 2.6% (falia.co). This shift makes AI citations one of the few remaining ways to capture attention above the fold.

Which user agents control what?

Providers now separate training and search crawlers, so you can block training agents (GPTBot, Google-Extended, ClaudeBot) while allowing search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) that make you eligible for AI-answer citations (dataimpulse.com).

OpenAI runs three bots. GPTBot crawls content for model training and real-time search retrieval, OAI-SearchBot crawls specifically for ChatGPT's search feature, and blocking OAI-SearchBot while allowing GPTBot means your content trains the model but doesn't appear in ChatGPT's search results (xseek.io).

Anthropic publishes its crawler IPs and operates ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user-triggered fetches. PerplexityBot is used for real-time retrieval, and when users ask Perplexity questions it fetches and reads web pages to generate cited answers, so blocking this crawler directly prevents your content from appearing in Perplexity's answers.

Google-Extended (introduced September 2023) controls AI training inclusion separately from Googlebot, and blocking it does not affect search ranking. Google states that blocking Google-Extended does not impact a site's inclusion in Google Search, and AI Overviews draw on the same Googlebot index as ordinary results.

How should you decide by business model?

If you sell services or advice and clients find you by searching for problems you solve, allow the search crawlers. Blocking AI crawlers in robots.txt hurts brand visibility directly and completely for the engines you block, and if PerplexityBot, OAI-SearchBot, or Googlebot cannot read your pages, the AI engines they power will source answers from competitors who do allow crawling, with most sites blocking AI crawlers by accident through a wildcard rule.

If you publish original research, long-form guides, or proprietary data your competitors will copy the moment a model regurgitates it, block the training bots. Sites that blocked training bots saw average traffic change stay within 1%, matching normal fluctuation, and you can block Google Extended apart from Googlebot so search visibility stays intact.

If you run e-commerce or local services where AI assistants recommend businesses by name, allowing both training and search crawlers keeps you in the conversation. The free AI-visibility check (/check) shows whether Claude, ChatGPT, Gemini, and Perplexity name your business when someone asks who to hire, with each assistant reported separately.

The hidden blocker most people miss

CDN and WAF bot-management rules silently block AI crawlers by user-agent, excluding pages from OpenAI's index even when robots.txt explicitly allows OAI-SearchBot and GPTBot. Web Application Firewall rules override robots.txt, and Cloudflare changed default behavior on September 15, 2026, so new domains have training and agent crawlers blocked by default on ad-displaying pages.

Check your CDN settings. A correct robots.txt is necessary but not enough. If your host or CDN blocks broad categories like "AI bots" by default, you disappear from answers regardless of what your robots.txt says.

What to put in robots.txt

For maximum AI visibility while protecting training rights, allow the search and retrieval bots that drive citations and block the training-only bots. Add these lines to your robots.txt file:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

This configuration keeps you eligible for citations in ChatGPT, Claude, and Perplexity while opting out of contributing to future training runs. 34.2% of top-1,000 domains name at least one AI crawler, and 21.0% fully block at least one, which means most sites still run wide open or block everything without understanding the tradeoff.

Marketing agencies and consultancies can track which assistants name them and their competitors using ActiScore's pricing (/pricing) plans, which include side-by-side audits showing how Claude, ChatGPT, Gemini, and Perplexity answer category questions. For a one-time check before making any robots.txt changes, the guide (/guide) walks through the full decision framework step by step, including how to verify your CDN isn't blocking crawlers you meant to allow.