Should You Block AI Crawlers? GPTBot, ClaudeBot and Google-Extended Explained

Section 01

What AI crawlers are and why they visit

AI crawlers are automated programs that AI companies send to read web pages. They visit to collect training data, to build a search index for AI answers, or to fetch a page live because a user asked about it.

You control them the same way you control search engine crawlers: with robots.txt, a plain text file at yoursite.com/robots.txt that lists which bots may visit which parts of your site. Each bot identifies itself with a name called a user agent, such as GPTBot, and your file can allow or disallow each name separately.

Keep two limits in mind before you write any rules. Google's robots.txt guide (updated December 10, 2025) says the file "is not a mechanism for keeping a web page out of Google." It manages crawling, not indexing. And compliance is voluntary: Googlebot and other reputable crawlers obey robots.txt, but "other crawlers might not."

So robots.txt is a request. The major AI companies say they honor it, and that's what this guide relies on. Scrapers that ignore the rules need a firewall, which we cover below.

Section 02

The three kinds of AI bots

AI bots fall into three groups: training bots, search bots and user-fetch bots. Each group does a different job, and blocking each one costs you something different.

Training bots collect content that may be used to train AI models. OpenAI's crawler documentation says GPTBot "is used to make our generative AI foundation models more useful and safe," and that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Anthropic's ClaudeBot plays the same role: when a site restricts it, Anthropic says "it signals that the site's future materials should be excluded" from training. Google-Extended and Common Crawl's CCBot belong here too.

Search bots build the indexes AI tools use to find and cite websites. OpenAI's OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." Anthropic says "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization." Perplexity's bot documentation says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models."

User-fetch bots visit a page because a person asked an AI tool about it, for example by pasting a link. OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion," and "Because these actions are initiated by a user, robots.txt rules may not apply." For Claude-User, Anthropic says blocking it "prevents our system from retrieving your content in response to a user query." Perplexity says its Perplexity-User fetcher "generally ignores robots.txt rules."

OpenAI makes the key point plainly: "Each setting is independent of the others." Blocking GPTBot doesn't block OAI-SearchBot, and the reverse is also true. That's what makes a "block training, allow search" setup possible.

Section 03

Google-Extended: what it does and doesn't control

Google-Extended controls whether Google can use your content to train Gemini models and to ground Gemini's answers. It doesn't control Google Search. Google's common crawlers page, updated October 15, 2026, says it directly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Three details are easy to miss:

• It covers training and grounding. Grounding, in Google's words, means "providing content from the Google Search index to the model at prompt time to improve factuality and relevancy" in Gemini Apps and Vertex AI.
• It isn't a separate crawler. "Google-Extended doesn't have a separate HTTP request user agent string." Googlebot does the crawling, and the Google-Extended token in robots.txt only tells Google how it may use what Googlebot collects. You won't see Google-Extended in your server logs.
• It isn't the control for AI Overviews. Google's crawler page doesn't mention AI Overviews. Google's separate AI features guide explains that "AI is built into Search," so Googlebot's robots.txt rules are the control for Search, AI Overviews included. To limit what appears from your pages, Google points to nosnippet, data-nosnippet, max-snippet and noindex.

For most businesses, the takeaway is simple. Blocking Google-Extended is a choice about Gemini training. It won't hurt your Google rankings, and it won't remove you from AI Overviews. Never block Googlebot itself unless you want to leave Google Search.

Section 04

Should you block them? A decision by business type

For most service businesses, allow AI search bots and decide on training bots based on how you feel about content reuse. The answer changes when your content is the product you sell.

Local service businesses (roofers, clinics, attorneys, HVAC firms): allow search and user-fetch bots. Your website is a sales tool. When someone asks ChatGPT for "a good emergency plumber in Sacramento," you want the AI tool to be able to read your pages. Training bots are a personal call. Many owners allow them, since their service pages hold little that competitors could profit from.

B2B firms and agencies: usually allow search bots, for the same reason. Buyers can now ask AI assistants for vendor shortlists, and a blocked search bot keeps you off them. If you publish original research or detailed guides you'd rather not see absorbed into model training, block training bots and keep search bots open.

Publishers and sites whose content is the product (news sites, paid courses, recipe sites, research firms): the case for blocking training bots is stronger, because AI answers can replace the visit that earns your revenue. Some publishers block search bots too. That's a bigger trade: less AI visibility in exchange for keeping content out of AI answers.

Ecommerce stores: allow search bots so product pages can appear in AI shopping answers. Product descriptions rarely need protecting from training.

A few points apply whatever you choose:

• No setting guarantees AI citations. Allowing search bots makes citation possible; content quality, clear answers and trust signals decide whether it happens.
• Blocking training bots only affects future training. Content already collected isn't removed.
• Revisit the decision once a year. Bots, products and your own goals change.

Section 05

Copy-paste robots.txt examples

Pick the example that fits your decision, add it to your robots.txt file, and keep any existing rules for Googlebot and Bingbot in place.

Example A: block training, allow AI search. This is our usual starting point for service businesses that want AI visibility but would rather not supply training data.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Example B: block all listed AI bots. For sites that want no AI use of their content. Expect to lose visibility in ChatGPT, Claude and Perplexity search.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Example C: allow all. If your robots.txt has no rules naming AI bots and no blanket "User-agent: *" disallow, they're already allowed. You only need explicit rules if a plugin, host or earlier change added blocks.

Changes take time to apply. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results, and Perplexity says it "may take up to 24 hours." Anthropic says its bots honor robots.txt and that it supports "the non-standard Crawl-delay extension to robots.txt" if you want to slow its crawling rather than block it.

Section 06

Check your CDN and host settings

Your robots.txt may not be the only thing deciding which bots reach your site. CDNs, firewalls and hosting platforms can block AI crawlers before they ever read your file.

Cloudflare is the biggest example. Its July 1, 2025 announcement said that "every new domain will now be asked if they want to allow AI crawlers" when it signs up. It added that since Cloudflare introduced one-click blocking in September 2024, "More than one million customers have since chosen this option." Cloudflare manages and protects traffic for "20% of the web," so these settings shape a large share of sites.

How to check yours:

1. Cloudflare: look in your dashboard's bot and AI crawler settings for any rule blocking AI bots, and check whether a managed robots.txt has been added for you.
2. Other CDNs and firewalls: search your provider's settings for "bot management," "AI bots" or "verified bots."
3. WordPress and other platforms: some security plugins and hosts add AI blocking rules. Review your plugin settings and visit yoursite.com/robots.txt to see what's actually served.

Robots.txt and firewalls do different jobs. Robots.txt is a request that well-behaved bots follow, and it's easy to change. A firewall rule enforces the block, which stops bots that ignore robots.txt, but it can also block the AI search bots you wanted. If you use both, make sure they agree.

Want to be found in ChatGPT without giving away more than you choose? Our AI search team sets up crawler access that matches your goals. Talk to our AI search team

Section 07

Why this decision grows more important each year

The balance between what AI companies take and what they send back is moving, and that's why this setting deserves a yearly review.

In his "Content Independence Day" post on July 1, 2025, Cloudflare CEO Matthew Prince compared how hard it is for a site to earn traffic from each company. Over 10 years, he wrote, "it's gotten almost 10 times more difficult" to get the same traffic from Google. With OpenAI, "it's 750 times more difficult to get traffic than it was with the Google of old." With Anthropic, "it's 30,000 times more difficult."

In other words, these companies crawl far more than they refer, which helps explain the wave of blocking. They also point to the reason search bots are different. A search bot that lets ChatGPT cite your page is one of the few ways AI tools send visitors back. Blocking it removes that channel along with the crawling.

Section 10

AI crawler FAQs

What does GPTBot do?

GPTBot is OpenAI's training crawler. It collects public web content that may be used to train OpenAI's AI models. Blocking it in robots.txt tells OpenAI not to use your content for training. It's separate from OAI-SearchBot, which powers ChatGPT search.

Does blocking GPTBot remove my site from ChatGPT search?

No. ChatGPT search uses a different crawler, OAI-SearchBot, and OpenAI says each setting is independent of the others. To appear in ChatGPT search, allow OAI-SearchBot. To stay out of training, block GPTBot. Search changes take about 24 hours to apply.

Does Google-Extended affect my Google rankings?

No. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It controls whether your content is used to train and ground Gemini models. To limit what appears in AI Overviews, Google points to snippet controls such as nosnippet.

Is robots.txt still used, and do AI bots obey it?

Yes. Robots.txt is still the standard way to tell crawlers what they may access. OpenAI, Anthropic, Google and Perplexity say their crawlers follow it, though user-triggered fetchers such as ChatGPT-User may not. Robots.txt is voluntary, so bots that ignore it need a firewall block.

Is it legal for AI crawlers to scrape my site?

That's a legal question we can't answer, and the law is still developing. What we can say is what robots.txt does: it's a voluntary request that reputable crawlers honor, not a legal barrier or an enforcement tool. For legal advice about your content, speak to a qualified attorney.

For more on AI search visibility, start with our guide to ranking in ChatGPT, then see earning AI Overview citations, GEO vs SEO and what generative engine optimization is. When you want help, our ChatGPT optimization service sets up crawler access and content for AI search.

Section 11

Sources

1. OpenAI, "Overview of OpenAI Crawlers" (no date shown; read October 21, 2026): https://developers.openai.com/api/docs/bots
2. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", April 7, 2026: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
3. Perplexity, "Perplexity Crawlers" (read October 21, 2026): https://docs.perplexity.ai/guides/bots
4. Google Search Central, "Google's common crawlers" (Google-Extended), updated October 15, 2026: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
5. Google Search Central, "AI features and your website," updated December 10, 2025: https://developers.google.com/search/docs/appearance/ai-features
6. Google Search Central, "Introduction to robots.txt," updated December 10, 2025: https://developers.google.com/search/docs/crawling-indexing/robots/intro
7. Cloudflare press release, July 1, 2025: https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/
8. Cloudflare blog, Matthew Prince, "Content Independence Day," July 1, 2025: https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/