TL;DR
- Checking robots.txt tells you nothing about whether GPTBot, ClaudeBot, or PerplexityBot can actually reach your pages — the block usually happens at the edge, before the request hits your server.
- Since July 1, 2025, Cloudflare blocks all known AI crawlers by default on every newly onboarded domain, which affects roughly 20% of the public web without site owners touching a setting.
- Run a direct
curltest with a spoofed crawler user-agent to see the real HTTP status code — 200 is clear, 403 or 429 means something at the edge is rejecting the bot. - Cloudflare retired its one-click "Block AI bots" toggle in favor of separate Search, Agent, and Training controls — know the difference before you flip anything.
- 403 rates for AI bot traffic have more than doubled year-over-year, so this isn't a one-time fix — it needs to be part of your regular technical audit.
A WAF (web application firewall) sits in front of your origin server and filters requests before your CMS, your app, or even your robots.txt ever gets a say. Bot management products — Cloudflare's, but also Akamai's, Fastly's, and the managed rulesets AWS and Vercel ship by default — classify incoming traffic and silently drop anything that looks automated. AI crawlers look exactly like what these systems were built to stop: no browser fingerprint, no cookies, high request volume, predictable user-agent strings. The crawler never sees your content. It sees a 403 and moves on, and nothing in your content strategy will fix that.
Why robots.txt audits give you false confidence
A robots.txt audit answers one question: did you tell the crawler it's allowed to fetch your pages. It says nothing about whether the request is actually permitted to reach your origin. You can have a perfectly clean robots.txt — no disallow rules, no crawl-delay, GPTBot and ClaudeBot explicitly welcomed — and still be invisible to every major AI assistant because your WAF is returning 403 on the TCP handshake before robots.txt is ever parsed.
This matters because most GEO audits stop at robots.txt. It's visible, it's text, it's easy to check in a browser. The actual block is invisible unless you test with the right tool, which is why so many sites unknowingly disappear from AI training and retrieval pipelines while believing they're fully open.
The July 2025 default-block change
Cloudflare's shift to block-by-default for new domains was a quiet but consequential change. Every domain onboarded after July 1, 2025 starts with all known AI crawlers blocked unless someone actively changes that setting. Because Cloudflare sits in front of a large share of the internet, this single default affected an estimated 20% of the public web's crawlability overnight. If your site was provisioned through Cloudflare, a partner host, or a platform that wraps Cloudflare (many site builders and CDNs do), there is a real chance you were opted into blocking AI crawlers without ever seeing a settings screen.
The practical implication: don't assume your site is open just because nobody on your team configured a block. The block may have been the default from day one.
How to actually test — curl, not robots.txt
The only reliable test is to send a request that mimics the crawler and read the response code:
curl -I -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com
A 200 OK means the request reached your origin cleanly. A 403 Forbidden or 429 Too Many Requests means something at the edge — WAF rule, rate limit, bot-management heuristic — intercepted it before your content was served. Run this for each crawler user-agent you care about: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Amazonbot. They are often treated differently by the same WAF, so a pass for one doesn't mean a pass for all.
This is also worth running periodically, not once. 403 rates for AI bot traffic have more than doubled year-over-year, climbing from 3.63% to 8.56% according to a September 2026 analysis of Cloudflare's network. Vendors tune their bot-detection models continuously, and a crawler that passed cleanly last quarter can start getting flagged as "suspicious automated traffic" with no changes on your end.
Fixing it: Cloudflare, Vercel, and AWS
In Cloudflare, go to Security → Bots → AI Crawl Control (or AI Bots, depending on your dashboard version). As of September 15, 2026, the old single "Block AI bots" toggle is gone. It's now split into three controls: Search, Agent, and Training. This distinction matters — setting Training to "Block" doesn't just stop AI model training crawlers, it also blocks Googlebot and Bingbot, because those crawlers are flagged under the same category in Cloudflare's classification. If you want to prevent your content from being used in training sets without losing search visibility, use the "Disallow AI Training" option instead of a blanket block — it signals a training refusal without rejecting the crawler's request outright.
On Vercel, check the Firewall tab under project settings for any Bot Protection or Attack Challenge Mode rules — these are often enabled by default on the Pro and Enterprise tiers and will silently challenge or block non-browser user-agents. On AWS, the culprit is usually a WAF Web ACL with a Bot Control managed rule group attached to your CloudFront distribution or ALB; check the rule group's "Common Bot Category" settings and add explicit allow rules for the AI crawler user-agents and their published IP ranges before the Bot Control rule evaluates the request.
Whatever platform you're on, the fix is the same pattern: find the bot-management layer, identify which crawlers it's classifying as "bad bot" traffic, and add explicit allow-list rules by user-agent and verified IP range rather than disabling bot protection entirely.
What to allow-list
Allow-listing everything isn't the goal — allow-listing the crawlers whose citations you actually want. At minimum: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and Amazonbot if commerce visibility matters to you. Verify IP ranges against each company's published list rather than trusting user-agent strings alone, since user-agents can be spoofed by scrapers with no relationship to the real crawler.
FAQ
How do I know if Cloudflare is blocking AI crawlers on my site?
Run the curl test with a GPTBot or ClaudeBot user-agent against your domain. A 403 or 429 response means Cloudflare's bot management or WAF is blocking the request before it reaches your origin, regardless of what your robots.txt says.
Does fixing robots.txt fix this problem?
No. Robots.txt is a voluntary signal that well-behaved crawlers check after they've already been allowed to connect. If the WAF rejects the connection first, robots.txt is never read.
What's the difference between Cloudflare's Training and Search bot controls?
Training controls govern crawlers used to build AI model training data, Search controls govern retrieval-time crawlers used to answer live queries, and Agent controls govern AI agents acting on a user's behalf. Blocking Training can inadvertently block Googlebot and Bingbot too, so use "Disallow AI Training" if you want to opt out of training without losing search crawlers.
