TL;DR
- Pull
https://yourdomain.com/robots.txtand confirm it's at the root — subdirectory files are ignored. - Make explicit allow/block decisions for three categories: training crawlers (GPTBot, Google-Extended), answer-engine crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot), and traditional search bots (Googlebot, Bingbot).
- Anthropic ships multiple user agents — one directive for "ClaudeBot" does not cover them all.
- Blocking AI training bots has no effect on Google Search rankings, but it can quietly cut off citation surfaces in ChatGPT and Perplexity.
- Robots.txt is a policy file, not a firewall. Verify with server logs; enforce at the edge if you need teeth.
Most robots.txt files I audit in 2026 still look like they were written in 2019: a Disallow: /wp-admin/ line, a sitemap reference, and nothing about AI. That gap is now a strategic decision, not a technical oversight. Below is the 5-minute audit I run before touching anything else on a GEO engagement.
Step 1: Confirm the file exists at the root (30 seconds)
Open a browser tab to https://yourdomain.com/robots.txt. If you get a 404, a redirect, or HTML, stop and fix that first. Crawlers only check the root of the host — /blog/robots.txt or /en/robots.txt is invisible to them, as Google's own spec reiterates and MO Agency's audit walkthrough flags as the most common failure mode.
While you're there, check:
- Status code is 200 (not 301/302 to a different host).
- Content type is
text/plain. - File is under 500 KB — Google truncates beyond that.
- No accidental
Disallow: /underUser-agent: *left over from a staging deploy.
Step 2: Inventory current AI directives (60 seconds)
Search the file for these user-agent strings. If any are missing, you have no policy for that crawler — which means the default is "allow."
| Category | User agents to look for |
|---|---|
| OpenAI training | GPTBot |
| OpenAI answers | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-Web, anthropic-ai |
| Perplexity | PerplexityBot, Perplexity-User |
| Google AI training | Google-Extended |
| Common Crawl | CCBot |
Two non-obvious points. First, Google-Extended is a robots.txt-only token — there is no meta tag or HTTP header equivalent, so robots.txt is the only lever for opting out of Gemini training while keeping Google Search indexing intact. Second, Anthropic operates several distinct user agents for training, fetching, and indexing; a single ClaudeBot block leaves the others untouched.
Step 3: Make a deliberate decision per category (90 seconds)
The decisions are not symmetric. I think of crawlers in three tiers, an architecture CapConvert's configuration guide lays out well:
- Training crawlers (GPTBot, Google-Extended, CCBot, anthropic-ai). These scrape content to train base models. Blocking them does not affect search rankings and does not directly block citations. If your content is your moat, block. If you want maximum surface area in future model weights, allow.
- Answer-engine crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-Web). These fetch pages at query time to ground answers. Blocking these reduces the odds of being cited in live AI responses. Most publishers should allow.
- Traditional search bots (Googlebot, Bingbot). Do not block unless you know exactly why.
A reasonable default for a B2B SaaS that wants AI citations but is cautious about training:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
Order matters less than people think — each User-agent block is independent — but keep one blank line between blocks and avoid wildcards you don't fully understand.
Step 4: Verify with logs, not just syntax (90 seconds)
Robots.txt compliance is voluntary. Well-behaved crawlers honor it; less scrupulous ones spoof user agents or ignore the file entirely. Cloudflare's analysis of AI crawler traffic shows meaningful gaps between declared policies and observed behavior.
Two verification steps:
- Syntax check. Paste your file into Google's robots.txt Tester (still available in Search Console legacy tools) and test specific URLs against
GPTBotandGooglebotto confirm the rules resolve the way you expect. - Log check. Grep your access logs for
GPTBot,ClaudeBot,PerplexityBot. If you blocked them last month but they're still hitting 200s on your articles, you have a real problem — likely a CDN cache layer serving before the directive takes effect, or a bot ignoring the file.
If logs show non-compliance and you care, enforce at the edge. Cloudflare offers a managed robots.txt feature that publishes a maintained block list and pairs it with network-level enforcement so blocked bots get a 403 regardless of whether they read the file.
Step 5: Set a review cadence (30 seconds)
New AI user agents appear every quarter. GPTBot adoption alone grew sharply through 2024–2025 as ChatGPT's browsing features expanded. Put a calendar reminder — quarterly is fine for most sites, monthly if you publish high-value research that gets scraped aggressively. Each review: pull the file, diff against the previous version, check logs for new user agents you haven't seen before, decide, ship.
FAQ
Does blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's crawler; Googlebot is Google's. They are completely separate systems. You can block GPTBot and Google-Extended without any impact on Google Search visibility — only Gemini training and ChatGPT-derived traffic are affected.
Will blocking PerplexityBot stop Perplexity from citing me?
Not reliably. Answer engines can cite pages they retrieve through other means — third-party indexes, cached snippets, or user-submitted URLs. Blocking does reduce the odds of fresh, accurate citation and is more likely to result in being skipped or misquoted than fully suppressed.
Is one ClaudeBot directive enough for Anthropic?
No. Anthropic runs at least three documented user agents (ClaudeBot, Claude-Web, anthropic-ai) for different purposes. If you want a complete block or allow for Anthropic, write a directive for each.
