TL;DR
- AI crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended) treat 429 and 503 responses as a signal to back off, not a prompt to retry harder — repeated throttling can cause them to deprioritize or abandon a domain.
- "Protect from scraping" presets in CDNs and WAFs often apply the same rate ceiling to AI crawlers as to malicious bots, which silently suppresses the exact traffic you want.
- Your server logs will show the pattern before your citation numbers do: spikes of 429/503 status codes against known AI user agents, often clustered in short bursts.
- Per-user-agent rate limits (not a single global ceiling) let you throttle abusive scraping while leaving legitimate AI crawlers enough headroom to finish a crawl pass.
- Fixing this is a config change, not a content change — and it's one of the few GEO fixes you can verify in logs within a day.
Rate limiting is standard practice for protecting origin servers from scraping, credential stuffing, and traffic spikes. The problem is that most rate-limiting rules were written for human-adjacent bot traffic, not for the crawl patterns of LLM providers. When PerplexityBot or GPTBot hits your rate ceiling, it doesn't fight through it — it reads the 429 as "not now" and moves on, sometimes permanently deprioritizing your site in future crawl schedules. Nobody notices because the failure mode isn't an error message, it's an absence: your content just stops showing up as a source.
Why 429s hit AI crawlers harder than you think
Standard web crawlers (Googlebot, Bingbot) have spent two decades tuning retry and backoff behavior against rate limits, and they crawl the same domains repeatedly over years, so a bad day doesn't matter much. AI crawlers are newer, often run leaner crawl budgets per domain, and in several documented cases explicitly reduce or pause crawl activity after sustained 429/503 responses rather than retrying aggressively — this is described in OpenAI's GPTBot documentation and echoed in crawler-behavior guidance from Cloudflare's AI bot research. The practical effect: a crawler that gets throttled once may come back with a smaller request budget next time, and if it keeps getting throttled, it stops bothering with your domain at all. Since these bots feed the index that LLMs draw citations from, a crawl that never completes means your pages never make it into the pool of citable content in the first place.
Where "protect from scraping" settings quietly block citations
Most CDN and hosting dashboards have a single toggle like "Bot Fight Mode" or "Scraping Protection" that applies one aggressive rate ceiling across all non-human traffic. This is convenient and it is also the single most common reason sites lose AI visibility without anyone touching robots.txt. The setting doesn't distinguish GPTBot from a scraper farm — it just counts requests per IP or per UA within a window and fires 429s past the threshold. If that threshold is tuned for defending against scraping (often very low, sometimes under 10 requests per minute per UA), a legitimate AI crawler doing a routine pass over a sitemap will get throttled almost immediately. This overlaps with the WAF misconfiguration problem covered in Cloudflare blocking AI crawlers — but rate limiting is a distinct failure mode from an outright block, and it's easier to miss because the crawler isn't denied, it's just slowed to the point of giving up.
Reading the pattern in your server logs
You don't need a specialized tool to catch this — grep your access logs for known AI user-agent strings and look at the status code distribution. The tell-tale signature of throttle-induced abandonment is:
- A burst of requests from a given UA (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Amazonbot) concentrated in a short window
- A rising share of 429 or 503 responses as the burst continues
- A sharp drop-off in request volume shortly after, sometimes followed by no return visits for days or weeks
Compare this against your 2xx-only baseline for the same UA over the prior month. If the 429 share is climbing while total request volume from that UA is falling, you're watching a crawler get rate-limited into leaving. This is a companion diagnostic to the broader log-monitoring workflow in monitoring AI crawler server logs — the difference here is you're specifically isolating throttle responses, not just presence/absence of hits.
Setting sane per-UA limits
The fix is rarely "remove all rate limiting" — that reopens you to actual scraping abuse. The fix is per-user-agent rules instead of a single global ceiling:
- Allowlist verified AI crawler UAs (verify via reverse DNS or published IP ranges, since UA strings can be spoofed) and give them a higher, separate rate ceiling than anonymous traffic.
- Set the AI-crawler ceiling based on your page count and crawl frequency needs — a site with a few hundred pages rarely needs more than a moderate requests-per-minute allowance to complete a pass, but syndication-heavy or large catalog sites need meaningfully more headroom.
- Return 503 with a
Retry-Afterheader instead of a bare 429 when you do need to shed load — this tells well-behaved crawlers when to come back instead of signaling permanent rejection. - Monitor for a few days after any change and confirm 429 share drops for the allowlisted UAs specifically, not just overall.
FAQ
How do I know if rate limiting, not robots.txt, is the problem?
Check robots.txt first — if AI crawler UAs aren't disallowed there, pull logs for those UAs and look at status code distribution; a high share of 429/503 responses points to rate limiting rather than a hard block.
Will raising rate limits for AI crawlers open me up to scraping abuse?
Not if you scope the higher ceiling to verified AI crawler user agents (confirmed via reverse DNS or published IP ranges) rather than raising the global limit, since spoofed UAs won't pass verification and still hit the default ceiling.
Do AI crawlers retry after a 429, or do they just leave?
Behavior varies by provider, but documented guidance from crawler operators indicates repeated throttling reduces future crawl frequency rather than triggering aggressive retries, so sustained 429s function more like a soft block than a temporary delay.
