← / BlogResearch
ai-crawlersrobots-txtresearchgeo

What % of sites block AI crawlers in 2026, and who unblocked

Roughly a quarter of top sites block major AI crawlers in 2026, but the internet-wide rate is far lower — and news publishers are moving to allowlists.

By 5 min read

TL;DR

  • Among top-ranked sites (Tranco top 2,000 sample), about 25.1% block at least one major AI training crawler; GPTBot sits at 19.7%, ClaudeBot at 18.8%.
  • Across the open web — 90.8 million sites with a robots.txt file — only 6.88% block GPTBot, showing that blocking is concentrated among large, well-resourced publishers rather than the median site.
  • GPTBot's block rate among top sites climbed from ~5% in early 2023 to ~25% by 2024 and has been flat through 2026 — the "blocking wave" has largely finished among sites that were going to act.
  • The newest pattern isn't blanket blocking, it's splitting: roughly 30% of top sites now disallow training bots (GPTBot, CCBot, ClaudeBot, Google-Extended) while leaving search-and-cite bots (OAI-SearchBot, PerplexityBot) open.
  • News publishers are the outlier: 60% block at least one AI crawler, up from 23% in September 2023, and several large outlets have flipped to default-deny allowlists to gain licensing leverage.

Robots.txt blocking of AI crawlers refers to publishers adding Disallow rules in robots.txt for user agents like GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, or OAI-SearchBot. These rules are advisory — compliant bots honor them, but nothing technically prevents a crawler from ignoring the file, which is part of why enforcement debates have intensified alongside the numbers below.

The top-site rate vs. the web-wide rate

The single most important thing to get right when citing a "percentage of sites blocking AI" statistic is which population it describes. Studies of the Tranco top 2,000 domains — the sites that generate most traffic and most training-data value — put the block rate for at least one major crawler at 25.1%, with GPTBot specifically blocked by 19.7% of that sample and ClaudeBot by 18.8%.

Zoom out to the full web, however, and the picture changes sharply. Across 90.8 million sites that actually serve a robots.txt file, only 6.88% (about 6.25 million sites) block GPTBot. That's a more than 3x gap between the top-of-the-web rate and the internet-wide rate. The explanation is straightforward: blocking requires someone to notice the issue, understand robots.txt syntax, and care enough about training-data extraction to act. Large media companies, SaaS platforms, and high-traffic publishers have the resources and incentive to do that; the long tail of small business sites, blogs, and low-traffic domains mostly don't, either because they don't know AI crawlers exist or because they see no downside to being crawled. If you're auditing your own site's exposure, a robots.txt AI crawler checklist is a faster way to find out where you stand than relying on either of these aggregate numbers.

The blocking curve has flattened

GPTBot blocking among top sites didn't rise steadily — it spiked. From roughly 5% in early 2023, the rate climbed to about 25% through 2024 as the initial wave of publisher concern about AI training hit its peak. Since then, through 2026, the top-site block rate has been essentially flat. That plateau suggests the population of sites willing to block training crawlers has mostly already acted; new blocking at this point is more likely to come from new site launches or from sites responding to renewed licensing disputes than from a continuing wave of first-time blockers. This matters for forecasting: if you're trying to predict whether a given competitor or publisher will start blocking, the base rate of "sites that haven't blocked yet start blocking" is now much lower than it was in 2023-2024.

Split access is the dominant 2026 strategy

The more interesting trend isn't the block rate itself but what publishers are blocking. Roughly 30% of top sites now follow a "split" configuration: they disallow crawlers associated with model training — GPTBot, CCBot, ClaudeBot, Google-Extended — while explicitly allowing crawlers associated with real-time answer generation and citation, such as OAI-SearchBot and PerplexityBot. This is a deliberate distinction between "don't use my content to train a model" and "do use my content to answer a user's question right now, with attribution." For site owners trying to balance AI visibility against training-data concerns, this split approach is quickly becoming the default playbook rather than an edge case. It also means that a single aggregate "AI crawlers blocked" number is close to meaningless without specifying which user agent and which purpose.

News publishers are blocking at triple the general rate — and some are reversing into allowlists

News sites behave nothing like the general top-site population. 60% of reputable news publishers block at least one AI crawler, compared to just 9.1% of sites flagged as misinformation sources, and up sharply from 23% in September 2023, according to peer-reviewed research presented at the ACM Web Conference 2026. The gap between reputable and low-quality sources is itself notable — it suggests blocking correlates with having something valuable (and legally defensible) to protect, not with general caution about AI.

The more consequential shift in 2026 is a move away from simple blocklists toward default-deny allowlists. Reuters and Time both switched to blocking all crawlers by default and only permitting specifically approved bots as of May 2026, joining People Inc. and The Atlantic, which had already adopted this posture. The stated rationale isn't pure refusal — it's leverage. A default-deny allowlist forces any AI company that wants access to negotiate a licensing deal rather than simply crawl and later face a takedown request, which is a materially stronger negotiating position for the publisher.

Whether this strategy pays off is contested. A Rutgers/Wharton working paper published December 31, 2025 found that news publishers who blocked AI crawlers saw a 23.1% decline in total monthly visits (per SimilarWeb) and a 13.9% decline in human-only visits (per Comscore panel data) after blocking took effect. If that finding holds up under further scrutiny, it implies blocking carries a real traffic cost even when the goal is licensing leverage rather than outright exclusion — a tradeoff every publisher weighing a block decision should model before flipping the switch. For a broader sense of how citation patterns shift once publishers change crawler access, AI citation volatility tracking is a useful companion read.

If you want to know whether your own site is unintentionally blocking AI crawlers — a common issue with WAFs and CDN defaults — CiteFlow's free scan checks crawler access as one of its seven audit dimensions, and paid plans include a daily check that flags when AI crawlers become blocked.

FAQ

What percentage of websites block GPTBot in 2026?

It depends on the population: about 19.7% of top-ranked sites (Tranco top 2,000) block GPTBot, but only 6.88% of all sites with a robots.txt file across the broader web do so.

Are news publishers more likely to block AI crawlers than other sites?

Yes — 60% of reputable news sites block at least one AI crawler, roughly triple the top-site average, and the rate has risen from 23% in September 2023.

Why are some publishers switching to allowlists instead of blocklists?

Reuters, Time, People Inc., and The Atlantic adopted default-deny-with-allowlist configurations in 2026 to create negotiating leverage for AI licensing deals, rather than simply refusing crawler access outright.

Sources

Drafted with AI assistance and published by CiteFlow. Sources are linked in the text.