An Anthropic bot now crawls 20,583 pages for every single visit it sends back to the publisher that produced them. That number, surfaced by several 2026 server-log analyses, captures the dilemma every website owner has faced over the past year: should you let AI crawlers harvest your content, block them outright, or find a middle path? The answer isn’t a matter of principle anymore — it’s a matter of data.
AI bots already make up a third of all web traffic
According to 2026 server traffic measurements, bots now account for roughly 30.6% of all web traffic, a share that has kept climbing for two straight years. But not all bots are equal: of the traffic generated specifically by AI crawlers, 89.4% is training-related or mixed-purpose, only 8% is search-related, and barely 2.2% corresponds to an actual user query being answered in real time.
In practical terms, for a mid-sized site this means the vast majority of AI bot visits deliver no immediate value at all — no click, no reader, no conversion. They consume bandwidth, load your server, and feed models that may never send a single visitor back.
Who crawls the most — and who gets blocked
An analysis of robots.txt files across a sample of more than 4,000 sites shows a clear shift in 2026: the share of sites blocking at least one AI crawler jumped from 3.63% to 8.56% within a few months. The ranking of the most-blocked bots tells its own story:
- GPTBot (OpenAI) — blocked in 5.52% of analyzed Disallow rules
- CCBot (Common Crawl) — 5.08%
- ClaudeBot (Anthropic) — 4.88%, climbing sharply since April 2026
- Google-Extended — 4.44%
- Bytespider (ByteDance) — 4.23%
ClaudeBot’s rapid climb up this ranking isn’t a coincidence — it’s exactly the crawler with the most lopsided crawl-to-referral ratio in the industry, which explains publishers’ growing frustration with it.
The real problem isn’t the crawl, it’s the ratio
Blocking a bot on principle makes little sense if it sends you qualified traffic in return. The metric that actually matters is the ratio between pages crawled and visitors sent back:
- ClaudeBot: roughly 20,583 pages crawled for every 1 referral sent back
- GPTBot / OpenAI: roughly 1,255 pages crawled for every 1 referral
- Meta: a referral rate close to zero, regardless of crawl volume
These gaps radically change the right strategy. A bot that crawls heavily but almost never sends a visitor back is a pure cost on your server resources with no clear business return, while a bot that regularly cites your pages in generative answers can become a high-intent traffic source — much like a traditional search engine.
Block, allow, or split the difference?
The strategy leading publishers are using
Roughly 30% of high-traffic sites have adopted a differentiated approach: they block bots dedicated to training (GPTBot, CCBot, ClaudeBot, Google-Extended) while allowing bots dedicated to real-time search and citation (such as OAI-SearchBot or Claude-User, which are distinct from ClaudeBot itself). The goal is simple — stay visible in AI answers without freely feeding competitors’ model training.
Why a blanket block can backfire
Blocking an AI crawler entirely also means disappearing from the generative answers that bot feeds. And these assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) are becoming a high-intent source of traffic for purchase decisions and precise information needs. An overly broad block therefore costs you potential citations, without necessarily reducing server load if other bots keep crawling aggressively anyway.
How to decide for your own site
Before touching your robots.txt file, a few checks are worth doing first:
- Audit your server logs to measure the real share of bot traffic, bot by bot, over the last 30 days
- Tell user-agents apart: a training crawler (GPTBot) doesn’t deserve the same treatment as a real-time search crawler (OAI-SearchBot)
- Pair robots.txt with an llms.txt file if you want finer control over what models are allowed to cite
- Revisit your policy every quarter, since AI vendors’ user-agents and crawling practices keep changing fast
There’s no universal rule here: an e-commerce site that lives off organic traffic doesn’t face the same trade-offs as a media outlet monetizing through display advertising. What matters is basing this decision on your own log data, not on a general hunch.
Want to know which AI bots are actually visiting your site and how they’re affecting your performance? The team at seoforge.io offers a full SEO audit that includes server log analysis and concrete recommendations for your robots.txt and llms.txt strategy.
