On September 15, 2026, Cloudflare rolled out a major infrastructure change affecting millions of websites: AI crawlers are now blocked by default on any page carrying ads, while traditional search bots like Googlebot remain untouched. Behind this technical switch lies a serious SEO and GEO issue — if you haven’t checked your setup, your site could be silently losing citations in ChatGPT, Perplexity, and Google’s AI Overviews.
What exactly changed on September 15, 2026?
Cloudflare sits in front of a massive share of global web traffic, making it a strategic checkpoint for AI crawling activity. Starting September 15, 2026, every new site built on Cloudflare, along with existing free-tier customers, has AI training bots and autonomous AI agents blocked by default on any page that displays advertising.
This rule doesn’t touch traditional search engines: Googlebot, Bingbot, and other indexing crawlers keep full access to content as before. The technical distinction specifically targets bots used to train models or to power conversational assistants and autonomous AI agents fetching content in real time.
An evolution of the Pay Per Crawl program
This default shift builds on Cloudflare’s earlier Pay Per Crawl program, now evolving into a “Pay Per Use” model. The goal: let publishers charge AI companies not just when their content is fetched, but when it actually creates value for those platforms. Pricing starts around one cent per page, and each site owner can set their own terms.
Why this shift changes the rules for your SEO and GEO strategy
Over the past two years, a growing share of discovery traffic no longer comes solely from classic search results, but from answers generated by conversational AI. Being cited by ChatGPT, Perplexity, Claude, or Google’s AI Overviews has become its own acquisition channel — often called GEO (Generative Engine Optimization).
The problem: a default block, activated automatically without any action on your part, can cut off that access overnight. An e-commerce site or a business website that relied on these citations for qualified traffic could see that source vanish silently, with no visible warning inside standard tools like Google Search Console.
The concrete risk for unchecked sites
- Loss of visibility in AI-generated answers with no notification
- Breaking of citations already earned from Perplexity or other generative engines
- Confusion between a deliberate block (content protection) and an unintended default block
- No tracking in traditional SEO reports, which don’t measure this type of traffic
How to check if your site is affected
If your site sits behind Cloudflare, even on the free plan, the first step is to check your current AI bot management configuration. There are several things to verify before jumping to conclusions.
Verification steps
- Log into your Cloudflare dashboard and check the section dedicated to AI bot traffic management
- Identify whether your ad-monetized pages are affected by the new default block
- Review your robots.txt file to confirm it actually reflects your intentions toward AI crawlers
- Distinguish training bots from the bots used for real-time citations in generative answers
- Test access to your key content using AI crawl verification tools
Monetize instead of losing out: the potential of the Pay Per Use model
Rather than treating this change as a constraint, some publishers see it as an opportunity. The Pay Per Use model turns previously invisible, unpaid bot traffic into an additional revenue stream. Early adopters like Ceramic.ai and You.com are among the first partners testing this usage-based billing system.
For a site producing high-value, technical, or expert content, this option deserves a case-by-case evaluation: keep free visibility to maximize citations, or enable monetization that may reduce how much certain AI agents crawl your content.
Action plan for website owners
When facing this kind of infrastructure-level policy change, systematic verification beats assumption every time.
- Audit your Cloudflare configuration (or your equivalent CDN, if you use a different provider)
- Define a clear policy per AI bot type, aligned with your traffic and monetization goals
- Monitor your citations in generative AI through Search Console’s new AI performance reports
- Document your choices so you can adjust quickly if another default change occurs
Balancing visibility, content protection, and monetization is becoming its own SEO skill. For a full diagnosis of how your site is being crawled by both search engines and AI systems, the seoforge.io team offers an SEO audit and GEO analysis to help you secure your visibility, whatever direction platform policies take next.
