On July 1, 2026, Cloudflare gave the AI industry a deadline. From September 15, its default settings will block "mixed-use" AI crawlers on ad-supported pages, and the block applies automatically to every free-plan site whose owner never touched the settings. If part of your pipeline now starts with a buyer asking ChatGPT or Gemini for a recommendation, this is worth ten minutes of your attention.
Cloudflare announced on July 1, 2026 that from September 15 it will block, by default, AI crawlers that blend search, training, and agent use. The new defaults allow search but block training and agent access on pages with ads, chosen for you unless you set your own. If being discoverable in AI answers matters to your business, your crawler access is now a setting you must choose deliberately, not a default you can ignore.
What is actually changing
Until now, an AI company's crawler could fetch your site for three different purposes at once: powering search-style answers, training future models, and feeding live AI agents. One user-agent, one robots.txt line, no way to say yes to one purpose and no to another. Cloudflare's move forces the split. Crawlers that refuse to separate those intents get blocked by default on pages that carry ads.
| Date | What happens |
|---|---|
| July 2025 | Cloudflare launches Pay Per Crawl in private beta: sites can allow, charge, or block individual AI crawlers |
| July 1, 2026 | New classifications announced. Pay Per Crawl evolves into Pay Per Use: publishers get paid when content powers an answer rather than per fetch. First partners: Ceramic.ai and You.com |
| July to September 2026 | Cloudflare runs tests and gathers ecosystem feedback to finalize the new defaults |
| September 15, 2026 | Defaults flip: search allowed, training and agent use blocked on pages with ads. Mixed-use crawlers blocked on all ad pages. Applies to new customers, new sites, and all existing free-plan customers who have not changed settings |
Source: Cloudflare's announcement and TechCrunch's coverage, both July 1, 2026. The 2025 launch is documented in Cloudflare's own pay per crawl announcement.
Why one company's defaults move the whole web
Cloudflare sits in front of about a quarter of all websites (W3Techs, August 2026), so its defaults function like policy. And the numbers behind the decision explain the urgency: automated agents and bots now drive more than half of all web requests, and Cloudflare's own data shows over 50% of AI crawl traffic is spent re-fetching pages that have not changed.
Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.Matthew Prince, co-founder and CEO of Cloudflare, July 1, 2026
The part that concerns you: defaults decide your AI visibility
Most coverage frames this as publishers versus AI companies. For a service business the frame is different. You probably do not monetize content with ads. You monetize being found, and increasingly your buyers ask an answer engine before they ask Google. In that world, crawler access is distribution.
Here is the uncomfortable part: your settings may already be wrong, and you would never know. When we ran a GEO audit on our own site this spring, we found our Cloudflare configuration quietly blocking the very AI crawlers we wanted citing us. Nothing was broken. Nothing alerted anyone. The answer engines simply could not read the site. We fixed one setting; that is not a capability problem, it is an awareness problem.
If your site runs on a Cloudflare free plan and nobody has ever opened the AI crawl settings, the new defaults will be chosen for you. They may match what you want. They may not. The point of the deadline is that "we never decided" stops being an option.
Search, training, agents: decide each one separately
The useful outcome of Cloudflare's move is vocabulary. There is no longer one question ("allow AI?") but three, and a business can sensibly answer them differently.
| Intent | What the crawler does | Typical call for a service business |
|---|---|---|
| Search / citation | Reads pages to answer user questions with links and attribution | Allow. This is how AI-era buyers discover you |
| Training | Ingests content to train future models, no attribution | Your choice. For most B2B sites the content is marketing, not IP; visibility usually outweighs protection |
| Agent use | AI agents fetch pages live to complete tasks for a user | Usually allow. An agent comparing vendors for a buyer should be able to read your site |
AEO is now official vocabulary
Buried in the same announcement, Cloudflare introduced an Attribution Business Insights dashboard and named the discipline it serves: Answer Engine Optimization. Beyond knowing who crawls you, AEO tracks how often and how prominently your content is cited in AI answers across models. This is the measurement layer that Generative Engine Optimization has been missing.
The playbook itself has not changed, and it favors smaller players: research on GEO methods found that adding citations, quotations, and statistics lifted visibility in AI answers by up to 40%, and a Semrush study found about 90% of the pages ChatGPT cites rank at organic position 21 or lower. Answer engines do not inherit Google's rankings. That is the opening.
What to do before September 15
- Find out whether your site sits behind Cloudflare, and on which plan. Your developer or hosting invoice knows.
- Open the AI crawl controls and read what is currently allowed and blocked. Write it down; that is your baseline.
- Decide the three intents deliberately: search, training, agent use. Involve whoever owns marketing, because this is a distribution decision.
- Check robots.txt too. Cloudflare is one layer; your robots file is another, and they can contradict each other.
- After September 15, verify the defaults did not overwrite your choices. Then check your pages are actually being read by the engines you allowed.
If you want the deeper version, with citability scoring and per-engine access checks, that is exactly what our GEO and AI-visibility diagnostic does. We check the plumbing before we tune the message; it took our own site from silently blocked to cited.
Q.Will blocking AI training remove my business from AI answers?
Not by itself. Search-intent crawling and training are separate controls, and Cloudflare's new defaults keep search allowed. One nuance from Google's world: opting out via Google-Extended stops Gemini training use, but Google's AI Overviews are fed by the normal Googlebot, which you almost certainly want to keep allowing.
Q.My site is not on Cloudflare. Does any of this apply?
The specific September 15 defaults do not, but the direction does. Crawler access is becoming a per-intent decision across the industry, and your robots.txt plus your CDN or firewall settings are where that decision lives today. The 30-minute check above works anywhere.
Q.Will AI companies actually pay small sites for content?
Not meaningfully, yet. Pay Per Use launched with two partners, and the 50+ licensing deals signed so far are with large publishers. For a service business the realistic near-term value of AI crawlers is pipeline, not licensing revenue: being the cited answer when a buyer asks.
Q.Is this the same as GEO?
It is the infrastructure half of it. GEO covers both access (can engines read you?) and citability (do they quote you?). Cloudflare's change moves the access half from "set and forget" to "decide by September 15", and its new analytics make the citability half measurable.
