Cloudflare Disallow AI Training: What It Blocks

On 15 September 2026 Cloudflare turned on a setting called Disallow AI Training for every customer on every plan, from Free to Enterprise. It does one specific thing: it tells Google, Apple and Microsoft not to train on your pages while leaving their search crawlers free to index you normally. Until now those were the same switch.

The number that explains why this took so long is 36.6%. That is the share of verified crawler traffic on Cloudflare's network coming from mixed-use crawlers, bots that fetch a page once and feed both a search index and a training corpus. It is the largest crawler category Cloudflare tracks. The gap in what site owners want is just as stark: fewer than 1% of sites on the network block search bots, while 17% have already switched on some mechanism to block AI training. Seventeen percent of the web wants half of what a mixed-use crawler does and has had no way to ask for it.

The setting is live now and it is genuinely new. It is also narrower than the headline suggests, and the narrowing matters more than the feature.

What Cloudflare actually shipped

Three things landed together. The first is a reclassification. Cloudflare now sorts AI bots by behaviour rather than by owner, into Search, Training and Agent. A single company can appear in more than one class, which is the point: Googlebot is a search crawler and a training crawler wearing one user agent.

The second is the Disallow AI Training control itself, exposed in domain-level Security Settings in the Cloudflare dashboard. Existing customers do not need to do anything to keep their old behaviour. Cloudflare's press release confirms the legacy Block AI Bots setting migrates automatically into the new Training control.

The third is a label. Cloudflare has created an "Accountable" designation for mixed-use operators, and three companies currently qualify: Apple, Google and Microsoft. Accountable is not a compliment, it is a checklist. An operator must offer site owners a way to opt out of AI training through robots.txt or a similar standard, offer a separate opt-out for AI summaries, provide URL-level visibility into which pages were made available for training alongside metrics on how the content appeared in search, and give assurance that opting out of training will not degrade traditional search results.

Matthew Prince, Cloudflare's co-founder and chief executive, framed it as "preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used".

The split that makes this possible, and the half that is only a promise

There are two enforcement models hiding inside one switch, and the difference decides how much you should trust it.

Training-only crawlers, the ones from Amazon, Anthropic, Meta and OpenAI that exist purely to collect training data, get blocked at Cloudflare's edge. That is a network refusal. The request does not complete.

Mixed-use crawlers from the three Accountable operators are handled differently. Cloudflare writes a preference into your robots.txt using each operator's own opt-out token, Applebot-Extended for Apple and Google-Extended for Google. Googlebot and Applebot still arrive, still fetch the page, and are trusted to honour the line. Nothing is blocked. You are relying on a published commitment.

That is not a criticism of the design so much as the only design available. You cannot block Googlebot without leaving Google's index. But it does mean the protection you get is uneven, and worth seeing in one view.

Read the right-hand column before the left. Two of the six rows are dated 2027, and one of them is the row most creators actually care about.

How to turn it on, in six steps

The whole change is a dashboard toggle, but the steps around it are where the value is. Budget about twenty minutes.

1. Decide what you are protecting. Training is the model learning from your work. Search is discovery. Agents are AI assistants fetching your page live on a user's behalf. These are three separate economic questions and Cloudflare now lets you answer them separately, so answer them separately rather than reaching for the strictest option.
2. Check what you have already set. Open your zone, go to Security and then Settings, and look at the current AI bot configuration. If you turned on Block AI Bots at some point in the last two years, it has already been migrated into the Training control and you may be done.
3. Set Training to disallow, leave Search allowed. This is the new capability. Everything else on the page is older behaviour.
4. Decide Agent separately. If your pages carry ads or affiliate links, an agent fetch is a page view with nobody there to see them. Cloudflare's own recommendation for ad-supported domains is search allowed, training disallowed, agents blocked on ad-carrying pages.
5. Verify the robots.txt. Fetch yourdomain.com/robots.txt and confirm the new directives appear. Cloudflare generates the file through Bot Preference Sync, which has been generally available since 21 August 2026 and prepends its block to whatever you already had, so your existing rules survive. If you maintain robots.txt by hand in a repo, this is the step where you find out that two systems now write to the same file.
6. Write down the date. You are making a claim about consent that starts today and is not retroactive. Anything already scraped is already in a corpus.

What the setting does not cover

Four gaps, and none of them are hidden. Cloudflare states all of them.

AI summaries are a separate fight. The setting governs training, not whether your text is summarised in a results page. Granular control over summaries through Cloudflare is dated early 2027. Until then it is per-operator: Google AI Overviews is managed in Search Console, and Microsoft's equivalent runs through the NOARCHIVE and nosnippet directives it introduced for Bing Chat. This is the gap that costs money. Cloudflare cites research that over 50% of consumers now read AI summaries in search results and are over 40% more likely to end the search there.

Bing is not ready. Microsoft holds the Accountable designation on a commitment, not a shipped mechanism. Its robots.txt support for training opt-out is targeted for early 2027. Google's URL-level transparency reporting, the third Accountable requirement, is described as arriving in the weeks to come.

Ad pages cannot be singled out. Cloudflare can detect which of your pages serve ads, but says that list is too large and changes too often to enumerate in robots.txt. So the recommended posture for ad-supported sites is partly aspirational at the robots.txt layer.

Everything outside the Accountable three is unchanged. Training-only crawlers get blocked, which is real. But a mixed-use crawler from an operator that has not signed up to the checklist is in neither bucket, and the longer-term fix is a standard rather than a vendor setting. That work is happening at the IETF, in the AI Preferences working group.

Who should turn this on, and who should not bother

If you run a portfolio, a tutorial blog, a stock or preset shop, or anything where your own images and prose are the product, turn it on. The cost is close to zero and the signal is now legally and technically legible in a way that a bare robots.txt line from 2023 was not. It pairs naturally with per-asset provenance work of the kind we covered in Apple's Reference Image: one says how this was made, the other says what may be done with it.

If your traffic depends on being cited by AI assistants, be deliberate rather than reflexive. The whole premise of this feature is that refusing training does not cost you search, and Cloudflare cites AI search referrals converting at three to five times the rate of traditional search. That premise is the thing to monitor, not assume. We track what models actually recommend in our AEO tracker, and the honest position today is that nobody has a clean before-and-after on a site that flipped this switch, because the switch is one day old.

If you are a video creator whose work is mostly on someone else's platform, this setting does nothing for you. Your leverage is at the platform's terms, not your own domain, which is the same structural problem behind synthetic video detection: the control sits with whoever hosts the file.

The realistic summary is that Cloudflare has split one bad choice into two better ones and shipped the easier half first. The half that is enforced covers crawlers most creators had already blocked. The half that covers Google and Apple is a promise with a reporting requirement attached, which is more than existed on 14 September and less than control.

Frequently asked questions

Does Disallow AI Training remove my site from Google search?

No. That separation is the entire feature. Cloudflare writes the Google-Extended opt-out token, which governs training, and leaves Googlebot's search crawling untouched. The fourth Accountable requirement is an explicit assurance from the operator that opting out of training will not affect traditional search results.

Is this available on the free plan?

Yes. The setting is available to all customers from Free through Enterprise. Bot Preference Sync, the mechanism that writes the robots.txt entries, is also available on every tier and is on by default for new customers.

What happens to my existing robots.txt rules?

They stay. Cloudflare prepends its generated preferences to the existing file rather than replacing it. The thing to watch for is a conflict of authorship: if your robots.txt is generated by a static site build or edited in a repo, you now have two writers.

Does this stop my content appearing in ChatGPT or Claude answers?

Partly, and not in the way most people mean. The training-only crawlers run by OpenAI, Anthropic, Meta and Amazon are blocked at Cloudflare's edge under the Training control, so future training collection is refused. It does not retroactively remove anything already trained on, and it does not stop an AI agent fetching your page live to answer a user's question. That is the separate Agent control.

Can I block AI summaries of my pages today?

Not through this setting. Cloudflare's granular AI summaries control is dated early 2027. Today it is per-operator: Search Console for Google AI Overviews, and the nosnippet and NOARCHIVE directives for Bing. Given the click-through data, this is the gap worth tracking.

Why does the Accountable list only have three companies on it?

Because the designation requires four specific commitments including URL-level reporting on what was used for training, and Apple, Google and Microsoft are the three mixed-use operators who have met or committed to all four. Microsoft's robots.txt piece is not shipped until early 2027, so the designation currently reflects a commitment as much as a capability.