Cloudflare adds a Disallow AI Training setting that keeps Google, Apple and Bing crawling for search

Cloudflare has launched a new Disallow AI Training setting on the Training control it ships to every zone, and it says sites that already block AI training will be migrated onto it. The setting publishes a no-training preference in robots.txt while still letting Googlebot, Applebot and Bingbot crawl the site for search. It went live on 15 September.

The change reverses a plan Cloudflare described in July, under which sites that blocked AI training on 15 September would also lose those three crawlers, because each is used for both search and training. Cloudflare says most customers will see no change: existing Training selections of Block or Block on pages with ads will move to Disallow AI Training, and Block AI Bots and the Managed Robots.txt feature are being retired.

What the three controls now do

Cloudflare classifies bots by behaviour and offers three independent controls: Search, Training and Agent. Training now has four settings. Allow lets every crawler in. Disallow AI Training publishes a Disallow for the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI, and keeps mixed-use crawlers from labelled companies in for search. Block on pages with ads blocks every training crawler, including mixed-use ones, on pages that carry an ad. Block stops all training crawlers, including for search, which is the setting Cloudflare says now cuts a site off from Google, Apple and Bing entirely.

The Accountable label decides which crawlers stay in

Cloudflare has introduced a designation it calls Accountable and applied it to crawlers whose operators meet or commit to four requirements: a robots.txt opt-out for training, an opt-out for AI summaries, URL-level visibility into what was made available for training, and an assurance that opting out will not affect search ranking. Cloudflare says Apple, Google and Microsoft all qualify. Amazon, Anthropic, Meta and OpenAI qualify too, because they run separate search and training crawlers rather than a mixed-use one, so their training crawlers are blocked under Disallow AI Training without touching search.

How the opt-out reaches each engine

At Google, Disallow AI Training publishes a Disallow for Google-Extended, the robots.txt token Google offers for opting out of Gemini training. Google's own documentation says Google-Extended does not affect inclusion in Google Search or ranking. AI Overviews and AI Mode are controlled separately in Search Console.

At Apple, the setting publishes a Disallow for Applebot-Extended. Apple documents this as not affecting search ranking, and says keeping content out of Siri and Search AI answers takes the nosnippet meta tag on top of it.

Bing does not yet honour a robots.txt no-training preference, so Disallow AI Training sends nothing to Microsoft for now. Cloudflare says Microsoft's support for the directive is targeted for early 2027. In the meantime the training opt-out on Bing is the NOARCHIVE meta tag, which also removes links from Chat and Copilot.

Cloudflare says the next piece it is working on is a single AI-summaries control, so a site owner can set one preference for how much of its content ends up in an AI answer instead of tuning it per operator.

Source

• Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training, 15 September 2026.
• Search Engine Journal, Cloudflare Lets Sites Disallow AI Training Without Blocking Googlebot, by Matt G. Southern.