Cloudflare lets sites block AI training while staying in search

Cloudflare published a new setting today called Disallow AI Training, which publishes robots.txt directives that refuse AI model training while leaving search crawling in place. Alongside the setting, Cloudflare named Apple, Google and Microsoft as Accountable crawler operators, meaning their AI bots honour the split.

What the setting does

The setting publishes per-bot directives in robots.txt, aimed at the identifiers those companies already use for training crawlers: Applebot-Extended, Google-Extended, and the equivalents from Microsoft. A site owner who turns it on continues to be crawled by Googlebot, Bingbot and Applebot for search, and stops being crawled by Google-Extended and Applebot-Extended for model training. Cloudflare says the three behaviours, search, training and agent access, are now three separate controls on the same domain rather than one block-everything switch.

The Accountable designation

To qualify as an Accountable operator, Cloudflare says a crawler owner must meet four requirements: honour training opt-outs, honour opt-outs for AI summaries, provide URL-level visibility into what was used for training, and confirm that refusing training will not hurt search ranking on the operator's own surface. Apple, Google and Microsoft qualified. The Cloudflare post notes that each combines capabilities available today with time-bound commitments for parts still in development, so the designation partly rests on promises about the future.

The number that made this necessary

Cloudflare says 17% of sites now enable some mechanism to block training, while fewer than 1% block search bots. That one figure explains why separate controls are needed: a lot of sites want to refuse training and almost none want to refuse search, and until now the practical choice on many platforms was refuse both or refuse neither. New domains on Cloudflare get preset defaults based on how the site makes money, and ad-supported sites default to Disallow AI Training with search left open.

Why this changes things

Refusing training used to cost search visibility on Google specifically. Google-Extended was framed as decoupling training from search, but many publishers still blocked everything because they did not trust the split to work and did not want to test it against ranking. Getting Google itself onto the Accountable list, on Cloudflare's terms and with Cloudflare's audit of the four requirements, turns the split from a promise on a docs page into a claim that a large company between sites and crawlers is willing to state and enforce with directives.

For anyone whose GEO strategy has been to be quotable by ChatGPT, Perplexity and Google AI Overviews while refusing to be fine-tuned on, this is the first day both positions can be held at once on Cloudflare. Set the training opt-out, leave search on, and the retrieval crawlers still index the site so it can be cited when a live query arrives. The Cloudflare post says the setting is available now to all sites on the platform.

Source

Source: Cloudflare, 15 September 2026.