Google, OpenAI, Microsoft and Anthropic agree on one AI opt-out signal

Telling the big AI labs not to train on a site has needed a different note for each lab. The Verge, citing Search Engine Roundtable, reported that OpenAI, Google, Microsoft and Anthropic now agree. All four will read one new signal in robots.txt. The signal is a short IETF draft called AIPREF. Kevin Kelley, a technical lead for standards at Anthropic, proposed it. A site owner will write one line. Each lab will read the same line.

The four companies announced the agreement together. Each one set its own start date. Microsoft said on its Bing blog that it will start honouring the preferences on 1 February 2027. The company will put a temporary setting in Bing Webmaster Tools from 15 December 2026 as a stopgap. Google, OpenAI and Anthropic have not yet given a date.

What the signal looks like

The one line to add is user-agent: __usage_ai-training in robots.txt, followed by Disallow: /. The Verge quoted the proposal: publishers can "apply their AI training preferences across multiple crawlers" with one entry. The signal sits beside the lines that control search crawlers. A site that already uses a company specific block, like Google-Extended, can keep it. Both will be read.

Google's John Mueller gave an example. A site may use the new policy token to opt out of training. Google will then read it as a Disallow on Google-Extended, Mueller said. A site that already blocks Google-Extended keeps that behaviour. A site that only writes the new line gets the same effect at Google.

Microsoft breaks the choice into two parts

Bing's version of the control lets a site owner choose two things separately. One is whether Bing may train on the site. The other is whether Bing may quote the site in its AI answers. Microsoft calls the quoting part grounding. The two settings are separate. A site can allow grounding and block training. Microsoft also added a "limited grounding" option. Under that option, Microsoft's own products may still quote the site, but every third party service is blocked.

Krishna Madhavan is a Principal Product Manager at Bing. He said the new controls are simpler and clearer. They let a site owner choose about training, grounding, or both. The site stays eligible for Bing search results either way.

What this means for a site owner is one short task. Add the AIPREF line now. Keep the per-company blocks that are already set. Each lab will turn the new line on at its own date next year. A team that lives on AI answer traffic needs to watch the training versus grounding split in Bing. Blocking training while allowing grounding is the only way to stop training and stay visible in AI answers.

The old company by company system was a treadmill. A new AI company appeared, you added another User-agent block, and your robots.txt grew. One shared signal is a smaller surface to maintain. It only works if the labs honour it on the schedule they named.

Source

The Verge: The biggest names in AI are backing a new way to opt out of scraping, by Emma Roth. Search Engine Roundtable: Google Adopts New AI Training Control and Microsoft Bing New AI Controls, by Barry Schwartz.