Four AI firms adopt one robots.txt signal for AI training opt-out

Managing opt-outs by vendor was workable with four crawlers in robots.txt. It stopped scaling at twenty. This week four of the biggest AI companies said they will all honor the same line.

Microsoft said this week that from 1 February 2027 it will honor a shared robots.txt User-Agent called __usage_ai-training. Google, OpenAI and Anthropic have made parallel commitments. The token comes from a recent IETF Internet-Draft.

What the token does

A site owner adds one block to robots.txt. The User-Agent line reads __usage_ai-training. A Disallow: / under it tells every participating AI provider not to use the site for training. A path can be disallowed instead, so part of a site stays out while the rest is open.

The token is a shared signal, not a user agent for a crawler. The draft is draft-kelley-aipref-training-preference-signal, from the IETF AI Preferences working group. Microsoft says the token "gives web publishers a simpler, common way to express training preferences in robots.txt."

How vendors plan to implement it

Microsoft will honor the token on 1 February 2027. It is also adding a separate AI Grounding preference. That control lets a publisher opt out of being quoted in AI answers. Bing search indexing is unaffected. Microsoft says the two choices sit alongside each other.

Google plans to support it in early 2027. John Mueller, Google Search Advocate, said in a comment that the token acts as a Google-Extended disallow. That applies when a site uses the token to opt out of training. It also requires that Google-Extended is not explicitly set in the file. Google's crawlers page now says: "Google intends to support the AIPREF shared machine-readable training preference signal (with 'user-agent: __usage_ai-training') in early 2027."

Google added that existing per-vendor controls, like Google-Extended, will keep working alongside the shared signal. A site owner can set a broad choice with the industry signal and still name individual vendors.

What a site owner should know now

The token has no effect yet. Nothing in a current robots.txt file changes. The signal is a work in progress at the IETF, and the writing group acknowledges open points. Nate Hake, a working group participant, said one open point remains. The question is whether the preferences apply to things a user does with content. The decision affects AI-driven copying of creators.

The practical move is to track when each named vendor turns on honoring. Microsoft is first with a date. Google plans the first quarter of 2027. OpenAI and Anthropic have joined the commitment and have not published a date. Per-vendor opt-outs through tokens like Google-Extended, GPTBot and ClaudeBot still work. They are still needed, since not every AI crawler is part of this group.

Source

Microsoft's post, Your content, your choice: clearer preferences for training and grounding, and the IETF draft, Machine-Readable Training Preference Signal. Google's parallel adoption as reported by Search Engine Roundtable.