Google plans one robots.txt token for AI training across many providers

A site that opts out of AI training sets a separate rule for each company today. Google plans to support one shared robots.txt token for many providers at once. The signal comes from the AIPREF working group at the Internet Engineering Task Force. John Mueller, Google Search Advocate, posted the news on the Google Search Central Blog on 30 October 2026.

Google-Extended, the training control Google has offered since 2023, will keep working. The shared token is a new line that works for every AI company that supports it.

How the two signals fit together

Google wrote the key rule as a quote. A site can set only the new policy token to disallow training. If it does not also set Google-Extended, Google reads that as if the site had disallowed Google-Extended. The new token alone is enough. A site that sets only the shared token has opted Google out of training.

A site that still wants Google but not others can mix the two. Set Google-Extended to allow. Then set the shared token to disallow. Other AI companies read the second line.

Google also tied one search outcome to the training signal. A site that allows Google-Extended is eligible to appear in Gemini App answers and receive clicks. Blocking Google-Extended keeps the site out of those answers.

Where this comes from

Mueller wrote that the shared token is based on work from the AIPREF group at the IETF. The group is defining a shared way for websites to say how machines may use their content. The training control is the first part. Google said it will keep working with the group on the rest.

Mueller also linked the proposal to European rules. He said it is consistent with the EU Code of Practice for General Purpose AI Model Providers. That code sits under the EU AI Act. It names robots.txt as a primary way to express preferences a machine can read.

What is not yet there

Google did not publish the exact token in the blog post. The AIPREF draft is still open. Google said it aims to support the new control, along with other AI providers, in 2027. There is no sample robots.txt line in the announcement, and no date for a Search Console setting.

The existing Google-Extended user-agent is unchanged. Any current block or allow in robots.txt still applies today. The shared token is coming on top of that. The two will be read together.

For a team that runs the serving side, the useful preparation now is a robots.txt review. The site owner should know which AI companies the site allows today, and why. The new line will apply to all of them at once when it is live.

Source

Supporting a shared AI training control for websites, by John Mueller, Google Search Central Blog.