Google's Illyes says one robots.txt rule cannot stop AI training

A single rule in robots.txt will not stop an AI company from training on your pages. Google's Gary Illyes said this plainly in a Search Central video on 22 October. The standard, RFC 9309, decides which URLs a crawler may request. It says nothing about what the crawler does with the content later. To keep one AI crawler out, Illyes said, the file has to name that crawler's user-agent token.

Search Engine Journal reports the exchange from the second episode of a Google show called The Non-Canonical Cabinet. Illyes and Cherry Prommawin of Google Search Relations answered a reader question. The question asked whether robots.txt can allow AI crawling for search and block it for training on the same pages.

What Illyes said

Illyes said RFC 9309 is "an access preference protocol". The standard, in his words, "doesn't dictate what a system does with the data after it has already crawled it". The episode showed a made-up directive on screen, No-AI-Chicken-Training. Illyes gave it as an example of a line a parser would ignore.

Search Engine Journal notes that Google's own robots.txt page lists only four supported elements: User-Agent, Allow, Disallow, and Sitemap. Any other line is dropped. RFC 9309 itself, in Section 2.2.4, lets a crawler decide how it reads a record outside the four basic elements. So the only way to block an AI crawler is to list its user-agent and set Disallow for it.

Why Google-Extended is the exception

Google-Extended is a robots.txt token with no crawler of its own. Google crawls under its usual user agents. The token controls whether Google can use that crawled content to train Gemini. It also covers grounding in Gemini apps and in Grounding with Google Search on Vertex AI. The token has effect only because Google documents it as a usage control.

Illyes said the token does not affect a site's place in Google Search and is not a ranking signal. The meaning comes from Google's own product documentation, not from the RFC. Other AI companies pick their own names or ignore the hint.

The draft that would change this

Illyes co-wrote an IETF Internet-Draft, with Mozilla's Martin Thomson, that would update RFC 9309. The draft, dated 19 August, adds a Content-Usage rule to robots.txt and a matching HTTP header. The example in the draft includes "train-ai=n". That line opts out of AI training on paths the crawler is already allowed to fetch. Allow and Disallow would still decide which URLs may be crawled.

The draft is a Working Group document. Search Engine Journal reports the group had not sent it to the IETF Steering Group as of 22 October. The vocabulary draft, dated 14 September, states that the "train-ai" label has no consensus in the group yet.

Prommawin added a separate point for non-English paths: Google treats raw UTF-8 and percent-encoded paths the same way, but other crawlers may not. She recommended percent-encoding non-English paths in robots.txt rules.

What a site owner has to do today

To block AI training on your site, you need a list of the AI crawlers you want to stop. For each one, you need its user-agent token. For Gemini, add Google-Extended. For other models, read the vendor's own documentation and add each token. There is no single rule that blocks them all.

A page you let Google crawl can still answer a Gemini question unless Google-Extended is also set. For any site that does not want that outcome, the user-agent list is the only tool the standard provides today.

Source

Primary source: The Non-Canonical Cabinet episode two with Gary Illyes and Cherry Prommawin.

Reporting: Google Explains The Limit Of Robots.txt For AI Content Use by Matt G. Southern on Search Engine Journal.