Google's search team restates how to block AI crawlers in robots.txt

Barry Schwartz at Search Engine Roundtable published a summary of a Google Search Off the Record podcast episode in which Gary Illyes and Cherry Prommawin went back over how robots.txt should be written for the current mix of search and AI crawlers. The guidance was already in Google's documentation. The podcast makes four points that site owners still get wrong.

The scope robots.txt actually covers

Illyes and Prommawin describe robots.txt as an access-preference protocol. It tells a crawler whether to fetch a page, and that is all. It gives the site owner no say over what happens to the content after it has been fetched. Schwartz quotes the guidance that a site owner wanting to block a specific AI crawler has to "target its specific user-agent token explicitly in the file." A general rule like a wildcard for all user agents misses crawlers that do not follow the common conventions, and crawlers can and do choose which rules to honour.

Emojis and international characters

The episode spends time on URL paths that contain emojis or non-English letters. Google's parser handles the auto-encoding step on its own, so a path written with those characters will usually match. The guidance for a safer file is to percent-encode emojis and non-English characters in the robots.txt rules directly, because other crawlers do not all agree on how to normalise the raw characters. The site owner should look at a readable file and remember that the readability does not travel.

The standard behind it

The piece names RFC 9309 as the standard the robots.txt file should follow. The practical implication: validate the file with a parser that checks syntax, rather than relying on how it looks in a browser. A rule that reads sensibly to a human can still be a line that one crawler accepts, one ignores, and one treats as a sitewide block, depending on how each parses whitespace and comments.

What a team should do with this

Three edits most sites still need, in order.

First, name the user-agent tokens for the specific AI crawlers the team wants blocked or allowed, rather than a single User-agent: * rule. Google-Extended, GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended and others have their own tokens, and treating them as one group has already produced sites that mean to say yes to search and no to training and end up saying no to both or neither.

Second, percent-encode every emoji and non-English character in a disallow path, even on a site that only serves English content, because sitemap entries, product slugs and campaign URLs often creep in.

Third, run the file through a validator that implements RFC 9309, rather than reading it aloud. The second is where the surprises come from.

For any team setting content policy with answer engines in mind, this is the current statement of what the robots.txt file actually controls.

Source

Search Engine Roundtable: Google on robots.txt rules for AI crawlers, emojis, international URLs and more, by Barry Schwartz, summarising the Google Search Off the Record podcast with Gary Illyes and Cherry Prommawin.