Google's Mueller says AI crawlers read his sitemap.xml and RSS files

An AI training crawler has nowhere to submit a sitemap. Google's John Mueller said on the 1 October episode of "Search Off the Record" that he sees them finding his anyway, by reading sitemap.xml and his RSS feeds straight out of his server logs, and that this is the discovery path a publisher can actually plan for.

Mueller's quote, as reported by Search Engine Journal on 5 October, is that he has seen it happen in his server logs where an AI crawler accesses his sitemap file and his RSS files. He did not name the crawlers or say what the companies behind them do with the files.

The one thing a publisher does differently

Mueller's recommendation is short. Either stick with the generic name sitemap.xml, or publish an RSS feed, because those are the two filenames an AI crawler will try without being told. His reasoning is that AI training crawlers "usually don't have any kind of a Console or any setup where you can submit a sitemap file", which is the submission path a search crawler offers. The file has to sit where the crawler already looks.

For a site that uses a custom sitemap filename, this is a direct change: either add a sitemap.xml that redirects or holds an index of the real sitemaps, or add an RSS feed. For a site that already uses the generic name, nothing changes.

What it does not say

Mueller did not quantify how often the AI crawlers he sees come back. He did not list the crawlers. He did not say the AI companies then use what they read for training, retrieval, or anything else. The log line is that the file was requested, and that is all his quote supports.

He also clarified, in the same episode, that llms.txt cannot replace an XML sitemap because it has no strict format. A sitemap gives a crawler a date and a change frequency per URL. An llms.txt is a list of links in a text file, with no structure a machine can rely on.

The last practical note from the episode is about the "couldn't fetch" status that a valid sitemap sometimes shows in Search Console. Mueller said this is either host-load pressure or Google reading the site as low enough quality that it does not want to spend crawl budget on it.

Source

Search Engine Journal, 5 October 2026, reporting the "Search Off the Record" podcast episode of 1 October 2026 with Martin Splitt, titled "Do sitemaps still matter?".