Google's Gary Illyes blames AI scraping for the rise in Googlebot impersonation

A rogue bot can call itself Googlebot in about 30 seconds, and more of them are doing it. Gary Illyes from Google said this on the Odys Podcast with host Aneta Samkoff. Search Engine Roundtable's Barry Schwartz covered the interview on 30 October. Illyes said AI companies looking for training data are the main reason bot impersonation has become a bigger problem.

The point is practical for anyone running a site. A visitor claiming to be Googlebot is not Googlebot unless its IP address and reverse DNS record match Google's own. Illyes said he has set up honeypots that attract crawlers. Many requests claiming to be Googlebot come from addresses with no link to Google.

Why AI changed the mix

Illyes framed the shift in Schwartz's write-up. Companies want tokens to train models, and that needs content. The internet is treated as a farm to harvest. Firewalls and CDNs have improved at spotting fake crawlers, but they still miss some. Smaller AI engines and scrapers can borrow Googlebot's name because publishers have a reason to let Googlebot through.

The owner of a smaller crawler would be blocked. The owner of a Googlebot-named crawler, often, is not. That is the gain for the person doing the impersonation, Illyes said.

What a site owner can check

Schwartz lists the key points from the interview. A Googlebot user-agent header alone is not proof. Reverse DNS verification and the published IP ranges are the proof. Google documents both.

Illyes also said robots.txt is a suggestion, not a security control. Honest crawlers respect it. Dishonest ones do not. A site that needs to block traffic should use network protections such as a web application firewall. A line in robots.txt is not enough.

Fake crawlers can also take a site offline. Google has spent years tuning its own crawling to avoid exhausting a site's resources. Others have not. Illyes said much of the damage today may be ignorance or poor engineering rather than deliberate attack.

The signal and training split

Google already lets site owners separate the two. Googlebot can crawl pages for Search while the Google-Extended token in robots.txt limits what Google uses for training. Apple runs Applebot-Extended the same way. Illyes expects access controls for automated clients to become more granular over time.

Nothing in the interview tells a site owner to block Googlebot. The call is to verify the identity of anything that says it is Googlebot. The training tokens matter where the choice does.

Cryptographic signatures as the next step

Illyes also discussed Web Bot Auth. The idea is cryptographically signed HTTP requests. A site would not need to keep an IP range list current. He described the work as an effort, not a finished standard. For now, reverse DNS and the published IP ranges are what a site has.

What this means for GEO and crawl budget

Teams tracking generative engine traffic read access logs for a reason. A site that mistakes a scraper for Googlebot sees inflated signals. Those signals look like search crawl. Real search crawl can be mixed with harvesting that never earns a citation. The split only matters if the site can tell the two apart. Illyes said the way is reverse DNS and the IP ranges, applied before analytics.

For a team building the serving side, the practical change is small and specific. Any check that whitelists on user-agent alone is giving a free pass to a growing class of scrapers.

Source

Google's Gary Illyes: AI Made Bot Impersonation Worse, by Barry Schwartz, Search Engine Roundtable. The source interview is the Odys Podcast with Aneta Samkoff.