Google tells platforms to noindex shareable AI pages by default

Any team running a product that lets a user share an AI chat has made the same bet. Leave the share pages indexable to catch some search traffic, or hide them from Google.

Google updated its user-generated spam prevention guide on 26 October. The changelog entry on Google's own site is titled "blocking indexing of link-shared content". The change added one new section to the guide, called "Only index content meant for Search". Search Engine Journal's Matt G. Southern wrote up the change the same day.

What the new section says

Google writes that a service may let users create shareable pages, generated outputs or interactive tools. The team then has to decide whether that content is meant for search. The guide says the default should be noindex, or a block in robots.txt. That applies when a page is made for direct sharing, collaboration or utility.

Noindex is set with a robots meta tag or an X-Robots-Tag HTTP response header. Robots.txt blocks the crawl.

The trap in robots.txt

Google's own robots.txt introduction page is the one the new section links to. That page says a URL blocked in robots.txt can still be indexed if another site links to it. Google will not crawl the blocked content in that case. The URL and the anchor text of inbound links can still appear in search results, without a description.

Google's noindex page says the noindex rule only works when robots.txt is not blocking the page. Otherwise the crawler never sees the noindex rule in the first place.

The practical reading: robots.txt alone does not keep a URL out of the index. Noindex alone keeps the page out, as long as it is reachable to the crawler.

The Search Console checks

The guide also tells platforms to verify the site or subdomain in Search Console. That is how Google sends a manual action notice or a security alert. The guide adds that teams should watch the Page Indexing report and the Search performance report. The target is unexpected spikes in indexed pages, impressions or queries. These are spikes that do not match the platform's intent.

A new bullet suggests tracking creation and indexing at the platform level. Teams should set alerts for spikes in newly published public pages. The guide also names another signal: a small number of users making a large share of indexable URLs.

What already changed on one AI product

Search Engine Journal records one earlier observation that fits the new section. The writer had checked Claude's share path and found a Disallow directive in robots.txt. The share pages also returned a value of "none" for the X-Robots-Tag header. Google treats "none" as both noindex and nofollow in its documentation. The share pages were not supposed to be in search results.

Source

Primary source: Google: Prevent abuse of your hosting or redirect service. Reporting: Matt G. Southern, Search Engine Journal.