Cloudflare feeds OpenAI's crawler signals so ChatGPT finds more pages

Most people who publish on the web know that a lot of their best pages are hard to reach by following links alone. A product manual for one model, a passport fee for one country, a menu for one restaurant: useful, specific pages that few other sites bother to link to. Cloudflare and OpenAI now say they have been running a pilot that gets ChatGPT to those pages anyway.

Cloudflare published the post with OpenAI as a joint announcement, signed by Matthew Conroy and Jack Galilee at Cloudflare and Henry Aspegren and James Balfour at OpenAI. They say the pilot ran on "a participating cohort of ~1% of Cloudflare's total network" and "has enabled OpenAI to discover and index hundreds of millions of new web pages". No breakdown of that total is given by site, country or language.

The four kinds of signal

The pilot sends OpenAI's crawler four kinds of signal from Cloudflare's own network. Discovery points at pages the crawler had not seen, including ones no popular site links to. Filtering keeps the crawler away from addresses that do not exist or redirect to the homepage, which Cloudflare says is a common pattern in bot traffic. Change tells the crawler when a page is worth another visit and when it is not, so the same unchanged page is not fetched again and again. Safety flags a single compromised page, such as a fake login form on an otherwise legitimate site, without blocking the whole domain.

OpenAI still decides whether a discovered page is good enough to rank in ChatGPT's results, the post says: "OpenAI then independently evaluates and ranks how those pages appear based on how relevant and helpful they are."

Metadata only, with the owner's settings on top

The post states one limit plainly: "This research uses metadata about the web for search and discovery, rather than sharing page content for model training." The signals describe pages, not their contents, and Cloudflare says a signal about a page does not grant OpenAI permission to fetch it. The domain owner's own crawler settings still apply.

Cloudflare points site owners who want to control this at AI Crawl Control, its existing dashboard for picking which AI systems may read which parts of a site. The feature itself did not change with this announcement. The post describes what signals leave Cloudflare's edge, and the owner's existing settings in that dashboard decide which crawlers may act on them.

What is still unmeasured

The post is short on numbers beyond the two headline ones. It does not say how many of the hundreds of millions of pages ChatGPT ended up citing, how many search queries received better answers because of the pilot, or how often the "change" signal correctly caught a page that had been updated. Cloudflare says the next phase will "improve the selection of pages, test faster and more reliable change signals, and measure what happens after content is discovered".

For a site that wants to be reached from a ChatGPT answer, the practical point is that writing a page to answer one visitor's specific question is still the main thing a site owner can do, since the pilot is designed to find exactly those pages. For a site that would rather not be read by ChatGPT at all, nothing changes: the signals never override the owner's existing crawl settings, and AI Crawl Control is where those settings live. For a team already paying for Cloudflare, this is also an argument that the company treats crawl waste, from bots asking for files that do not exist, as a load problem worth filtering out at the network.

Source

Cloudflare and OpenAI: making AI discovery better for everyone, Matthew Conroy, Jack Galilee, Henry Aspegren and James Balfour, Cloudflare blog.