Cloudflare and OpenAI say ChatGPT indexed hundreds of millions of new pages

Cloudflare and OpenAI shared early results from a research pilot. OpenAI says it discovered and indexed hundreds of millions of new web pages using signals Cloudflare provided about which pages are worth fetching. The pilot ran across a cohort of about one percent of Cloudflare's total network. The blog post was published on 15 October and was co-authored by two Cloudflare engineers and two OpenAI engineers.

Cloudflare says the signals describe which URLs lead to a real page, which lead to redirects or copies, which pages have changed, and which pages are "potentially harmful" and should be kept out of an index. OpenAI then ranks the discovered pages on its own. In Cloudflare's words: "A signal about a page doesn't grant permission to access it. This research uses metadata about the web for search and discovery, rather than sharing page content for model training."

What the "about 1 percent" cohort actually did

Cloudflare says the cohort was about one percent of its network, which it describes as a "participating" set of sites rather than the full population. The pilot used that cohort to test whether network signals could help ChatGPT's search crawler reach pages that were previously unknown to it. Cloudflare describes the gap this way: "A legitimate site can have a single compromised page, such as a fake login form planted by an attacker. Flagging that one page, rather than the whole domain, keeps people safe without penalizing the site owner." That is a different thing from blocking a site at a robots.txt level.

Two mechanics mattered for site owners. Filtering: Cloudflare says bots repeatedly fetch files that are not there, and some sites redirect almost any address to the homepage. Cutting those out before the crawler fetches the page reduces wasted work for both sides. Change signals: telling a crawler when a page has changed and when it has not reduces repeated downloads of unchanged pages. Cloudflare says new discovery can increase crawling at first, as new pages are evaluated and indexed, before total load falls.

What a site owner still controls

The post is explicit that domain owners' choices decide what happens. Cloudflare points at its "AI Crawl Control" dashboard as the place to understand and manage how AI systems access a site. That is where to set, per bot and per purpose, which crawlers may read the site at all. Cloudflare's own writeup separates search from training on purpose, because the robots.txt file alone does not do that split.

Two groups of readers can act on this. The first has been writing deep, specific pages that no other site has, and hoping an AI answer engine finds them. For them, the pilot is a reason to keep going: pages with few inbound links are reachable to ChatGPT in a way they were not three months ago. The second group has given up on AI referral traffic and set robots.txt to block AI crawlers across the board. For them, this result is a reason to split the setting: allow the OpenAI search crawler explicitly, and keep the training-only crawlers blocked, so a specific page can show up as a cited source without the full site being used for model training. The content on the page still has to be distinct enough to be worth citing, which is the one part of the result the Cloudflare signals cannot change.

Source

Cloudflare and OpenAI: making AI discovery better for everyone, Cloudflare

ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot, Search Engine Journal