SEJ column shows the AI crawl-to-refer ratio for Anthropic has been quoted at seven different values in 13 months, all from Cloudflare

Duane Forrester, founder of UnboundAnswers.com, published a column in Search Engine Journal on 10 September tracing seven different values that have been given for Anthropic's crawl-to-refer ratio: 70,900 to one, 38,000 to one, 23,951 to one, 11,122 to one, 10,300 to one, 4,580 to one, and 2,237 to one. Every one is attributed to Cloudflare, every one was published inside about 13 months, and two of them claim the same month and differ by roughly a factor of 17.

Where the number came from

Cloudflare introduced the metric in July 2025: divide HTML requests from a platform's user agents by HTML requests whose Referer header names that platform, and normalise to one referral. The formula is unambiguous and reproducible from server logs. Cloudflare also, Forrester writes, published the parts of the metric that undercut it.

Four denominators wearing one coat

Forrester names four decisions hidden inside the single ratio.

The first is the window. In Cloudflare's launch post, the sample was 19 to 26 June 2025, one week, and Anthropic came out at 70,900 to one while Mistral came out at 0.1 to one, sending 10 referrals for every crawl request. In a separate Cloudflare post the same month, the June figure for Anthropic is 73,000 to one. Cloudflare later reported Google's ratio moving 19.4 percent in seven days because of a change in GoogleBot scheduling. Analysts choosing different windows in good faith produce different answers and both are right.

The second is which bot got counted. Cloudflare aggregates a platform's training crawler and its user-request crawler under one platform name. One consumes at scale and returns nothing by design; the other fetches on demand and can produce a citation. Rolled together, the figure describes neither, and the comparison against Google is structurally unsound because some operators run split crawler fleets and others do not.

The third is whose sites. Cloudflare sees the traffic that crosses Cloudflare, which is a large sample and still a sample. Forrester writes that when one analyst measured the same metric against Cloudflare's network and against a smaller commercial panel over the same period, one platform's ratio roughly doubled.

The fourth is the one Forrester says matters. A referral only enters the calculation if the request carried a Referer header naming the platform, and Cloudflare states directly that traffic referred by Claude's native app does not include that header, that they believe the same holds for other providers' native apps, and that the calculation may therefore overstate the ratios. Cloudflare's own words on the size of the distortion: it is unclear by how much.

What is riding on it

Publishers are using crawl-to-refer ratios to decide which AI crawlers to allow and which to block. Marketing teams are using the same figures to argue AI referral traffic is not worth pursuing. If a large share of a platform's referrals arrive through a native app sending no Referer header, the ratio overstates the imbalance by an unquantified amount. If the figure blends a training crawler with a user-request crawler, a block aimed at the training pass may cut off the one that could have cited you.

Forrester's test for any number in this space is to ask three things before acting on it: what period it covers, what got grouped to produce it, and where the collection stopped. A vendor that cannot answer those on request, he argues, is asking for trust in a number they have not defined.

Source

• Search Engine Journal: Everyone Is Quoting The Same Number And Getting Different Answers
• Cloudflare: AI Search Crawl-to-Refer Ratio on Radar