Cloudflare will score verified bots on robots.txt and 429 obedience

A site that lets a named AI crawler through its bot defences has, until now, been trusting a label the crawler was issued once. Cloudflare said that is going to change: every bot on its Verified Bots list will carry an explainable trust band that moves based on what the bot actually does, starting with two signals the operators of large AI crawlers already know about.

Cloudflare framed the shift in a blog post titled "Continuous accountability for Verified bots". The post compares the current state to a driver's licence that records somebody passed a test once, with no record of how they drive next week. A verified bot, Cloudflare notes, can still ignore site preferences, overwhelm origins or evade controls, sometimes through a bug and sometimes on purpose.

The two signals

The first signal is whether the bot's traffic is consistent with the site's robots.txt over time. The post notes that robots.txt is the main way a site communicates what it would prefer crawlers avoid, and that Cloudflare will watch for traffic that keeps hitting paths the file names. The company adds a caveat: a robots.txt sets a floor on behaviour, and a site may still use other signals to tell a bot it is no longer welcome in areas the file never listed.

The second signal is how a bot responds to an HTTP 429 Too Many Requests response. A 429 is an instruction to slow down, often with a retry hint, and Cloudflare says its system will check whether a bot's request rate actually falls after one. Repeated 429s followed by the same pace lower the band; consistent back-off raises it.

The trust band

The output is a range describing how consistently a bot has respected site preferences, with the observed behaviours that informed it. Cloudflare calls this a trust band and frames it as explainable: a site owner can see which signals moved the score and when, with reasons attached. The post does not state the numeric range, how often the band updates, or what the thresholds are for a verified bot to lose its status.

What is not yet in the post

Three things a site owner would want to know are not in the announcement. First, how a bot's drop in band translates into enforcement, if at all: whether a low band produces a visible warning, a rate limit, or removal from the Verified Bots list. Second, how a site that wants a stricter rule than Cloudflare's judgement can apply it. Third, when the trust-band readout will appear in the Cloudflare dashboard or the public Verified Bots page. The post describes the signals and the measurement as a first step and leaves the enforcement surface for later.

For a team that already runs behind Cloudflare and allows named AI crawlers through, the practical change is that the allow decision now has an evidence trail attached. For an operator of a large AI crawler, the practical change is that ignoring a 429 or a disallowed path now goes on a record the destination can read. The scores to watch are what the band says about ChatGPT, Claude, Perplexity, Gemini and the Google-Extended crawler, because publishers have been asking for that readout and so far could only guess at it from their own logs.

Source

Chris Tsoukaladelis, "Continuous accountability for Verified bots", Cloudflare.