Google documents the Retry-After header for cutting Googlebot's crawl rate

The documentation page every site operator ends up reading at 2 am now names a specific header to add to the 503 Google already respects. Google updated its reduce-crawl-rate documentation and added explicit guidance on the Retry-After HTTP header for the urgent case, pointing site owners at a single line that tells Googlebot when to come back.

Barry Schwartz at Search Engine Roundtable flagged the update. Most of the changes are organisational, moving existing content around on the page, but the new paragraph on Retry-After is the one that matters to anyone handling a live incident.

What is actually new

Google's documentation at developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate has long recognised three status codes, 500, 503 and 429, as reasons for Googlebot to cut back its crawl rate automatically. That is the mechanism, and nothing about it has changed.

What is new is the explicit mention of the Retry-After HTTP header, defined in RFC 9110 HTTP Semantics. A site can include this header on the same 503 or 429 response, and it accepts two formats: a delta-seconds integer, meaning "come back after this many seconds," or an absolute UTC date, meaning "come back at this time." Google's documentation now shows a short code example for each.

Google already honoured the header where crawlers saw it; the update makes it a documented option, which matters because an incident runbook now has a line of HTTP it can point at by name.

When this helps, and when it does not

The automatic signals, a 503 under load without a header, still work and are still the first response you reach for. Retry-After is for the urgent case where the operator knows when the spike will clear and wants Googlebot to come back at a specific moment, rather than letting it fall into the exponential backoff schedule. A planned maintenance window, a load-shedding plan triggered by a dashboard, or a deployment that will complete in a known number of minutes are the cases where Retry-After is tighter than the default.

A few constraints worth remembering. The header is advisory, which Google's documentation on crawl-rate reduction states plainly. Googlebot may ignore an absolute date that is far in the future. And the sustained-error reduction, triggered by long periods of 5xx responses, is separate from Retry-After: Google warns that if a site keeps returning errors, the crawl rate can stay reduced for months after the errors stop. Retry-After does not reset that curve. Fixing the errors does.

For a working engineer, the pattern is familiar. If you already serve a 503 under load, add a Retry-After line to the response with the number of seconds you expect the incident to clear. If you run scheduled maintenance, add Retry-After with the delta to the end of the window. The signal is now in the docs by name, which is useful mostly because you can point a colleague at it instead of a Twitter thread.

Source

Primary: Reduce the Googlebot crawl rate, Google Search Central developer docs. Press: Google Search Crawl Rate Doc Adds Retry-After HTTP & More, Search Engine Roundtable.