Google warns a new CDN speeds Googlebot up and overloads the origin

A team moves a site to a Content Delivery Network, expecting the CDN to shield the origin from traffic. Google says crawler load goes up for the first few days instead. Googlebot speeds up because the site now answers faster. A cold CDN cache then sends every new URL back to the origin server. A blanket bot rule flipped on in panic can drop pages out of Search.

Cherry Prommawin and Gary Illyes posted the warning on the Google Search Central Blog. It is the fifth and last episode of a video series the Google Search team call the Non-Canonical Cabinet. Search Engine Roundtable reported it the same day.

Why the crawl rate goes up

Google's crawlers measure how fast and how steadily a host answers. A slow host gets fewer requests. A fast host gets more. The number Google sets is called the crawl capacity limit, also called hostload. When DNS flips to a CDN edge, response times drop at once. To Google's crawlers, the site looks like it was given a hardware upgrade. The crawlers raise the request rate to match.

Why the origin still gets hit

A CDN only shields the origin on the second visit to a URL. On the first visit the edge has no copy of the page. It fetches from the origin, stores a copy, and then answers. That first visit is a cache miss. Human visitors cluster around the popular pages, which the cache warms fast. Google's crawlers work through the whole URL inventory. "Every uncached URL triggers a cache miss that passes straight through to your origin server," the post says.

The bot-protection trap

A worried admin who sees crawler traffic spiking will turn on blanket bot protection in the CDN dashboard. The CDN then answers Google's crawlers with a CAPTCHA. Googlebot cannot solve a CAPTCHA. The crawler sees a page it cannot read and backs off. "If left in place, our crawlers will back off," the post says. Pages, shopping listings and images "will begin to drop out of Search".

What Google says to do

The post lists three steps for the switch. First, exempt verified bots in the CDN and firewall. Most CDNs have a toggle for it. Google publishes the IP ranges it uses for crawling. Second, warm the cache before the switch. Fetch the highest-traffic and most-crawled URLs through the CDN at a controlled pace. That fills the edge in advance. Third, keep a temporary rate-limit rule ready. If the origin is still under load, return a 429 Too Many Requests or a 503 Service Unavailable. Google says keep it on for a few hours, or one to two days at most.

No number for how big the crawl spike is

Google states no request number. The post says low latency signals a healthy host. It does not say how much faster the crawl goes or for how long. The 429 and 503 window is the only number. Google writes it as "a few hours, or 1-2 days at most". A team watching a specific site has to read its own logs to see when the edge has warmed up.

Source

Primary source: Google Search Central Blog, Cherry Prommawin and Gary Illyes. Reporting: Search Engine Roundtable.