Google says moving to a CDN can get your site deindexed

A CDN is meant to make a site faster. Google now says the switch can take the site out of Search. Cherry Prommawin and Gary Illyes posted the guidance on the Search Central blog on 11 November.

Google's point is that a faster origin tells crawlers to crawl more, not less. The new CDN starts with an empty cache. Googlebot then asks for every URL on the site. Every miss hits the origin server. The origin is now serving more traffic than it did before the switch.

What the firewall does

The second half of the problem is the CDN's bot protection. Many CDNs show a CAPTCHA to anything that looks like a bot. Googlebot cannot solve a CAPTCHA. Google says a crawler stops asking for a page it keeps getting a challenge on. The URL then falls out of the index.

The post names this as the mistake to avoid. Google writes: "Allow verified bots in your CDN and firewall settings." Most CDNs now carry a one-click toggle for verified search bots. The site owner has to turn it on. Generic bot protection without it is what sends Googlebot a CAPTCHA.

Warm the cache before the switch

Google gives two steps to protect the origin during the move. First, fetch the site's busy URLs and its most-crawled URLs through the CDN before pointing the DNS over. Those pages are then in the CDN's cache when live traffic arrives. Google calls this pre-warming.

Second, if the origin is still under too much load, return a 429 or a 503. Hold that for hours, or at most a day or two. Both codes tell Googlebot to slow down and come back. Googlebot already handles that signal. A CAPTCHA is not a signal it reads.

What a site owner does on Monday

A team releasing a CDN switch now has a short checklist from Google itself. Turn on the verified-bot allow list in the CDN before the cutover. Pre-warm the pages that get the most traffic and the most crawl. If the origin still struggles, use a 429 or a 503 for a day, not a block. Do not let the CDN serve a CAPTCHA to search crawlers. Google's own advice is the citation if a security team pushes back on the allow list.

Source

Moving to a CDN without overwhelming your origin (or blocking Googlebot), Google Search Central, by Cherry Prommawin and Gary Illyes.