Git.kernel.org now spends 14 CPU cores rendering commits for AI scrapers, and legitimate users make up 2 percent of traffic

Konstantin Ryabitsev, who runs the Linux kernel's public git and mail infrastructure at kernel.org, published traffic numbers on 29 August that put a size on the "AI crawlers" problem for a site with real audience data. Simon Willison surfaced the post on 7 September through his weblog, which is how it reached most people outside the kernel community.

The scale of it

Ryabitsev writes that git.kernel.org runs across 5 geographically distributed nodes with 90 CPU cores in total, and that at any moment 14 to 16 of those cores are doing nothing but rendering git commits as HTML for scraper bots. That is about 20 percent of capacity, though he notes the graph is spikier than a flat 20 percent because the swarms arrive in waves.

The site now sees about 6 million daily requests demanding random commits. Two thirds of those still get blocked at the perimeter by the Anubis proof-of-work challenge. The remaining third solve the challenge and reach the main site. Ryabitsev's own estimate is that only around 2 percent of git.kernel.org's total traffic is legitimate.

He gives one reason the numbers look absurd: linux.git alone has about 1.48 million commits, and git.kernel.org carries about 922 forks of it, so a scraper walking every URL sees "several billion valid URLs" that resolve to 922 duplicates of the same commits. He describes this as "the stupidest possible way" to gather the data, because git clone would produce the same content in one request per repository.

Every defence has burned out in turn

The post reads as a timeline of defences that worked, then stopped working. User-agent bans stopped working when bots started forging plain browser strings. IP bans stopped working when the traffic moved to whole subnets, then to entire ASNs. Blocking Google Compute and similar cloud IPs was justified but did not last, because the crawlers moved to residential and mobile addresses served by what Ryabitsev calls "proxy SDK monetisation": SDKs bundled into consumer apps and smart TVs that resell their owner's internet connection. Each IP would make four or five requests and never appear again.

Anubis, the proof-of-work challenge kernel.org now runs, held the line at difficulty 4 for a few months. Bots solved it. Difficulty 5 held for a few more months and warmed users' phones for seconds at a time. Bots solved that too. As of the post, difficulty 5 is still stopping 66 percent of requests, but the other 34 percent are burning CPU to get through.

What this looks like from a smaller site

Ryabitsev is careful to say kernel.org is not overwhelmed today. The site is still snappy for real users, and the outages he sees are still caused by badly configured CI pipelines shallow-cloning stable.git from 20 nodes at once. What has changed is the cost baseline: one fifth of a 90-core budget is now permanent overhead paid to training crawlers.

A team running a public git host, a doc site, or an issue tracker on far less than 90 cores can read those numbers as a floor rather than a ceiling. Ryabitsev's own next step is to turn off features that generate crawlable URLs and gate off actions that are expensive to serve anonymously, which is the same trade the rest of the open web is being pushed toward.

Source

• Creepy crawlies, Konstantin Ryabitsev, 29 August 2026
• Simon Willison's link post, 7 September 2026