AI NewsProductivityReported

GitHub's monthly commits doubled to 2.9 billion, and the platform went down

GitHub's CTO reported a 7 hour 47 minute outage on 17 August and gave the reason: monthly commits doubled since April. GitHub added 3 million CPU cores. Nobody has added review capacity.

AI News

Editorial3 min read

LinkedInX

Why it mattersAI can now write more code whenever a company pays for more machines. Checking that a change works still needs more people, and this is the first public dataset large enough to show the gap.

GitHub's chief technology officer Vlad Fedorov published a postmortem on 20 August covering an outage on 17 August that lasted 7 hours and 47 minutes. It stopped github.com, authentication, Actions, the APIs, pull requests, issues and Copilot from working.

His explanation of the cause is one sentence and it is the reason this is worth reading: "Since April, monthly commits have grown from 1.4 billion to 2.9 billion."

What GitHub said

Neither the 17 August outage nor an earlier Actions failure on 6 August came from a code or configuration change. Both were capacity failures. A critical component in the Central US data centre failed to scale when traffic reached a new peak, and the problem spread to authentication.

Copilot took longest to recover, because errors there triggered a client-side retry loop that added traffic during the recovery.

The fix is the kind only one of the largest cloud companies can carry out. More than 3 million CPU cores added, 120 petabytes of high-speed storage, and an accelerated migration to Azure, which now carries roughly 58 percent of platform load and half of all Git operations, up from 12 percent in May.

Fedorov's own words: "If you were trying to ship software that day, we let you down."

The number the postmortem does not discuss

Arjun Iyer, writing in The New Stack on 29 August, made the argument that follows from this data. Note before reading further that his piece is a sponsored post, paid for by Signadot, a company that sells testing environments. Every GitHub figure above comes from GitHub's own postmortem, which is linked below and worth reading directly.

His point is valid whoever paid for it. Every one of those 2.9 billion commits carries an implicit claim: this change works and breaks nothing downstream. Almost nothing between writing the code and merging it tests that claim against a running system.

Commit volume used to track the number of developers, because commits came from people typing. It stopped tracking people. One developer running three agent sessions at once produces commits at a rate no hiring plan predicted.

GitHub's limiting step can be fixed with money. Cores and storage can be bought. Confidence that a change behaves correctly cannot, because the step that proves it still has to go through a shared staging environment or a full copy of the stack, and neither of those doubles in four months because a budget was approved.

Find the step that runs the code

Look at your own pull request count per engineer over the last two quarters. GitHub's curve is the aggregate of thousands of teams, so if yours has the same pattern, that is the normal case rather than a local problem.

Then ask which step in your pipeline actually runs the change against something real, and how many changes that step can handle in a day. That number is your true throughput, however much code AI is writing.

Static analysis, AI review triage and test selection all got faster this year. None of them runs the code.

Source

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project