GitHub rebuilds Git storage to handle 7,200 pushes a second to one repo

A team that runs ten coding agents on one repository has felt the push queue back up. GitHub says its old storage could only take a few hundred writes a second on one repo. That is why the queue builds.

GitHub rebuilt that part of the system. The new design is called Project Rubicon. In an internal test, GitHub ran 5,000 agents on one repository. They pushed at 7,200 pushes a second. The post was written by Karthik Raman and Brian Celenza. Both are GitHub engineers who worked on the rebuild.

What changed in the storage

Git on GitHub used to keep a repo's data on the same machines that served its reads and writes. A busy repo ran out of that machine before it ran out of disk. Project Rubicon splits the three jobs across three systems. A new service called Ref-CAS handles reference updates. A reference update is what says which commit a branch points at. Azure Blob Storage holds the Git objects themselves. A pool of what GitHub calls virtual replicas runs the reads and writes. Each virtual replica is throwaway and caches to local disk. GitHub can add more virtual replicas without moving any data between machines. GitHub also says cleanup jobs like garbage collection now run on their own nodes. They no longer steal time from a request a developer or agent is waiting on.

The numbers GitHub gives

GitHub says one customer repository went from 44 million to 181 million requests a day. That is a 4x rise over two months. GitHub says Ref-CAS runs tens of thousands of reference operations a second. An early prototype of the new design only reached a few hundred writes a second. The 5,000 agents and 7,200 pushes a second are from GitHub's own internal testing on one repository. No outside benchmark is cited. No customer name is given.

Where it is now

Project Rubicon is in a limited private preview. GitHub says the broader rollout is "early next year". The post does not date it more tightly. The post names no price. It says no plan for teams on the free or team tiers.

A team running many agents on one codebase has had Git itself as the write limit. If GitHub's figure holds up on a real repo with real agents, the limit moves. It moves back to the agents themselves. It also moves to the review step and the merge queue. The team controls each of those.

Source

Inside GitHub's new Git storage architecture, by Karthik Raman and Brian Celenza on The GitHub Blog.