OpenAI replaced its Git server to keep up with its own agents

The quiet cost of coding agents is that they commit a lot more than people do. OpenAI said on Monday that its own quarterly commits are up by over 2,400 percent since early 2024. The company's Git server could not keep up. So OpenAI built a new backend called Pando.

The New Stack's Frederic Lardinois reports on the engineering post by OpenAI's Aaron Friel, Taylor Blau and Ted Nyman. Pando serves close to a petabyte of Git data a day inside OpenAI. It cut push latency by about five times. The team wrote it in Rust in ten weeks. One engineer designed the core. Teams of Codex agents wrote the rest. Pando has been running in production since April.

What broke before Pando

OpenAI watched its Git reliability drop below 90 percent. Fetches got slow. Merges failed. The engineers asked a question. Was Git itself the limit, or was the way they served their monorepo the limit? They concluded that the Git client and the protocol were mostly fine. The way standard Git servers are built was the real problem.

A standard Git server keeps every branch consistent across the whole repository on every write. A push to one agent's branch can wait on an unrelated push. With millions of branches, the branch list a server sends on each connection can grow past 100 MiB, OpenAI says. And every fetch builds a fresh packfile for one client, which is slow to generate and hard to cache.

How Pando is different

Pando drops both requirements. It handles each branch on its own. Pushes to unrelated branches no longer wait on each other. Pando stores packfiles in blob storage and records each branch tip in a shared key-value store. A cache called Express then serves those packs from regional nodes. OpenAI says the cache hit rate is above 99 percent. On a cloud VM, clones and fetches can fill a 10 Gbps network link.

The tradeoff is small. A client reading several branches at once can see them at different points in time. Clients that write to one branch at a time, which covers almost every agent, do not notice.

Pando also leans on packfile-uri. That is a Git protocol feature from a 2019 Google proposal. It lets a server hand clients signed download links. By default Pando advertises only the trunk branch as a base. That keeps the branch list small. Repeat pushes to the same branch resend objects, but the median pack is still under 20 KiB.

What goes back to upstream Git

OpenAI is not open-sourcing Pando. Anthropic has said the same thing about its own Git replacement, Scion. But OpenAI is sending many client-side fixes back to Git itself. The team fixed how the Git client handles concurrent pack downloads, cleans up after interrupted transfers, and resumes finished downloads. It added tracing so a slow fetch can be blamed on either the server or the network.

The company publishes its own Git build to developer machines and inside the ChatGPT desktop app. Some patches are already upstream. Others live in OpenAI's public Git tree, tracked in a public patch tracker.

Source

The New Stack on OpenAI's Pando Git backend, reporting on OpenAI's engineering post by Aaron Friel, Taylor Blau and Ted Nyman.