OpenAI cut ChatGPT storage read latency 12 times to 10 ms

OpenAI has written up how it dropped read times on the storage behind ChatGPT. The global median went from 120 milliseconds to 10. That is a 12 times speed-up in one year. The post names three changes, and the trade-offs behind each.

OpenAI published Part II of its Habitat series on 23 October. Habitat is the storage platform behind ChatGPT. OpenAI says it serves more than one billion users across 40 Azure regions. The post is about the Azure Cosmos DB layer under Habitat. It names the changes that cut read latency and held the service up at scale.

The three changes that cut 120 ms to 10 ms

OpenAI says reads are over 90 percent of Habitat traffic. It made three changes this year to speed them up. First, it keeps data in the same region as the users who read it. This only works where the cloud has room for it. Second, it replicates the smallest and most read-heavy datasets to many regions at once. Login metadata is one example. Some sets replicate to all 40 Azure regions where OpenAI serves traffic.

Third, it put a distributed cache in front of the database. The cache runs on Valkey servers near the compute. The cache is invalidated through the Azure Cosmos DB change feed, so stale data clears within seconds. OpenAI says this design follows Meta's TAO. The post also says the Habitat service itself was rewritten from Python to Rust to cut proxy overhead.

OpenAI says these changes together brought global p50 read latency from about 120 ms to about 10 ms. It calls this a 12 times improvement.

Multi-tenancy, in three phases

OpenAI also walks through how it stored data across Azure Cosmos DB containers. It started with one shared container. One heavy tenant could slow all the others down. Different product teams would collide on common partition keys like user_id. So OpenAI moved each use case to its own container. Capacity planning then became every product team's problem. Launches would quietly outrun pre-provisioned throughput.

OpenAI then worked with Azure on a shared design with hot-key throttling and request tiering. A new serverless mode lets each physical partition reach 10,000 request units per second. It splits on either storage or throughput. OpenAI says it has had no provisioning or crowding incidents in the months since.

What it says about scale, and what Azure built for one customer

OpenAI says its largest Azure Cosmos DB account holds tens of petabytes. The same account serves more than 20 million queries per second. It says it is the only company that has deployed Cosmos DB at this scale. The company is also building an in-house system for the next step. The system uses RocksDB Cloud and Raft replication, and the same document model as Cosmos DB.

For most readers, the useful part is the pattern. A model-backed feature that reads state on every request is the common case now. Three choices decide how fast it feels. How close the data sits to the user. How much of a small working set is cached. How quickly the cache clears when a write lands. OpenAI's post shows one way to answer all three.

Source

Rapidly scaling online storage to serve over 1 billion ChatGPT users, Part II on OpenAI.