Google open-sourced TPU Sync for KV cache and weight transfer

A long request to a large language model usually waits twice: once for the prompt to go in, and once for the answer to come out. On Google's TPU chips, the first wait is often not the model itself. It is the data movement.

Google Cloud open-sourced TPU Sync on 12 November. InfoQ says it is a native C++ library. It moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage and the data centre network. Google also publishes Python wheels for JAX and PyTorch. The repository is at github.com/google/tpu-sync. The licence is Apache-2.0. Google uses the library inside vLLM-TorchTPU and SGLang-TorchTPU, two inference servers that already target TPUs.

What the library does

TPU Sync is three pieces. A KV cache manager stores and reuses the key-value state a model builds up when it reads a prompt. A KV store holds that state outside the TPU. A later request with the same prefix can pick up the state without a fresh read. A weight synchroniser moves model weights between machines. That step holds up a reinforcement learning loop between training runs and inference runs. An experimental module targets the weight-sync case for RL.

Google reports numbers against two long-context workloads, measured by Google. Median KV transfer latency on Qwen3 235B dropped from several seconds to about 230 milliseconds. Google says time-to-first-token fell by 60 percent on average for the 235B model. On the 480B model at low concurrency, Google reports a 90 percent drop. On 64K-context workloads with prefix caching on, Google reports up to a 90 percent lower average time-to-first-token. Every number comes from Google's own measurement.

Where it fits

The named gap is the gap between accelerators and the memory tier below them. A TPU holds its working data in HBM. The model's weights and the KV cache can live in host DRAM, on a shared disk, or across the network. Teams running long-context inference, long-running agents, or RL training loops hit the gap most often. The release does not change the TPU, the model, or the serving stack. It changes the layer that moves bytes between them.

The scope is narrow by design. A team building on GPUs does not need it. A team on TPUs that already runs a short-context model may not notice the change. The clear fit is a team that serves Qwen3 or an open-weight model at high context on TPUs. It also fits a team training with RL where the inference side waits for weights.

The licence and what it does not include

TPU Sync is Apache-2.0. Google publishes the library, the Python wheels, and the integration points for vLLM-TorchTPU and SGLang-TorchTPU. The RL weight-sync module is marked experimental in the release. The repository shows 174 stars and 1,327 commits on the main branch at the time of writing. Google does not publish a benchmark script that reproduces the 60 and 90 percent numbers end to end. A team should measure on its own serving stack before counting on them.

For a team that already runs on TPUs, the quickest check is a look at the logs. Does the time-to-first-token or the KV transfer match the pattern Google reports? If it does, the library solves a real step in the pipeline. If the serving stack is on GPUs, this is one to watch rather than one to install.

Source

• Google, google/tpu-sync, the open-source release.
• Daniel Dominguez at InfoQ, Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs.