NVIDIA shows GLM-5.3 at 607 tokens per second across two DGX stations

NVIDIA's developer blog has published a recipe for serving GLM-5.3 with vLLM. GLM-5.3 is a 753 billion parameter open model. The recipe runs it across two DGX Stations. The post was written by Fatima Fall. It reports throughput, latency and memory numbers from NVIDIA's own test runs.

Why two stations

The model activates 40 billion parameters per token through a mixture of experts. Its checkpoint takes about 464 GB of GPU memory. One GB300 GPU inside a DGX Station holds 252 GB. So the model does not fit on one station. NVIDIA used tensor parallelism to split the weights in half across two. Each GPU loaded about 231 GiB after the model and vLLM runtime buffers were in place.

The two stations were linked over ConnectX-8 RDMA. NVIDIA states the link ran at 400,000 Mb per second with an MTU of 9,000. That link carries the parts of the computation that cross stations during inference.

The throughput curve

NVIDIA ran the SPEED-Bench coding dataset through the server at four levels of parallel requests. The results NVIDIA reports:

Parallel requests	Overall throughput	Mean latency	Mean time to first token
1	107.4 tok/s	49.84 s	479.75 ms
8	386.0 tok/s	114.17 s	433.79 ms
16	607.8 tok/s	136.46 s	482.90 ms
32	385.0 tok/s	453.54 s	899.36 ms

Peak throughput sits at 16 parallel requests. At 32, the throughput drops back to the 8-request number. The mean latency triples. The per-token speed drops from 23.8 ms to 77.9 ms. Past 16 requests the system is overloaded, not faster.

What else is turned on

NVIDIA states that the FP8 KV cache halves cache memory compared to 16-bit storage. Prefix caching is on. The server does not reprocess a prompt prefix it has already seen. The model's multi-token prediction layer drafts four tokens per step. vLLM then verifies them with the full model before accepting them.

NVIDIA also prints one plain warning in the recipe. Binding the server to 0.0.0.0 with host networking exposes an unauthenticated API on the primary station. That includes the management interface. NVIDIA tells the reader to firewall the port. The reader should also add authentication and TLS before opening wider access.

A team thinking about serving a large open model on its own hardware now has a reproducible setup. Every number in it is measured. The ceiling on this pair of stations is 16 concurrent requests. A workload with more parallelism than that needs a larger cluster. Pushing the same two boxes harder will slow them down.

Source

When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM, NVIDIA Developer, Fatima Fall.