Two IBM engineers plugged SGLang into Spyre in a week

Hardware integrations usually run for months. A two-person IBM team put SGLang on IBM's Spyre accelerator in less than a week. SGLang is the open-source LLM serving stack. The report was just published on the PyTorch blog. The authors are Gal Bloch, Itay Etelis, Raghu Ganti and Mudhakar Srivatsa at IBM, with Alex Nails at RadixArk. The post describes both the method and the numbers.

What they did

The team built an SGLang plugin for Spyre of about 1,500 lines. They worked from an existing PyTorch integration for the hardware. The torch-spyre package already registered Spyre as a PyTorch device and connected it to torch.compile. SGLang already had clear extension points. The task was to connect the two.

Two engineers did the integration alongside AI coding agents. The agents used the existing PyTorch implementation of Spyre operations as reference code. They adapted that code to fit SGLang's backend interfaces. The human engineers kept the design in their heads and reviewed every change.

Four small patches went into SGLang itself. Two made existing execution paths available without CUDA-specific dependencies. Two registered Spyre as a supported device and attention backend without using Triton. The compatibility changes can support other accelerators too. The registration changes are specific to Spyre.

The numbers

SGLang on Spyre hit about 3.6 times the throughput of the reference PyTorch inference implementation using torch-spyre. The team also compared SGLang and vLLM on Spyre. Across the measured workloads for Granite 3.3 8B and Gemma 4 26B A4B, SGLang's performance was on par with vLLM.

Prefix sharing through SGLang's RadixCache worked with no Spyre-specific changes. This matters because Spyre is not a GPU. Many prefix-sharing features in open serving stacks carry CUDA assumptions in their code path.

Why the story is worth reading

A hardware integration is the kind of task that usually runs on a long timeline. A vendor releases a new accelerator. It then reaches out to serving-stack maintainers. The back-and-forth takes months. Two engineers finishing it in a week is the specific shape the post makes real.

The post does not hide what made the week possible. The compiler was already in place. The serving stack had explicit extension points. The coding agents were working on a known translation task, not inventing a design. That is a focused use of an agent and it fits the general pattern: an agent does well on bounded code work where the design is set and the reference is in the repository.

The team also does not generalise. It is one integration, measured on two workloads. The 3.6-times throughput number is against torch-spyre, not against another serving stack. The on-par number against vLLM is on two models on specific workloads. Each figure is traceable to the same post.

An engineering team evaluating where to point its own agents has one more current reference point. The pattern that works here is small team plus established hardware compiler plus a target framework with clean extension points. The integration itself is open on GitHub under the team's name.

Source

Integrating SGLang into Spyre in Under a Week with AI, by Gal Bloch, Itay Etelis, Raghu Ganti, Mudhakar Srivatsa and Alex Nails at the PyTorch Foundation.