AMD measured speculative decoding in vLLM at up to 2.87x on MI300X and MI355X GPUs

A post on the vLLM blog, written by AMD and Embedded LLM and dated August 23, measures five speculative decoding methods on AMD Instinct MI300X and MI355X GPUs. It reached the Hacker News front page on September 7, where it stood at 73 points at the time of this sweep.

Speculative decoding uses a small draft model to propose several candidate tokens, which the large target model then verifies in a single pass. When the target accepts most of the proposals, the server commits several tokens per step instead of one, and throughput climbs without changing the output.

Five methods, one framework

The post compares native MTP, where the token prediction path is built into the target model itself; a separate Gemma 4 MTP checkpoint that shares the target's activations and KV cache; EAGLE-3, an autoregressive draft network fed fused hidden states from three stages of the target model; DFlash, a parallel draft network that predicts whole blocks at once; and DSpark, a DFlash variant that adds sequential correction and confidence-based prefix selection.

The spread is the finding

The authors' best results came on reasoning and code workloads. They measured Gemma 4 26B-A4B at 2.87x baseline throughput with DFlash on the MATH500 dataset, and 2.83x with the Gemma 4 MTP draft. Kimi-K2.5 reached 2.68x with DFlash. Qwen3.5-122B-A10B reached 2.20x with native MTP, and Qwen3-8B reached 1.63x with DSpark on GSM8K.

The same tables carry the warning. Qwen3-8B with EAGLE-3 on MATH500 stayed below baseline, and the authors report that results varied with the target model, the drafting method, the workload, and the proposal length. A proposal length of 7 was frequently the best setting, with exceptions, and a longer proposal window sometimes cut throughput because acceptance dropped at the later draft positions. Their advice is to treat a model card's recommended configuration as a starting point, choose the final setting against representative workloads, and read acceptance metrics together with throughput.

The measurements come from AMD and a partner writing about AMD hardware. The post benchmarks each method against unassisted decoding on the same GPU and makes no comparison with other vendors' chips. Read that way, the numbers are still useful to a team serving open models on this hardware: they show which draft methods vLLM supports on ROCm today, which pairings nearly tripled throughput, and which pairings fell below it. The gap between 2.87x and below-baseline on the same GPU is the practical lesson. Speculative decoding is a per-workload tuning decision, and a team that enables it fleet-wide off a headline number can end up paying for a slowdown.

Source

Primary source: Exploring Speculative Decoding in vLLM on AMD GPUs on the vLLM blog.