vLLM has implemented speculative decoding on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ platform, enabling the system to verify multiple draft tokens in a single target-model pass. This mechanism separates the generation process into a lightweight drafting stage and a target-model verification stage to improve throughput over standard autoregressive decoding. The implementation supports five distinct drafting methods: native Multi-Token Prediction (MTP), Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods vary in architecture; native and Gemma 4 MTP use sequential drafting, EAGLE-3 utilizes an autoregressive draft network conditioned on fused hidden states, while DFlash and DSpark employ parallel drafting where multiple positions are predicted in a single forward pass. Experimental results demonstrate significant throughput gains, with the highest measured ratios reaching 2.87× for DFlash on HumanEval and 2.74× for Gemma 4 MTP on GSM8K. Performance is highly dependent on the model family, the specific drafting method, and the optimal number of speculative tokens (N). For example, Qwen3.5-122B-A10B achieved a maximum throughput of 2.20× using native MTP on MATH500. The study highlights that effective speculative decoding requires balancing drafting computational costs with the benefits of increased token acceptance rates across different workloads.
Read original
hackernews