vLLM has implemented speculative decoding on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ platform, enabling the system to verify multiple draft tokens in a single target-model pass. This mechanism separates the generation process into a lightweight drafting stage and a target-model verification stage to improve throughput over standard autoregressive decoding. The implementation supports five distinct drafting methods: native Multi-Token Prediction (MTP), Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods vary in architecture; native and Gemma 4 MTP use sequential drafting, EAGLE-3 utilizes an autoregressive draft network conditioned on fused hidden states, while DFlash and DSpark employ parallel drafting where multiple positions are predicted in a single forward pass. Experimental results demonstrate significant throughput gains, with the highest measured ratios reaching 2.87× for DFlash on HumanEval and 2.74× for Gemma 4 MTP on GSM8K. Performance is highly dependent on the model family, the specific drafting method, and the optimal number of speculative tokens (N). For example, Qwen3.5-122B-A10B achieved a maximum throughput of 2.20× using native MTP on MATH500. The study highlights that effective speculative decoding requires balancing drafting computational costs with the benefits of increased token acceptance rates across different workloads.

Read original