NVIDIA demonstrated three stacked optimizations that lifted Ling-3.0-flash decode throughput 42.5% on a single DGX Spark desktop system (GB10, 128 GB unified memory, 273 GB/s bandwidth). The 124B-parameter MoE model activates 5.1B parameters per token across 512 routed experts and one shared expert, using a hybrid of 35 Kimi Delta Attention and 7 Gated MLA layers with a native 256K context window. Baseline W4A4 inference with FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM, and an FP32 LM head ran at 37.81 tokens/s on 1,319 GSM8K questions (5-shot CoT, temp 0.6, top_p 0.95, top_k 29). First, replacing FlashInfer with Humming MXFP4 kernels—which JIT-compile via NVRTC for exact matrix shapes including the 768-wide expert layer in seconds versus tens of minutes—raised throughput 9.3% to 39.45 tokens/s. Second, quantizing the 157,184×2,560 BF16 LM head to FP8 at load time via SGLANG_ENABLE_FP8_LM_HEAD=1 halved weight reads, adding 11.4% over Humming alone for 43.94 tokens/s (21.7% over baseline). Third, enabling built-in MTP speculative decoding via the model’s NEXTN layer (drafting 3 tokens, verifying 4 positions at ~2.26× normal step cost) pushed synthetic throughput 185.1% to 102.88 tokens/s with 3.90 accepted tokens/round. On GSM8K, acceptance fell to 2.68 due to sampling, shorter outputs (median ~310 tokens), and less repetitive text, yielding a final 53.89 tokens/s, 24.5% lower mean latency (15.21→11.49 s), and statistically flat accuracy (96.89%→96.82%). Critically, stacking FP8 LM head and MTP requires a shared LM-head module; otherwise the draft misinterprets FP8 weights as BF16, collapsing acceptance to 1.00 and throughput below baseline.

Read original