Researchers propose eliminating calibration data for channel selection in Mixture-of-Experts (MoE) decoding. MoE models face bottlenecks during interactive decoding due to loading routed-expert weights from memory. Traditional methods use calibrated thresholds to filter channels, but these fail to control read budgets consistently: thresholds calibrated for an 18% read budget averaged only 12.9% across benchmarks. Instead, a top-k selection approach retains the k highest-scoring channels per token without calibration data, matching or outperforming threshold-based methods in 40 of 43 comparisons at 13% and 18% budgets, including all 23 generation tasks. Top-k also avoids repetition-loop collapse observed with calibrated thresholds on Gemma 4. The authors introduce a model-weight-derived reader using a ternary sketch to screen channels and a low-rank map to recover skipped ones. Across three post-trained MoE models, this reader preserves 98.1% of full-model accuracy at 17.8% of routed-expert reads. Optimized Triton kernels further accelerate MoE blocks 1.60–1.90x over vLLM at batch size one and up to 2.69x at larger batches. These results demonstrate that calibration-free top-k selection and efficient readers can maintain accuracy while drastically reducing memory bandwidth and latency during decoding.
Read original
huggingface/daily-papers