The article surveys seven Mixture-of-Experts (MoE) routing strategies deployed across major open models, detailing how routers assign tokens to experts ranging from 8 (Mixtral) to 384 (DeepSeek-V4-Pro) with active counts of 1 to 8 per token. Softmax Top-K remains the baseline: Mixtral selects 2 of 8 experts, gpt-oss-120b picks 4 of 128, and Qwen3-235B-A22B activates 8 of 128, typically balanced via auxiliary loss. Top-1 Switch routing, used by Switch Transformer and Llama 4 Maverick (with a shared expert), routes each token to a single expert using a capacity factor buffer, dropping overflow via residual connections. DeepSeekMoE popularized shared plus routed experts; DeepSeek-V3 employs 1 shared and 256 routed experts activating 8, mirrored by Kimi K2, GLM-4.5, and Qwen3.6-35B-A3B, though Qwen3 later removed shared experts. The dominant 2026 frontier approach is sigmoid gating with aux-loss-free bias balancing, adopted by DeepSeek-V3, V4, GLM-4.5, and Kimi K2; DeepSeek-V4 modifies scoring to sqrtsoftplus while retaining bias updates. This method reduced MaxVio from 0.72 to 0.04 at 1B parameters with lower perplexity. DeepSeek-V2/V3 implement node-limited routing, capping tokens at 4 nodes (8 groups) to overlap compute and communication, while Kimi K2 disables grouping. Expert Choice routing, where experts pick tokens for fixed bucket sizes, accelerates convergence over 2x but leaks future tokens in causal models, causing an abnormal ~10% loss drop; no surveyed production model uses it. DeepSeek-V4 applies hash routing in its first three MoE layers, and dynamic Top-K (variable experts per token) integrated into SGLang by SMIDT improves P99 time-to-first-token by 1.21-1.26x. The article advises checking config.json for expert count, experts per token, shared experts, and topk_method before serving, noting that changing Top-K at inference alters model quality.
Read original
marktechpost