An analysis of ten open Mixture-of-Experts (MoE) models reveals significant divergence across five routing parameters: expert count, top-k selection, shared experts, scoring functions, and load-balancing methods. Models scale from Mixtral 8x7B (8 experts, top-2, 47B/13B total/active parameters) to DeepSeek-V4-Pro (384 experts, top-6, 1.6T/49B) and Kimi K2 (384 experts, top-8, 1.04T/32B). Routed sparsity ranges from 4x (Mixtral) to 128x (Llama 4 Maverick: 128 experts, top-1). Seven models employ one shared expert; Mixtral, gpt-oss-120b, and Qwen3 omit them. DeepSeek-V3 and V4-Pro use sigmoid and sqrtsoftplus scoring respectively with bias-based noaux_tc balancing; V4-Pro introduces hash routing in its first three layers. Qwen3-235B-A22B uses 128 experts with top-8 and a global-batch auxiliary loss (coefficient 0.001), while Qwen3.6-35B-A3B scales to 256 experts plus one shared. gpt-oss-120b (128 experts, top-4) and Llama 4 Maverick withhold gating and balancing details in their configs. Kimi K2 reports sparsity 48 required 1.69x fewer FLOPs than sparsity 8 at equal validation loss. Deployment implications include increased memory residency and all-to-all traffic with higher expert counts, while lower top-k reduces per-token compute. MXFP4 quantization enables gpt-oss-120b to fit on a single 80 GB GPU, whereas DeepSeek-V4-Pro and Kimi K2 require multi-GPU expert parallelism.

Read original