This research investigates the mechanisms driving sparse Mixture-of-Experts (MoE) routing, challenging the assumption that expert benefits stem solely from geometric complementarity. By introducing the Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled 2x2 factorial design, the authors analyze architectures including OLMoE, Mixtral, and DeepSeek. The study reveals that while expert subspaces exhibit substantial overlap, actual routing provides superior representation explanations compared to matched alternatives. Notably, the researchers found that the selected candidate explains more of the residual representation than the strongest unselected rival in every factorial cell; however, the presence of a prefix narrows this advantage through negative interactions, as evidenced by all 95% confidence intervals falling below zero. Despite this geometric narrowing, the authors argue that multi-expert computation is not functionally redundant. In 24 out of 39 frozen-route comparisons, adding subsequent experts improved next-token prediction performance, and controlled training experiments consistently showed that Top-2 routing outperforms Top-1 across all three seeds. The authors define this phenomenon as "coherent overlap," where routing selects relevant experts from a shared geometric neighborhood, providing functional utility despite significant subspace overlap. This distinction clarifies that geometric similarity does not directly dictate the redundancy or pruning potential of MoE components.
Read original
huggingface/daily-papers