Researchers have introduced LLaDA MoE v2, a study that systematically characterizes the scaling laws and architectural requirements for Mixture-of-Experts (MoE) diffusion language models (dLLMs). By conducting an IsoFLOP analysis, the authors identified distinct scaling behaviors that differ from traditional autoregressive models. Key findings indicate that for MoE dLLMs, the optimal nominal batch size increases more aggressively with compute, while the optimal learning rate decays more rapidly. Furthermore, the study reveals a data-side tilt in resource allocation, where the optimal token budget grows faster than the activated model-side computation. Architecturally, larger scales favor increasing the total expert pool size while maintaining a fixed activated capacity, though the fraction of activated capacity dedicated to shared experts remains stable. Leveraging these principles, the team trained LLaDA MoE v2, a 30B-parameter model with 3B activated parameters, using 23.5T tokens. Despite using only 65% of the pretraining tokens utilized by Qwen3, LLaDA MoE v2 achieves comparable performance across knowledge, reasoning, and coding benchmarks. Notably, following supervised fine-tuning, the model outperforms SDAR Chat on seven out of eight reasoning and coding metrics, demonstrating the practical viability of scaling diffusion-based MoE architectures to large-scale language modeling tasks.
Read original
huggingface/daily-papers