Baidu has released ERNIE 5.5 Flash Base, a 280B-parameter sparse Mixture-of-Experts (MoE) base model checkpoint activating 14B parameters per token, pre-trained from scratch on 33.5T tokens using the Muon optimizer. The architecture comprises 43 layers with a 4096 hidden size and employs a hybrid attention mechanism: 37 layers utilize Heavily Compressed Attention (HCA) caching one entry per 128 tokens with 512-dim heads, while 6 layers use Multi-head Latent Attention with sparse attention (MLA-DSA) with 256-dim heads for full-context access. It features a Multi-Router Latent MoE design with 512 routed experts (top-10 routing) plus one shared expert operating in a 2048-dim latent space, alongside gated mixed heads reducing gate-projection parameters by 75% and Virtual Head Attention for low-rank cross-head mixing. ECHO repurposes the final backbone layer as a built-in speculative draft model, adding 0.36B parameters excluded from the headline count. The model supports 128K training context and 1M token inference context. In Baidu's self-reported unified evaluation, ERNIE 5.5 Flash Base leads 24 of 30 benchmarks, achieving MMLU-Pro 78.4 (vs. DeepSeek-V4-Flash-Base 69.2), HumanEval+ 82.3, and LongBench-v2 48.7, while trailing on HELMET-Rerank (56.2 vs. 60.0). Weights are available in FP8 and BF16 on Hugging Face and ModelScope, deployable via SGLang (--tp 4) and vLLM 0.32.0 with the ERNIE plugin under a license free below US$50M revenue excluding EU, UK, and South Korea.
Read original
marktechpost