The release of Kimi K3 marks a significant advancement in open-weights large language models, featuring a 2.8 trillion parameter Mixture-of-Experts (MoE) architecture. Achieving a 2.5x improvement in scaling efficiency over its predecessor, Kimi K2, the model utilizes a Stable LatentMoE configuration that routes tokens across 896 experts, activating only 16 per token to maintain an active parameter count of 104 billion. To mitigate information degradation in its deep transformer architecture, the model employs Kimi Delta Attention (KDA) and Attention Residuals, which optimize information flow across sequence lengths and model depths. These innovations support a massive 1-million-token context window, essential for long-horizon reasoning. The training process utilized a perfectly balanced expert-parallel strategy alongside advanced memory management to facilitate agentic reinforcement learning (RL) within sandbox states. While Kimi K3 currently trails proprietary models like Claude Fable 5 and GPT-5.6 Sol in aggregate benchmarks, it outperforms other open and proprietary alternatives. By combining high-capacity parameters with optimized inference efficiency through algorithm-system co-design, K3 provides a robust foundation for complex agentic workflows, such as multi-step code generation and repository-wide analysis, through its open-weight availability.

Read original