Moonshot AI has released Kimi K3, a massive 2.78 trillion-parameter multimodal Mixture-of-Experts (MoE) model featuring 104.2 billion active parameters and a one-million-token context window. To manage this scale, K3 utilizes a hybrid attention architecture combining Kimi Delta Attention (KDA)—a recurrent linear-attention mechanism using a bounded decay rule to maintain BF16 dynamic range—with Multi-head Latent Attention (MLA) every four layers to provide global content matching. The model employs a "Stable LatentMoE" design where the 7,168-dimensional hidden state is projected to a 3,584-dimensional space to interact with 896 routed experts, using SiTU-GLU activation and Quantile Balancing to ensure stable load distribution. For long-horizon agentic tasks, K3 was trained using reinforcement learning with persistent, pauseable sandboxes via Firecracker microVMs, allowing trajectories to survive multiple policy updates. While Moonshot claims K3 identifies 16 previously unknown vulnerabilities and achieves significant scaling efficiency, independent evaluations suggest that while K3 excels in tool-heavy, long-context workloads, it remains slightly behind closed-source frontier models like GPT-5.6 Sol in overall performance. The model is released with MXFP4 expert weights and MXFP8 activations, requiring specialized infrastructure to manage its complex hybrid cache and high communication overhead.

Read original