Moonshot AI's Kimi Linear introduces Kimi Delta Attention (KDA), a channel‑wise gated linear attention module that replaces head‑level gating with per‑feature decay scalars αₜ, enabling selective memory updates via Sₜ = (I − βₖₖᵀ)·Diag(αₜ)·Sₜ₋₁ + βₖₖᵀ vₖᵀ. The transition matrices are expressed in a Diagonal‑Plus‑Low‑Rank (DPLR) form, allowing chunk‑wise (256‑token) processing and triangular solves on GPU Tensor Cores. Rather than discarding full attention, the architecture stacks three KDA layers for each Multi‑Head Latent Attention (MLA) layer (3:1 hybrid), using NoPE where positional information emerges from decay dynamics. The released 48B‑parameter model activates 3B tokens per step via a sparse MoE. Benchmarks show: MMLU‑Pro (4 k) 51.0 pts, matching full‑attention MLA; RULER 128 k 84.3 pts with 3.98× speedup; RULER 1 M 94.8 pts; KV cache reduced by 75%; decoding throughput 6.3× faster at 1 M tokens; prefill latency 2.3× faster at 512 k tokens. The flagship Kimi K3 employs 69 KDA and 24 Gated MLA layers (≈2.8 T parameters), validating the design at scale. Open‑source artifacts include model checkpoints, FLA‑based KDA kernels, vLLM integration, and LMCache guidance for recurrent state caching.
Read original
dev.to