Muir introduces a linear attention architecture that improves upon Gated DeltaNet-2 (GDN-2) by reallocating gating parameters based on sensitivity analysis of state updates. The authors demonstrate that state updates are significantly less sensitive to erasure than to value writes: key activation and unit normalization concentrate key energy onto sparse channel subsets, allowing the erase gate to operate at lower rank, while value writes scale all channels. Additionally, slow-decaying state rows accumulate larger norms that dominate unweighted readout, necessitating stronger write-time erasure to suppress interference. Muir addresses this by fusing all gates into a sensitivity-allocated low-rank projection and applying a matched-filter readout gate that re-weights state rows. Saved parameters are reinvested into a wider feed-forward network. To enable stable training at large chunk sizes required for wide matrix units, a parallel update algorithm replaces standard chunkwise methods. Evaluated at 166M-1.34B parameters, Muir reduces perplexity by 1.8-2.0% over GDN-2 and 2.5-3.6% over Transformer, increases commonsense accuracy by up to 1.9 points, and improves extractive recall. At 1.34B, it outperforms baselines on multi-hop and multi-needle recall, extrapolates 2,560x from 4K training length to 10M tokens without perplexity drift, and maintains near-100% passkey retrieval to 1M tokens with static readout. Muir matches GDN-2 throughput, reaches 100B-token validation loss 1.2x faster, and trains 2.7x faster than Transformer at 32K context.
Read original
huggingface/daily-papers