MiniMax Sparse Attention (MSA): Scaling Long-Context Capabilities via Blockwise Sparsity

MiniMax Sparse Attention (MSA) introduces a blockwise sparse attention mechanism built upon Grouped Query Attention (GQA) to mitigate the quadratic computational overhead of standard softmax attention, enabling efficient processing of million-token contexts for agentic workflows and repository-scale reasoning.

The Challenge of Ultra-Long Contexts

As frontier Large Language Models (LLMs) evolve, the demand for ultra-long-context capabilities has become critical. Advanced applications—such as autonomous agentic workflows, large-scale codebase reasoning, and persistent memory systems—require models to attend to hundreds of thousands, or even millions, of tokens simultaneously. However, the quadratic scaling cost of traditional softmax attention creates a significant bottleneck, making deployment at this scale computationally untenable.

Introducing MiniMax Sparse Attention (MSA)

To address these efficiency constraints, MiniMax Sparse Attention (MSA) implements a blockwise sparse attention strategy. By leveraging Grouped Query Attention (GQA) as its foundation, MSA aims to reduce the memory and compute requirements associated with the attention mechanism without sacrificing the model's ability to retrieve critical information across vast contexts.

Technical Implementation: The Index Branch

A core component of the MSA architecture is the introduction of a lightweight Index Branch. This branch is designed to score key-value (KV) blocks, allowing the model to selectively attend to the most relevant segments of the context rather than computing the full attention matrix. This sparse approach effectively bypasses the quadratic cost associated with dense attention, optimizing the throughput for long-sequence inference.