Researchers have introduced Declarative Attention (DA), a novel protocol designed to optimize inference efficiency in long-context language models by enabling them to manage their own attention mechanisms. Rather than relying on extrinsic proxy scores that still require linear O(N) scanning of the KV cache, DA empowers models to explicitly declare their required attention scope within their chain-of-thought. This process partitions token generation into three distinct operational modes: <global> for full context access, <focus> for specific regions, and <local> for recent output only. The inference engine treats these declarations similarly to tool calls, allowing it to skip the majority of the KV cache reading process. In zero-shot evaluations across 15 long-context tasks using off-the-shelf models like Gemma-4-31B and Qwen-3.6-27B, the method demonstrated significant computational savings. Specifically, DA reduced total attended tokens during decoding by 52.0% for Gemma-4-31B and 31.1% for Qwen-3.6-27B. These efficiency gains were achieved with minimal impact on performance, resulting in modest accuracy decreases of only 1.27 percentage points and 2.75 percentage points, respectively. The study notes that these accuracy drops tend to diminish as model scale increases, suggesting that larger models are more adept at utilizing this sparse attention axis effectively.
Read original
huggingface/daily-papers