To address the computational bottlenecks caused by long token sequences in diffusion transformers during video generation, the researchers introduce Sol-Attn (Sparsifying Online Attention), a training-free method for dynamic sparse attention. Unlike existing sparsification techniques that suffer from rigid routing budgets or high overhead from materializing proxy scores, Sol-Attn integrates dynamic routing, sparse computation, and approximation correction into a single online-softmax pass. The method utilizes on-the-fly block thresholding with proxy-score reuse, which identifies critical key-value blocks by comparing proxy scores against a threshold during the softmax operation. This approach allows for a dynamic, controllable block budget without the need to materialize a full proxy map. Crucially, instead of discarding unselected blocks entirely—which often causes accuracy degradation—Sol-Attn reuses the proxy scores of discarded blocks to approximate their contribution to the attention mechanism. Empirical evaluations on image and video generation tasks demonstrate that Sol-Attn significantly advances the accuracy-efficiency frontier. Specifically, the method achieves end-to-end speedups of 2.1x for video generation and 2.3x for video editing tasks, all while maintaining high visual quality and avoiding the need for additional training.
Read original
huggingface/daily-papers