This study investigates optimal visual-token allocation strategies in long-video Multimodal Large Language Models (MLLMs) by isolating three variables: frame selection, spatial compression, and token reinvestment. To ensure experimental rigor, the authors utilized a controlled harness to avoid the confounding variables found in existing literature, such as varying prompt boundaries or resolution policies. The research reveals that selection is the most impactful lever; for instance, using eight query-selected frames improved performance on the LongVideoBench hour-long bin by 6.9 points compared to sixteen uniformly spaced frames. Notably, the Orthogonal Matching Pursuit algorithm, a sparse-approximation method, performed competitively with modern, purpose-built selectors. Regarding compression, halving the spatial budget per frame resulted in a minimal performance degradation of at most 0.44 points. The study demonstrates that compression only yields significant benefits when the saved tokens are reinvested; specifically, reallocating those tokens to process twice as many compressed frames increased accuracy by two to three points. Ultimately, the findings highlight that effective long-video understanding relies on a synergistic approach where the computational savings from spatial compression are aggressively reinvested into increased temporal coverage through smarter frame selection.
Read original
huggingface/daily-papers