OmniPack is a training-free framework designed to optimize the efficiency of Omni-modal Large Language Models (Omni-LLMs) by addressing the computational overhead caused by redundant audio and visual token sequences. To mitigate the loss of critical structural evidence and underutilized multimodal collaboration found in existing methods, OmniPack employs a two-stage compression strategy. The first stage occurs before the LLM, where it eliminates structural redundancy through modality-specific importance scoring, global coverage mechanisms, and similarity-aware merging. The second stage performs task-relevant semantic refinement within the LLM, consolidating representations using textual guidance and audio-visual collaboration after initial multimodal interactions have occurred. Experimental evaluations across five benchmarks using three different Omni-LLM backbones show that OmniPack achieves a superior performance-efficiency trade-off compared to existing techniques. Specifically, when tested on the Qwen2.5-Omni-7B model, the framework maintains 98.0% of original performance while reducing FLOPs to 16.7%. Furthermore, the method demonstrates extreme efficiency by retaining 92.9% of original performance while utilizing only 6.8% of the original FLOPs, making it highly effective for deployment in resource-constrained environments.

Read original