Researchers introduced EvaOmni, a new benchmark designed to evaluate audio-visual reasoning in multimodal large language models (Omni LLMs) by emphasizing evidence-grounded assessment. Unlike traditional benchmarks that rely on multiple-choice questions and score models solely on answer accuracy, EvaOmni requires models to provide both answers and supporting evidence intervals from audio and visual inputs. The benchmark includes 392 videos and 1,027 human-annotated question-answer pairs, spanning 15 capabilities across three difficulty levels: retrieval and localization, basic understanding, and complex reasoning. To ensure quality, the authors implemented rigorous annotation protocols involving expert labeling of evidence, creation of video-based distractors, human review against strict criteria, and filtering of textual and unimodal shortcuts by models. Two novel metrics were introduced: the Coverage Grounded Score (C.G.) measuring the extent of identified evidence, and the Bottleneck Grounded Score (B.G.) penalizing poor temporal localization in either modality. Evaluations across 7 open-weight and 17 proprietary Omni LLMs revealed that Gemini3.7-Flash performed best, achieving 79.4% overall accuracy, 70.8% C.G., and 35.1% B.G. At the highest complexity level (Level 3), it reached 79.1% accuracy but only 12.3% B.G., highlighting persistent challenges in grounded reasoning despite strong performance.
Read original
huggingface/daily-papers