Researchers have addressed performance degradation in multimodal entity linking, particularly regarding rare entities, by moving beyond popularity-based metrics like pageviews toward structural knowledge-graph metrics. By evaluating how well entities are documented and interconnected, the study identified failure modes in state-of-the-art models, where accuracy dropped by 15.4% to 39.9% on rare-entity slices. To mitigate these issues, the authors propose a training-free framework that utilizes a reasoning-capable vision-language model to iteratively perform dynamic evidence gathering through Wikipedia searches. Experimental results demonstrate that reasoning and retrieval are complementary components; while reasoning alone offers minimal gains on rare entities, and retrieval alone may decrease overall accuracy, their integration yields the best performance. Testing on the MERLIN benchmark—a multilingual multimodal dataset spanning Hindi, Indonesian, Japanese, Tamil, and Vietnamese—showed that the proposed framework improves overall state-of-the-art accuracy by 6.9% and enhances performance on rare-entity slices by up to 23.3%. To facilitate further research into these specific edge cases, the authors introduced MERLIN-Rare, a collection of test slices specifically designed for evaluating rare-entity performance.
Read original
huggingface/daily-papers