DistilVDR introduces a compact, end-to-end visual document retrieval (VDR) system designed to overcome the latency and computational costs associated with multi-billion-parameter models. The researchers developed a 524M parameter architecture using a dual-student distillation method, where a single 8B vision-language teacher supervises the student through a pointwise cosine alignment loss. This approach leverages the teacher's pre-trained embedding space, eliminating the need for explicit relevance labels, negative sampling, or contrastive loss terms. To address the inherent asymmetry between text-based queries and image-based documents, the authors implemented an asymmetric encoder-only architecture, allocating 70M parameters to the query side while concentrating visual capacity on the document encoder. The study presents two variants: DistilVDR-HiRes and DistilVDR-Fast. The HiRes version achieves an average NDCG@5 of 61.74 on the ViDoRe v1+v2+v3 benchmarks—retaining 86.9% of the 8B teacher's performance—and outperforms all reproduced sub-1B multi-vector baselines on the high-resolution-sensitive v3 benchmark. Conversely, DistilVDR-Fast reaches 59.98 NDCG@5 while utilizing a visual-token budget three times smaller than the HiRes version. Notably, these models facilitate indexing speeds an order of magnitude faster than existing methods and produce a document index that is 15.6 times smaller than the strongest sub-1B multi-vector baseline.
Read original
huggingface/daily-papers