Nozawa, Lin, and Ng propose two visual focusing loss functions for self-supervised Vision Transformers (ViTs) such as DINO to induce spatial attention head specialization without supervision. Standard self-supervised training lacks explicit focusing mechanisms, producing ambiguous or redundant attention maps. The authors leverage the multi-head attention architecture by designing losses that enforce correspondence between individual heads and distinct image regions, encouraging each head to attend to spatially localized, non-overlapping areas. This drives the emergence of diverse, specialized heads that collectively cover the scene. Quantitative evaluation shows the method substantially increases spatial selectivity and head diversity metrics compared to baseline DINO. The improved representations yield performance gains on dense prediction tasks, specifically zero-shot image segmentation, where the specialized heads provide more discriminative patch-level features. Qualitative analysis confirms heads learn interpretable, spatially focused roles—such as attending to object boundaries, textures, or semantic parts—without any pixel-level annotations. The approach adds minimal computational overhead during training and requires no architectural changes, offering a plug-in enhancement for self-supervised ViTs that improves both interpretability and downstream dense task performance.
Read original
huggingface/daily-papers