The authors systematically disentangle three bundled design choices in multimodal vision encoding—upstream visual priors, dedicated vision capacity, and pre-fusion placement—by pretraining all variants from scratch on common data at matched lifetime compute. They demonstrate that a vision encoder trained jointly with the language backbone matches the performance of a separately pretrained encoder on visual question answering and language benchmarks, eliminating the need for a distinct pretraining stage. However, dedicated vision capacity proves essential: an encoder-free architecture underperforms on visual understanding even when the backbone is expanded to absorb the encoder's parameter budget. Crucially, this capacity need not reside before fusion; integrating vision-only modules into early backbone layers (in-fusion) achieves comparable results to a pre-fusion encoder while reducing time to first token. These conclusions hold across backbone scales, when encoder pretraining compute is treated as free, and when initializing from a pretrained language model. The study concludes that jointly trained, dedicated vision capacity placed early within the multimodal backbone is the preferred configuration.
Read original
huggingface/daily-papers