Balestriero and LeCun prove a Platonic Representation Theorem showing that Joint-Embedding Predictive Architectures (JEPAs) trained independently on any number of modalities converge to identical representations up to a single rotation when positive pairs consist of two observations of the same scene—termed "good views"—with bounded misalignment otherwise. They demonstrate that next-token prediction constitutes a JEPA with "bad views" (parts of a single document), explaining why language models align with vision only through massive scale. Empirical validation spans 145k captions to 940M web-image texts, revealing a scaling law of alignment predicting unseen runs within seed noise. Good views require 42 times fewer training samples than next-token prediction on the same corpus to achieve equivalent alignment. A 109M-parameter text JEPA aligns with DINOv2 and CLIP image encoders better than all evaluated language models up to 72B parameters and outperforms CLIP's text encoder. A single rotation of separately trained encoders yields an image captioner achieving CIDEr 0.838 on COCO. On a pair-free benchmark, the text JEPA surpasses all reported zero-shot language-model accuracies up to 8B parameters, reaching 64% top-5 on ImageNet-100 versus a 56% maximum. The results establish positive views, not scale alone, as the critical lever for platonic representations.
Read original
huggingface/daily-papers