Researchers have transformed the Platonic Representation Hypothesis into a formal theorem using Joint-Embedding Predictive Architectures (JEPAs). The study proves that models trained on different modalities converge to identical world representations up to rotation when trained on synchronized observations of the same scene. A key finding reveals that next-token prediction functions as a JEPA with "bad views," explaining why language models align with vision only through scale.
The theory was validated across a dataset spanning 145k captions to 940 million web images, uncovering a scaling law of alignment. Crucially, good views require 42 times fewer training samples than next-token prediction to achieve equivalent alignment. A 109-million-parameter text JEPA outperformed larger language models, aligning better with vision encoders.
As a practical demonstration, a single rotation enabled separately trained encoders to function as an image captioner, achieving a CIDEr score of .838 on COCO. Furthermore, on a pair-free benchmark, the text JEPA surpassed zero-shot accuracy of language models up to 8B parameters, with 64% accuracy against a maximum of 56% top-5 on ImageNet-100. The findings establish that positive views, not scale alone, are the essential lever for building platonic representations.
Source: Hacker News · Summarized by HeadlinesBriefing