The authors investigated how multimodal language models (MLLMs) internally represent spatial information by identifying a linear subspace, termed S‑Space, in which the activations of object‑token embeddings encode continuous coordinates along horizontal, vertical, and distance axes. They derived these axes by computing gradients of spatial‑answer scores with respect to object‑token activations, showing that S‑Space exists intrinsically—it is readable even when the model is posed non‑spatial questions and is not merely an artifact of particular prompts or training data. Experiments demonstrated that perception stages write object locations into S‑Space, allowing direct readout for accurate spatial judgments, while internal reasoning stages transform the subspace to simulate viewpoint changes. Causal interventions that perturbed S‑Space altered spatial responses while leaving non‑spatial outputs largely intact, confirming selective access by verbal generation. Case studies revealed that S‑Space can assign locations to unseen entities and to the model’s own observer perspective, linking physical directions to abstract language. Using S‑Space as a diagnostic tool, the authors observed systematic coupling between cardinal and image‑relative reference frames via direction‑word readouts. A perception‑computation decomposition further exposed a bottleneck: although MLLMs can construct robust spatial representations in S‑Space, they struggle with the spatial transformations needed for reliable reasoning across viewpoint changes.
Read original
huggingface/daily-papers