Do multimodal models imagine electric sheep?
TLDR
Multimodal models develop mental imagery when solving spatial puzzles, forming an imperfect visual world model without explicit supervision.
Reasoning
The paper presents a novel finding that VLMs encode visual intermediate states as a byproduct of action prediction, supported by empirical gains from integrating visual tokens. However, the claim is limited to specific puzzle tasks and lacks comparison to explicit world model training.
Read-first score
Read-first score 50.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 43.
Field roles
Rank sensitivity
Stability: volatile; rank range: 329.
Keyword Scores
Deep Analysis
Innovations
- Demonstrates that large multimodal models develop mental imagery as a byproduct of learning to select correct actions, without explicit visual supervision.
- Proposes two methods to sharpen and utilize the mental images formed by the model.
- Shows that integrating as few as sixteen visual tokens per step into the chain of thought improves solve rate from 83% to 89%.
Methodology
The authors fine-tune a Qwen3.5 VLM on twelve diverse visual reasoning tasks (tangram, jigsaw, sokoban, 3D mental rotation, rush hour, etc.) that require geometry, spatial relationships, and action consequences. The model is supervised to predict the open-loop sequence of actions from an initial state, and activations are analyzed to assess encoding of intermediate visual states.
Key Results
The model's activations after each action encode meaningful visual information about the intermediate state, indicating formation of an imperfect visual world model. Adding sixteen visual tokens per step into the chain of thought raises the average solve rate from 83% to 89%, with strong gains on reasoning-heavy tasks like jigsaw and 3D mental rotation.