Planning with Reasoning using Vision Language World Model
TLDR
Introduces VLWM, a vision-language world model for planning that achieves state-of-the-art on Visual Planning for Assistance benchmarks.
Reasoning
Strengths: novel integration of vision-language model for world modeling with system-1 and system-2 planning, strong empirical results on multiple benchmarks. Weaknesses: limited details on methodology and limitations in abstract; reliance on LLM self-refine and Tree of Captions may be computationally expensive.
Read-first score
Read-first score 63.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 52.
Field roles
Rank sensitivity
Stability: volatile; rank range: 255.
Keyword Scores
Deep Analysis
Innovations
- Training a foundation model for language-based world modeling on natural videos
- Using iterative LLM Self-Refine conditioned on Tree of Captions to extract trajectory targets
- Learning both an action policy and a dynamics model for reactive system-1 and reflective system-2 planning
- Self-supervised critic model to measure semantic distance between hypothetical future states and goal state
Methodology
The Vision Language World Model (VLWM) first infers overall goal achievements from visual observations, then predicts a trajectory of interleaved actions and world state changes. Targets are extracted via iterative LLM Self-Refine conditioned on compressed future observations represented by Tree of Captions. The model learns both an action policy (for system-1 plan decoding) and a dynamics model (for system-2 planning via cost minimization), with cost measured by a self-supervised critic model.
Key Results
VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on benchmark evaluations and PlannerArena human evaluations, where system-2 improves Elo score by +27% over system-1. It also outperforms strong VLM baselines on RoboVQA and WorldPrediction benchmarks.