Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models

arXiv 25.6 2025 53.8 method

TLDR

VLMs struggle with forward dynamics prediction but use inverse dynamics to bootstrap world models, achieving competitive image editing performance.

Reasoning

The paper presents a novel approach leveraging the asymmetry between inverse and forward dynamics prediction to bootstrap world models, with empirical validation on Aurora-Bench. However, the evaluation is limited to action-centric image editing, and the method's generality to full world models is not demonstrated.

Read-first score

Read-first score 53.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 32.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,result

Topical relevance 42%

45.7

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 344.

Keyword Scores

world model

world dynamics prediction

video world model

generative world model

interactive world model

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Identifying asymmetry in multimodal grounding: inverse dynamics prediction (IDP) is significantly easier than forward dynamics prediction (FDP) for VLMs.
Bootstrapping FDP from IDP via two strategies: weakly supervised learning from synthetic data (using IDP to annotate actions for unlabeled frame pairs) and inference-time verification (using IDP as a reward model to score multiple FDP samples).
Achieving competitive performance with state-of-the-art image editing models on Aurora-Bench using general-purpose VLMs, with improvements of 7-13% according to GPT4o-as-judge.

Methodology

The paper investigates whether vision-language models (VLMs) can perform forward dynamics prediction (FDP) — predicting future image from previous observation and action in language. They find VLMs struggle, but identify that inverse dynamics prediction (IDP) — captioning the action between frames — is easier. They use IDP to bootstrap FDP via two strategies: (1) weakly supervised learning, where IDP annotates actions for unlabeled video frame pairs to expand training data for FDP, and (2) inference-time verification, where IDP assigns rewards to multiple FDP samples to guide search. Evaluation is conducted on the Aurora-Bench action-centric image editing task using two families of VLMs.

Key Results

The best model achieves performance competitive with state-of-the-art image editing models, improving on them by a margin between 7% and 13% according to GPT4o-as-judge, and achieves the best average human evaluation across all subsets of Aurora-Bench.

Limitations

VLMs struggle to generate physically plausible transitions between frames from instructions.
The approach relies on the asymmetry that IDP is easier than FDP, which may not hold for all multimodal foundation models or tasks.
Evaluation is limited to the Aurora-Bench benchmark and two VLM families, so generalizability to other domains or models is not established.

Resources

Paper