DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

arXiv 24.11 2024 52.9 method

TLDR

DINO-WM uses pre-trained DINOv2 features to learn world models from offline trajectories, enabling zero-shot task-agnostic planning.

Reasoning

The paper introduces a novel method that leverages pre-trained visual features to avoid pixel reconstruction, enabling offline training and zero-shot planning across diverse environments. Strengths include task-agnostic reasoning and strong empirical results, but limitations include reliance on DINOv2 features and evaluation only in simulated environments.

Read-first score

Read-first score 52.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 49.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=none

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

Candidate

Rank sensitivity

Stability: volatile; rank range: 488.

Keyword Scores

world model

world dynamics prediction

interactive world model

model-based reinforcement learning world model

generative world model

video world model

world simulator

Deep Analysis

Innovations

Using pre-trained DINOv2 spatial patch features for world models without reconstructing the visual world
Learning from offline behavioral trajectories by predicting future patch features
Zero-shot planning at test time across diverse tasks without expert demonstrations, reward modeling, or pre-learned inverse models

Methodology

DINO-WM leverages spatial patch features pre-trained with DINOv2 to model visual dynamics. It is trained on offline, pre-collected trajectories to predict future patch features, and at test time optimizes action sequences to match goal features, enabling task-agnostic planning.

Key Results

DINO-WM achieves zero-shot behavioral solutions on six environments including arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle scenarios, outperforming prior state-of-the-art work.

Resources

Paper