⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

RoboDreamer: Learning Compositional World Models for Robot Imagination

ICML 24 2024 79.8 method, application

TLDR

RoboDreamer learns compositional world models by factorizing video generation from language, enabling generalization to unseen tasks and goals.

Reasoning

The paper introduces a novel compositional factorization of video generation for world models, leveraging language compositionality to achieve generalization to unseen combinations. Strengths include addressing a key limitation of text-to-video models and enabling multimodal goal specification. Weaknesses are that robot execution is only demonstrated in simulation, and real-world validation is limited to the RT-X dataset without physical robot results.

Read-first score

Read-first score 79.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 56.

Reproducibility 25%
81

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=dataset,github

Topical relevance 42%
80

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%
80

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,dataset,evaluation

Recency 8%
75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Field roles

Methodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 22.

Keyword Scores

world model
10
video world model
10
generative world model
9
world dynamics prediction
9
world simulator
8
interactive world model
6
model-based reinforcement learning world model
4

Deep Analysis

Innovations

  • Factorizing video generation by leveraging the natural compositionality of language to parse instructions into lower-level primitives
  • Conditioning a set of models on these primitives to generate videos, enabling compositional generalization to unseen combinations of objects and actions
  • Incorporating multimodal goals (natural language instructions and goal images) to specify desired videos

Methodology

RoboDreamer learns a compositional world model by parsing language instructions into a set of lower-level primitives, each conditioned on a separate video generation model. During inference, a new instruction is decomposed into previously seen primitives, and the corresponding models are combined to synthesize a video plan. The approach also supports additional goal images as multimodal conditioning.

Key Results

The approach successfully synthesizes video plans on unseen goals in the RT-X dataset, enables successful robot execution in simulation, and substantially outperforms monolithic baseline approaches to video generation.

Limitations

  • Generalization is limited to combinations of primitives seen during training; entirely novel primitives may not be supported
  • Relies on accurate parsing of natural language instructions into predefined lower-level primitives
  • Evaluation is conducted only in simulation; real-world robot execution is not demonstrated

Resources

PaperCode

Tags