WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens

arXiv 24.1 2024 69.3 method

TLDR

WorldDreamer proposes a general world model for video generation by predicting masked visual tokens, enabling diverse video tasks across natural and driving scenes.

Reasoning

The paper introduces a novel approach to world modeling using masked token prediction, inspired by LLMs, and demonstrates versatility across multiple video generation tasks. However, the abstract lacks quantitative results and comparisons, and the claim of 'general' world model is only supported by two scenario types.

Read-first score

Read-first score 69.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 50.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=github

Topical relevance 42%

71.4

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=experiment,result

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 93.

Keyword Scores

world model

generative world model

video world model

world dynamics prediction

interactive world model

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Proposes a general world model for video generation that is not confined to specific scenarios like gaming or driving
Frames world modeling as an unsupervised visual sequence modeling challenge by mapping visual inputs to discrete tokens and predicting masked tokens, inspired by large language models
Incorporates multi-modal prompts to facilitate interaction within the world model

Methodology

WorldDreamer models world dynamics by mapping visual inputs to discrete tokens and predicting masked tokens in an unsupervised manner, akin to masked language modeling in LLMs. It incorporates multi-modal prompts to enable interactive generation. The model is trained and evaluated on diverse scenarios including natural scenes and driving environments, with tasks such as text-to-video, image-to-video, and video editing.

Key Results

WorldDreamer excels in generating videos across different scenarios, including natural scenes and driving environments, and demonstrates versatility in text-to-video conversion, image-to-video synthesis, and video editing tasks.

Resources

Paper Code