⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets

arXiv 25.4 2025 79.3 method

TLDR

UWM couples video and action diffusion in a unified transformer for pretraining on large robotic datasets, enabling policy and dynamics learning.

Reasoning

The paper presents a novel framework that integrates video and action diffusion, demonstrating strong empirical results on both simulated and real-world tasks. Its strengths include leveraging action-free video data and achieving better generalization than imitation learning; a weakness is that the abstract does not detail limitations or comparisons to other world model approaches.

Read-first score

Read-first score 79.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 56.

Recency 8%
86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Reproducibility 25%
85

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,dataset,github

Topical relevance 42%
80

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%
70

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset,experiment,result

Field roles

FrontierMethodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 4.

Keyword Scores

world model
10
generative world model
9
video world model
9
interactive world model
8
world dynamics prediction
8
world simulator
7
model-based reinforcement learning world model
5

Deep Analysis

Innovations

  • Unified framework coupling video and action diffusion within a single transformer architecture with independent diffusion timesteps for each modality
  • Flexible representation of policy, forward dynamics, inverse dynamics, and video generator by controlling modality-specific diffusion timesteps
  • Effective pretraining on large-scale multitask robot datasets using both dynamics and action predictions, leading to more generalizable and robust policies than imitation learning
  • Learning from action-free video data through independent control of modality-specific diffusion timesteps, improving finetuned policy performance

Methodology

Unified World Models (UWM) integrates an action diffusion process and a video diffusion process within a unified transformer architecture, where independent diffusion timesteps govern each modality. By controlling each diffusion timestep, UWM can flexibly represent a policy, forward dynamics, inverse dynamics, or a video generator. The model is pretrained on large-scale multitask robot datasets with both dynamics and action predictions, and can also leverage action-free video data via independent timestep control.

Key Results

UWM enables effective pretraining on large-scale multitask robot datasets, resulting in more generalizable and robust policies than imitation learning. Additionally, it facilitates learning from action-free video data through independent control of modality-specific diffusion timesteps, further improving the performance of finetuned policies.

Resources

PaperCode

Tags