DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

arXiv 26.2 2026 70.7 method

TLDR

DreamDojo is a foundation world model trained on 44k hours of human videos, enabling dexterous robot control via latent actions and real-time simulation.

Reasoning

The paper introduces a large-scale pretraining approach using human videos to learn world dynamics, with a novel continuous latent action mechanism to handle missing action labels. Strengths include the scale of data and real-time distillation, but weaknesses are the reliance on post-training on small robot datasets and limited evaluation details in the abstract.

Read-first score

Read-first score 70.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 56.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,dataset,evaluation

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset,github

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 102.

Keyword Scores

world model

generative world model

world dynamics prediction

world simulator

interactive world model

video world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Learning diverse interactions and dexterous controls from 44k hours of egocentric human videos, the largest video dataset for world model pretraining
Introducing continuous latent actions as unified proxy actions to address scarcity of action labels and enhance interaction knowledge transfer from unlabeled videos
Devising a distillation pipeline that accelerates DreamDojo to real-time speed of 10.81 FPS and improves context consistency
Enabling applications such as live teleoperation, policy evaluation, and model-based planning based on generative world models

Methodology

DreamDojo is a foundation world model pretrained on 44k hours of egocentric human videos covering diverse daily scenarios. It uses continuous latent actions as proxy actions to transfer interaction knowledge from unlabeled videos, then post-trains on small-scale target robot data. A distillation pipeline is employed to achieve real-time inference at 10.81 FPS.

Key Results

After post-training, DreamDojo demonstrates strong physics understanding and precise action controllability, and the distillation pipeline achieves real-time speed of 10.81 FPS. Systematic evaluation on multiple challenging out-of-distribution benchmarks verifies its significance for simulating open-world, contact-rich tasks.

Resources

Paper Project