VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

NeurIPS 24 2024 64.5 method, application

TLDR

VidMan uses video diffusion to learn implicit dynamics from robot data, improving manipulation via two-stage training and outperforming baselines.

Reasoning

Strengths include a novel two-stage training mechanism that leverages video diffusion for dynamics awareness, achieving strong empirical results on CALVIN and OXE. Weaknesses are the narrow focus on robot manipulation and lack of explicit comparison to world model literature, with potential dataset-specific overfitting.

Read-first score

Read-first score 64.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 45.

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,benchmark,dataset,result

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

64.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,dataset

Field roles

Methodology anchor

Rank sensitivity

Stability: volatile; rank range: 84.

Keyword Scores

video world model

world dynamics prediction

world model

generative world model

world simulator

model-based reinforcement learning world model

interactive world model

Deep Analysis

Innovations

Two-stage training mechanism inspired by dual-process theory from neuroscience for robot manipulation
Pre-training on Open X-Embodiment dataset (OXE) for future visual trajectory prediction via video denoising diffusion to develop long-horizon dynamics awareness
Layer-wise self-attention adapter that transforms the video diffusion model into an efficient inverse dynamics model via parameter sharing

Methodology

VidMan employs a two-stage training process. First, it is pre-trained on the Open X-Embodiment dataset to predict future visual trajectories using a video denoising diffusion approach, learning implicit dynamics. Second, a layer-wise self-attention adapter is introduced to convert the model into an inverse dynamics model that predicts actions, leveraging the learned dynamics through parameter sharing.

Key Results

VidMan outperforms the state-of-the-art baseline GR-1 on the CALVIN benchmark with an 11.7% relative improvement and achieves over 9% precision gains on the OXE small-scale dataset.

Resources

Paper