Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining
TLDR
DeFI decouples forward and inverse dynamics pretraining for robot learning, using video prediction and latent action inference, achieving state-of-the-art results on benchmarks and real-world tasks.
Reasoning
The paper presents a novel framework that separates visual forward dynamics (video prediction) and inverse dynamics (action inference) to leverage action-free videos, showing strong empirical results. However, the abstract does not detail limitations or comparisons to other world model approaches, and the term 'world model' is not explicitly used.
Read-first score
Read-first score 39, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 41.
Field roles
Rank sensitivity
Stability: volatile; rank range: 245.
Keyword Scores
Deep Analysis
Innovations
- Disentanglement of forward (video generation) and inverse (action prediction) dynamics pretraining to leverage action-free web videos
- General Forward Dynamics Model (GFDM) pretrained on diverse human and robot videos for future prediction
- General Inverse Dynamics Model (GIDM) trained via self-supervised learning to infer latent actions from unlabeled video transitions
- Integration of GFDM and GIDM into a unified architecture for end-to-end fine-tuning on downstream tasks
Methodology
DeFI pretrains two separate models: GFDM for forward dynamics (video generation) on human and robot videos, and GIDM for inverse dynamics (latent action inference) using self-supervised learning on unlabeled video transitions. The pretrained models are then integrated into a unified architecture and fine-tuned end-to-end on downstream robot tasks.
Key Results
DeFI achieves state-of-the-art performance with an average task length of 4.51 on CALVIN ABC-D, 51.2% success rate on SimplerEnv-Fractal, and 81.3% success rate in real-world deployment, significantly outperforming prior methods.