World Models Can Leverage Human Videos for Dexterous Manipulation
TLDR
DexWM predicts future latent states for dexterous manipulation using finger keypoints from egocentric videos, outperforming prior models on real robot tasks.
Reasoning
The paper introduces a novel world model for dexterous manipulation, leveraging large-scale human video data and an auxiliary hand consistency loss. Strengths include strong zero-shot transfer results on real hardware; weaknesses are limited detail on the latent state representation and lack of explicit comparison to model-based RL baselines.
Read-first score
Read-first score 56.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 42.
Field roles
Rank sensitivity
Stability: volatile; rank range: 304.
Keyword Scores
Deep Analysis
Innovations
- DexWM, a dexterous interaction world model that predicts future latent states conditioned on past states and dexterous actions
- Representing actions using finger keypoints extracted from egocentric videos to leverage over 900 hours of human and non-dexterous robot data
- Auxiliary hand consistency loss to enforce accurate hand configurations, addressing insufficiency of predicting visual features alone
Methodology
DexWM is a world model that takes past states and dexterous actions (represented as finger keypoints from egocentric videos) to predict future latent states. It is trained on over 900 hours of human and non-dexterous robot data and incorporates an auxiliary hand consistency loss. Evaluation is performed on future-state prediction and zero-shot transfer to unseen skills on a Franka Panda arm with an Allegro gripper, comparing against prior world models and Diffusion Policy.
Key Results
DexWM outperforms prior world models conditioned on text, navigation, or full-body actions in future-state prediction and demonstrates strong zero-shot transfer to unseen skills, surpassing Diffusion Policy by over 50% on average across grasping, placing, and reaching tasks.