ViPRA: Video Prediction for Robot Actions
TLDR
ViPRA turns video prediction models into robot policies by predicting future visuals and latent actions, finetuning with few demonstrations.
Reasoning
The paper presents a novel pretraining-finetuning framework that leverages actionless videos for robot control, showing strong empirical gains on both simulated and real-world tasks. However, it relies on teleoperated demonstrations for finetuning and does not explicitly claim to be a world model, limiting direct relevance to the specified keywords.
Read-first score
Read-first score 44.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 18.
Field roles
Rank sensitivity
Stability: volatile; rank range: 371.
Keyword Scores
Deep Analysis
Innovations
- ViPRA: a pretraining-finetuning framework that learns continuous robot control from actionless videos by predicting future visual observations and motion-centric latent actions as intermediate representations
- Training latent actions using perceptual losses and optical flow consistency to ensure physically grounded behavior
- Chunked flow matching decoder that maps latent actions to robot-specific continuous action sequences using only 100-200 teleoperated demonstrations
- Explicitly models both what changes and how, unlike prior latent action works that treat pretraining as autoregressive policy learning
Methodology
ViPRA first pretrains a video-language model to predict future visual observations and motion-centric latent actions, using perceptual losses and optical flow consistency. For downstream control, a chunked flow matching decoder is fine-tuned with 100-200 teleoperated demonstrations to map latent actions to robot-specific continuous action sequences.
Key Results
ViPRA outperforms strong baselines with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks.