DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models
TLDR
DreamPlan fine-tunes VLM planners via a video world model using RL, improving manipulation success without costly real-world interaction.
Reasoning
The paper introduces a novel framework that uses sub-optimal data to train a video world model, then fine-tunes VLMs within it via ORPO, addressing sample inefficiency and safety. Strengths include efficient grounding of VLMs; weaknesses include potential limited scope to deformable objects and reliance on video generation quality.
Read-first score
Read-first score 62.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 57.
Field roles
Rank sensitivity
Stability: volatile; rank range: 488.
Keyword Scores
Deep Analysis
Innovations
- Reinforcement fine-tuning of VLM planners via video world models, avoiding costly real-world interaction
- Using zero-shot VLM to collect exploratory data for training an action-conditioned video generation model
- Fine-tuning VLM planner within the 'imagination' of the video world model using Odds Ratio Policy Optimization (ORPO)
- Bridging the gap between semantic reasoning and physical grounding for robotic manipulation
Methodology
DreamPlan first leverages a zero-shot VLM to collect exploratory interaction data. This sub-optimal data is used to train an action-conditioned video generation model (video world model) that implicitly captures real-world physics. Then, the VLM planner is fine-tuned entirely within this video world model using Odds Ratio Policy Optimization (ORPO), avoiding costly physical rollouts.
Key Results
DreamPlan significantly improves manipulation success rates without large-scale real-world data collection, bridging the gap between semantic reasoning and physical grounding.