MODIP: Efficient Model-Based Optimization for Diffusion Policies
TLDR
MODIP uses a world model and MPC to efficiently fine-tune diffusion policies offline-to-online, outperforming baselines on D4RL and RoboMimic.
Reasoning
The paper presents a novel framework that addresses the challenge of RL fine-tuning for diffusion policies by leveraging a world model for trajectory generation and policy adaptation. Strengths include efficiency gains via terminal state value and policy-independent critics; weaknesses include limited evaluation to simulated tasks and potential scalability issues.
Read-first score
Read-first score 47, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 20.
Field roles
Rank sensitivity
Stability: volatile; rank range: 402.
Keyword Scores
Deep Analysis
Innovations
- MODIP framework for offline-to-online fine-tuning of diffusion policies using a world model to guide policy adaptation
- Use of model predictive control (MPC) to generate high-quality trajectories within the world model as supervised targets for fine-tuning
- Terminal state value instead of policy-dependent state-action value to reduce MPC inference time
- Policy-independent TD targets for training critics to reduce training time
Methodology
MODIP leverages a world model to guide policy adaptation of diffusion policies. It uses model predictive control (MPC) to generate high-quality trajectories within the world model, which serve as supervised targets for fine-tuning the diffusion policy. To improve efficiency, MODIP employs a terminal state value instead of a policy-dependent state-action value for MPC planning and trains critics with policy-independent TD targets.
Key Results
Experiments on D4RL (MuJoCo, Kitchen) and RoboMimic tasks show that MODIP improves diffusion policies beyond behavioral cloning and is competitive with or outperforms diffusion policy RL fine-tuning methods and strong model-based baselines such as TD-MPC2.