Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
TLDR
Proposes MBDPO, a framework unifying search and policy optimization via diffusion policy in world models to address structural misalignment in model-based RL.
Reasoning
The paper identifies a novel bottleneck (structural misalignment) and proposes a diffusion-based solution, which is a strength. However, the abstract is cut off, limiting full assessment of results and limitations; it lacks real-world experiments.
Read-first score
Read-first score 55.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 40.
Field roles
Rank sensitivity
Stability: volatile; rank range: 331.
Keyword Scores
Deep Analysis
Innovations
- Identifying structural misalignment between search and value learning as a critical bottleneck in scaling world-model RL
- Proposing MBDPO that unifies search and policy optimization through diffusion policy representations
- Reformulating policy optimization as a diffusion process over searched trajectories in latent world models
- Extracting implicit energy function from dataset to anchor policy and refine score field for policy optimization
Methodology
MBDPO uses diffusion policy representations to unify search and policy optimization. Instead of constructing an explicit planner over a learned world model, it reformulates policy optimization as a diffusion process over searched trajectories in latent world models. It extracts an implicit energy function from the collected dataset that anchors the policy, enabling refinement of the score field for policy optimization while mitigating misalignment between search and value learning.
Key Results
In the offline regime, pretraining on large-scale datasets shows consistent and monotonic performance gains with increasing model capacity. The method is evaluated across multi-task offline pretraining, online learning, and offline-to-online fine-tuning.