PWM: Policy Learning with Large World Models
TLDR
PWM uses well-regularized world models to enable efficient first-order policy optimization, achieving strong multi-task continuous control results.
Reasoning
The paper presents a novel insight that regularization smooths world model landscapes, enabling gradient-based policy extraction. Strengths include strong empirical results across many tasks and fast optimization. Weaknesses include reliance on offline pre-training and lack of explicit real-world validation.
Read-first score
Read-first score 64.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 46.
Field roles
Rank sensitivity
Stability: volatile; rank range: 164.
Keyword Scores
Deep Analysis
Innovations
- Reveals that well-regularized world models generate smoother optimization landscapes than actual dynamics, enabling effective first-order optimization
- Introduces PWM, a model-based RL algorithm that pre-trains a world model on offline data and extracts policies via first-order optimization in under 10 minutes per task
- Scales to 80-task settings with up to 27% higher rewards than existing baselines without relying on costly online planning
Methodology
PWM is a model-based RL algorithm for continuous control. It first pre-trains a world model on offline data, then extracts policies from the learned model using first-order (gradient-based) optimization. The approach is evaluated on tasks with up to 152 action dimensions and in an 80-task multi-task setting, comparing against methods using ground-truth dynamics and other baselines.
Key Results
PWM solves tasks with up to 152 action dimensions and outperforms methods that use ground-truth dynamics. In an 80-task setting, it achieves up to 27% higher rewards than existing baselines without online planning.