Deep SPI: Safe Policy Improvement via World Models
TLDR
DeepSPI provides theoretical guarantees for safe policy improvement in online RL using world models, achieving strong results on ALE-57.
Reasoning
The paper's strength lies in its theoretical framework linking prediction losses to representation quality for online SPI, with empirical validation on ALE-57. Weaknesses include limited evaluation to Atari games and potential scalability issues to more complex environments.
Read-first score
Read-first score 53.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 32.
Field roles
Rank sensitivity
Stability: volatile; rank range: 345.
Keyword Scores
Deep Analysis
Innovations
- Theoretical framework for safe policy improvement in general online settings with world models and representation learning
- Online deep analogues of classical SPI theorems from offline RL literature
- DeepSPI algorithm that couples local transition and reward prediction losses with regularised policy updates
Methodology
The paper develops a theoretical framework linking transition and reward prediction losses to representation quality, showing that restricting policy updates to a well-defined neighborhood ensures monotonic improvement and convergence. Based on this, they propose DeepSPI, an on-policy algorithm that uses world model losses and regularization to constrain policy updates. The method is evaluated on the ALE-57 benchmark against baselines including PPO and DeepMDPs.
Key Results
DeepSPI matches or exceeds strong baselines (PPO and DeepMDPs) on the ALE-57 benchmark while retaining theoretical guarantees.