SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation
TLDR
SimuDICE improves offline RL by iteratively refining policies using world model simulations adjusted by DICE and confidence estimates.
Reasoning
The paper introduces a novel framework combining world model updates with DICE estimation to address distribution mismatch in offline RL. Strengths include clear methodology and empirical validation showing comparable performance with fewer data and planning steps. Weaknesses are limited discussion of limitations and potential scalability issues.
Read-first score
Read-first score 52.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 44.
Field roles
Rank sensitivity
Stability: volatile; rank range: 404.
Keyword Scores
Deep Analysis
Innovations
- Iterative refinement of offline policy using synthetic experiences from a world model with sampling probabilities adjusted by stationary Distribution Correction Estimation (DICE) and model confidence
- Balancing experiences similar to frequently encountered ones with those having distribution mismatch to guide policy improvement
- Reducing required pre-collected experiences and planning steps while maintaining comparable performance
Methodology
SimuDICE learns a world model from offline data to generate synthetic experiences, then adjusts the sampling probabilities of state-action pairs based on stationary DICE and estimated model confidence. This guides iterative policy improvement by balancing experiences that are similar to frequent ones with those that have distribution mismatch.
Key Results
SimuDICE achieves performance comparable to existing offline RL algorithms while requiring fewer pre-collected experiences and planning steps, and remains robust across varying data collection policies.