Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
TLDR
CoWorld uses online simulators to improve offline visual RL by mitigating cross-domain discrepancies via collaborative world models.
Reasoning
The paper presents a novel approach combining offline and online RL using world models, addressing overfitting and overestimation bias. However, it relies on off-the-shelf simulators and lacks real-world validation, limiting generalizability.
Read-first score
Read-first score 60.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 35.
Field roles
Rank sensitivity
Stability: volatile; rank range: 273.
Keyword Scores
Deep Analysis
Innovations
- Building more flexible constraints for value estimation without impeding exploration of potential advantages
- Leveraging off-the-shelf RL simulators as a 'test bed' for offline policies to enable online interaction
- Introducing CoWorld, a model-based RL approach that mitigates cross-domain discrepancies in state and reward spaces for effective online-to-offline knowledge transfer
Methodology
CoWorld is a model-based reinforcement learning approach that uses off-the-shelf RL simulators as an online test bed for offline policies. It mitigates cross-domain discrepancies in state and reward spaces to enable effective knowledge transfer from online to offline settings. The method addresses overfitting in representation learning and overestimation bias in value estimation.
Key Results
CoWorld outperforms existing RL approaches by large margins on offline visual reinforcement learning tasks.