Continual Reinforcement Learning by Planning with Online World Models

ICML 25 2025 52.9 method, theory

TLDR

Proposes online world models for continual RL, using Follow-The-Leader planning to avoid catastrophic forgetting, outperforming deep world models.

Reasoning

The paper introduces a novel approach with a theoretical regret bound and empirical results on a custom benchmark. However, the shallow model may limit scalability, and the benchmark is not standard, raising questions about generalizability.

Read-first score

Read-first score 52.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 39.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,result

Topical relevance 42%

55.7

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 396.

Keyword Scores

world model

world dynamics prediction

model-based reinforcement learning world model

world simulator

interactive world model

generative world model

video world model

Deep Analysis

Innovations

Proposing a Follow-The-Leader shallow model learned online for world dynamics in continual reinforcement learning
Using model predictive control with an online world model that is immune to catastrophic forgetting by construction
Introducing the FTL Online Agent (OA) that updates incrementally and plans solely based on the latest online model
Designing Continual Bench, a dedicated environment for evaluating continual reinforcement learning

Methodology

The paper learns a Follow-The-Leader shallow model online to capture world dynamics. It then uses model predictive control to plan actions for tasks specified by any reward functions. The agent, called FTL Online Agent (OA), updates incrementally and is evaluated on a custom environment called Continual Bench against deep world models with various continual learning techniques.

Key Results

The FTL Online Agent learns continuously to solve new tasks while not forgetting old skills, outperforming agents built on deep world models with various continual learning techniques.

Resources

Paper