DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing

ICLR 24 2024 43.6 method

TLDR

DreamSmooth improves model-based RL by smoothing reward predictions, achieving state-of-the-art on sparse-reward tasks.

Reasoning

The paper addresses a clear bottleneck in MBRL (reward prediction) with a simple, intuitive method. Strengths include strong empirical results on standard benchmarks; weaknesses are that the method's novelty is limited to reward smoothing and it does not introduce new world model architectures.

Read-first score

Read-first score 43.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 25.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark

Topical relevance 42%

35.7

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

Candidate

Rank sensitivity

Stability: volatile; rank range: 248.

Keyword Scores

model-based reinforcement learning world model

world model

world dynamics prediction

generative world model

world simulator

interactive world model

video world model

Deep Analysis

Innovations

Identifying reward prediction as a bottleneck in model-based reinforcement learning, especially for sparse rewards
Proposing DreamSmooth, a simple yet effective reward smoothing approach that predicts temporally-smoothed rewards instead of exact rewards
Achieving state-of-the-art performance on long-horizon sparse-reward tasks in both sample efficiency and final performance

Methodology

DreamSmooth modifies the reward prediction component of a model-based RL agent to predict a temporally-smoothed reward rather than the exact immediate reward. The approach is evaluated on long-horizon sparse-reward tasks and standard benchmarks (Deepmind Control Suite, Atari) against prior MBRL methods, measuring sample efficiency and final performance.

Key Results

DreamSmooth achieves state-of-the-art performance on long-horizon sparse-reward tasks in both sample efficiency and final performance, while maintaining performance on common benchmarks such as Deepmind Control Suite and Atari.

Limitations

The paper does not explicitly discuss limitations, but potential issues include sensitivity to the smoothing hyperparameter and reliance on the quality of the learned world model
The method's effectiveness may vary across different reward structures not tested in the benchmarks

Resources

Paper