Bounded Exploration with World Model Uncertainty in Soft Actor-Critic Reinforcement Learning Algorithm

arXiv 24.12 2024 36 method

TLDR

Bounded exploration integrating soft and intrinsic motivation improves SAC and its model-based extension, achieving top scores in 6/8 experiments.

Reasoning

The paper presents a novel exploration method that combines soft and intrinsic motivation, showing clear empirical improvements over baselines. However, the abstract lacks details on how world model uncertainty is used, and no real-world experiments are mentioned, limiting the strength of the claims.

Read-first score

Read-first score 36, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 8.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,experiment,metric

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

11.4

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Methodology anchor

Rank sensitivity

Stability: volatile; rank range: 103.

Keyword Scores

world model

model-based reinforcement learning world model

world simulator

generative world model

interactive world model

video world model

world dynamics prediction

Deep Analysis

Innovations

Bounded exploration method integrating soft and intrinsic motivation exploration
Improvement of Soft Actor-Critic algorithm's performance and its model-based extension's converging speed
Alternative method to introduce intrinsic motivations when the original reward function has strict meanings

Methodology

The methodology integrates soft exploration with intrinsic motivation based on world model uncertainty within the Soft Actor-Critic framework. Specific model design, data, training setup, baselines, and metrics are not detailed in the abstract.

Key Results

Bounded exploration achieved the highest score in 6 out of 8 experiments and improved the converging speed of the Soft Actor-Critic algorithm and its model-based extension.

Resources

Paper