Planning to Explore via Self-Supervised World Models

ICML 20 2020 59.4 method

TLDR

Plan2Explore uses self-supervised world models for planning-based exploration, enabling zero/few-shot task adaptation, outperforming prior methods on image-based control tasks.

Reasoning

Strengths include a novel exploration strategy using planning for future novelty and strong empirical results nearly matching an oracle. Weaknesses are the lack of real-world validation and reliance on simulated environments.

Read-first score

Read-first score 59.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 43.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,github

Topical relevance 42%

61.4

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Recency 8%

42.4

Uses a gentle age decay so recent papers surface without erasing older foundations. 2020

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=none

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 308.

Keyword Scores

world model

model-based reinforcement learning world model

world dynamics prediction

world simulator

generative world model

interactive world model

video world model

Deep Analysis

Innovations

Planning to seek out expected future novelty during exploration, rather than retrospectively computing novelty after reaching observations
Self-supervised exploration that enables zero-shot or few-shot adaptation to multiple downstream tasks without task-specific interaction or reward supervision

Methodology

Plan2Explore uses a self-supervised world model to plan actions that maximize expected future novelty, enabling efficient exploration. After exploration, the agent adapts to downstream tasks in a zero or few-shot manner without additional task-specific interaction. The method is evaluated on challenging control tasks from high-dimensional image inputs.

Key Results

Plan2Explore outperforms prior self-supervised exploration methods and almost matches the performance of an oracle that has access to rewards, all without any training supervision or task-specific interaction.

Resources

Paper Code