Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction

arXiv 24.8 2024 44 method

TLDR

Proposes hierarchical object-centric abstraction for efficient exploration and discriminative world model learning in RL, outperforming baselines in 2D environments.

Reasoning

Strengths include a clear hierarchical abstraction that simplifies dynamics and enables efficient exploration, transfer, and long-horizon planning. Weaknesses are the limitation to 2D crafting and MiniHack environments and reliance on predefined object mappings, with no real-world validation.

Read-first score

Read-first score 44, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 34.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

48.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=none

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

Candidate

Rank sensitivity

Stability: volatile; rank range: 293.

Keyword Scores

world model

model-based reinforcement learning world model

world dynamics prediction

world simulator

interactive world model

generative world model

video world model

Deep Analysis

Innovations

Hierarchical object-centric abstraction that models items at a higher level of state abstraction than pixels and attribute changes at a higher level of temporal abstraction than primitive actions
Discriminative world model that simplifies transition dynamics and enables efficient prediction of future abstract states
Count-based intrinsic reward for efficient exploration within the abstract state space
Zero-shot and few-shot transfer across item types and environments
Long-horizon planning capability using the learned abstract model

Methodology

The paper proposes a fully model-based reinforcement learning algorithm that uses an object-centric abstraction to represent states as items and their attributes. It learns a discriminative world model that predicts future abstract states, employs a count-based intrinsic reward to guide exploration, and plans to reach any discovered abstract states. The object mapping is learned via supervised learning, and low-level object-perturbing policies are learned via reinforcement learning.

Key Results

The model significantly outperforms state-of-the-art low-level methods (without abstraction) as well as performant model-free and model-based methods using the same abstraction across a suite of 2D crafting and MiniHack environments. It demonstrates efficient single-task solving, zero-shot and few-shot transfer, and long-horizon planning.

Resources

Paper