Transformers are Sample Efficient World Models

ICLR 23 Oral 2023 68.6 method

TLDR

IRIS uses a discrete autoencoder and autoregressive Transformer as a world model for sample-efficient deep RL, achieving state-of-the-art on Atari 100k.

Reasoning

The paper presents a novel combination of a discrete autoencoder and Transformer for world modeling, demonstrating strong empirical results on the Atari 100k benchmark. However, the evaluation is limited to simulated Atari games, and the abstract does not mention real-world experiments or broader applicability.

Read-first score

Read-first score 68.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 51.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,github

Topical relevance 42%

72.9

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Recency 8%

65.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2023

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 192.

Keyword Scores

world model

world dynamics prediction

model-based reinforcement learning world model

generative world model

world simulator

interactive world model

video world model

Deep Analysis

Innovations

Introduces IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer
Achieves state-of-the-art sample efficiency on the Atari 100k benchmark with only two hours of gameplay equivalent
Outperforms humans on 10 out of 26 games without using lookahead search

Methodology

IRIS is a model-based reinforcement learning agent that learns a world model consisting of a discrete autoencoder for encoding observations and an autoregressive Transformer for modeling environment dynamics. The agent is trained on the Atari 100k benchmark, using the equivalent of two hours of gameplay, and evaluated by mean human normalized score.

Key Results

IRIS achieves a mean human normalized score of 1.046 on the Atari 100k benchmark, outperforming humans on 10 out of 26 games and setting a new state of the art for methods without lookahead search.

Resources

Paper Code