Transformer-based World Models Are Happy with 100k Interactions

ICLR 23 2023 73.9 method

TLDR

A transformer-based world model (TWM) achieves sample-efficient RL on Atari 100k by autoregressively processing states, actions, and rewards.

Reasoning

The paper introduces a novel transformer architecture (Transformer-XL) for world modeling that directly attends to past states, improving long-term dependency learning. Its strength lies in achieving state-of-the-art results on the Atari 100k benchmark with only 100k interactions, but it is limited to simulated environments and lacks real-world validation.

Read-first score

Read-first score 73.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 55.

Topical relevance 42%

78.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=github

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation

Recency 8%

65.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2023

Field roles

Methodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 110.

Keyword Scores

world model

model-based reinforcement learning world model

world dynamics prediction

generative world model

interactive world model

world simulator

video world model

Deep Analysis

Innovations

Applying transformer architecture to world models in reinforcement learning with autoregressive modeling of latent states, actions, and rewards
Using Transformer-XL to capture long-term dependencies in world models while maintaining computational efficiency
Feeding all three modalities (latent states, actions, rewards) into the transformer to enable flexible attention across time steps

Methodology

The paper proposes a transformer-based world model (TWM) that processes sequences of latent states, actions, and rewards autoregressively using the Transformer-XL architecture. The model is trained on real-world episodes from the Atari 100k benchmark and used to generate synthetic experience for training a reinforcement learning policy.

Key Results

TWM outperforms previous model-free and model-based reinforcement learning algorithms on the Atari 100k benchmark, demonstrating sample-efficient world modeling with only 100k interactions.

Limitations

Evaluation is limited to the Atari 100k benchmark; generalization to other environments or tasks is not demonstrated
The abstract does not discuss potential computational costs or scalability of the transformer-based approach

Resources

Paper Code