STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning

NIPS 23 2023 80.2 method

TLDR

STORM combines Transformers and VAEs for efficient world models in RL, achieving 126.7% human performance on Atari 100k with fast training.

Reasoning

The paper presents a novel architecture that effectively integrates sequence modeling and stochasticity, achieving state-of-the-art results on a standard benchmark with notable training efficiency. However, the evaluation is limited to the Atari domain, and the abstract does not discuss real-world applications or limitations beyond simulation.

Read-first score

Read-first score 80.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 58.

Topical relevance 42%

82.9

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,github

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,metric,result

Recency 8%

65.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2023

Field roles

Methodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 47.

Keyword Scores

world model

model-based reinforcement learning world model

generative world model

world simulator

world dynamics prediction

interactive world model

video world model

Deep Analysis

Innovations

Combining Transformer architecture with variational autoencoder for stochastic world modeling in model-based reinforcement learning
Achieving state-of-the-art mean human performance of 126.7% on Atari 100k benchmark without lookahead search techniques
Efficient training requiring only 4.3 hours on a single NVIDIA GeForce RTX 3090 for 1.85 hours of real-time interaction

Methodology

STORM is a world model architecture that integrates the sequence modeling and generation capabilities of Transformers with the stochastic nature of variational autoencoders. The world model is trained via self-supervised learning on visual input from the environment, and the agent's policy is improved by leveraging imagined trajectories from the model. Evaluation is performed on the Atari 100k benchmark, comparing against state-of-the-art methods that do not use lookahead search, with metrics including mean human performance and training time.

Key Results

STORM achieves a mean human performance of 126.7% on the Atari 100k benchmark, setting a new record among methods without lookahead search. Training an agent with 1.85 hours of real-time interaction on a single RTX 3090 requires only 4.3 hours, demonstrating improved efficiency over prior approaches.

Resources

Paper Code