Benchmarking World-Model Learning

arXiv 25.10 2025 53.4 benchmark

TLDR

Proposes WorldTest protocol and AutumnBench benchmark to evaluate world models on multiple environment-level queries, showing humans outperform frontier models.

Reasoning

The paper addresses a clear gap in world-model evaluation by introducing diverse query families beyond next-frame prediction. Strengths include human comparison and systematic protocol; weaknesses are limited to grid-world environments and lack of real-world complexity.

Read-first score

Read-first score 53.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 28.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,experiment

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 410.

Keyword Scores

world model

interactive world model

world dynamics prediction

world simulator

model-based reinforcement learning world model

generative world model

video world model

Deep Analysis

Innovations

Proposing WorldTest, a protocol for evaluating world models using environment-level queries that test model generality across multiple query types (e.g., reachability, intervention effects).
Instantiating AutumnBench, a benchmark of 43 interactive grid-world environments and 129 tasks across three query families for both humans and learning agents.

Methodology

WorldTest is a protocol that evaluates agents on multiple environment-level queries (e.g., reachability, effects of interventions) that collectively assess model generality across query types. It is instantiated as AutumnBench, a benchmark of 43 interactive grid-world environments and 129 tasks across three query families. Experiments were conducted with 517 human participants and five frontier models, comparing their performance on these tasks.

Key Results

Humans substantially outperform the five frontier models on AutumnBench tasks. The performance gap is attributed to differences in exploration and belief updating.

Limitations

Only tested in grid-world environments, limiting generalizability to richer domains.
Limited to three query families, which may not cover all possible environment-level queries.
Current models significantly underperform humans, suggesting the benchmark may be challenging or that models require further development in exploration and belief updating.

Resources

Paper