⌁ Awesome Auto Research Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research

arXiv 2026 53.3 method

TLDR

Introduces AutoWorldModel-Bench, a closed-loop benchmark where frontier coding agents autonomously improve world-model starters across eight game environments, showing research-style modifications dominate.

Reasoning

Strengths: novel benchmark for open-ended research, isolates dynamics from perception, and provides strong empirical results across 64 sessions. Weaknesses: limited to game environments and structured-state representations, with no baseline or metric details in the abstract.

Read-first score

Read-first score 53.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 59.

Recency 8%
100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%
60

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation

Topical relevance 42%
49.2

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Reproducibility 25%
38

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 47.

Keyword Scores

autonomous research agent
8
automated research
8
research automation
8
AI scientist
7
AI for scientific research
7
automated scientific discovery
6
scientific discovery agent
6
automated experimentation
5
experiment design agent
4
literature review agent
0
survey generation
0
paper writing agent
0

Deep Analysis

Innovations

  • Closed-loop benchmark for AI coding agents to autonomously improve world models as open-ended research
  • Unified structured-state representation using ground-truth entity state across eight game environments
  • Isolation of dynamics modeling from perception via shared tensor format, enabling minutes-per-run iteration
  • Evaluation of frontier coding agents on research-style modifications rather than hyperparameter tuning

Methodology

The benchmark provides a world-model starter and a fixed compute budget to frontier coding agents (Codex-5.4 and Claude Opus 4.6) across eight game environments. All environments use a unified structured-state representation—ground-truth entity state consumed through a shared tensor format—to isolate dynamics modeling from perception. Improvement is measured by whether agents produce a winning edit, categorized as a nontrivial research modification or a hyperparameter tweak.

Key Results

Agents improved the starter in 63 out of 64 sessions; in 91% of sessions the winning edit was a nontrivial research-style modification (new objective, representation, rollout procedure, or architectural change), indicating genuine research behavior.

Resources

Paper

Tags