WorldModelBench: Judging Video Generation Models As World Models

arXiv 25.2 2025 64.5 benchmark

TLDR

Proposes WorldModelBench to evaluate video generation models as world models, focusing on physics adherence and instruction-following with human labels.

Reasoning

Strengths include a novel benchmark addressing overlooked world modeling aspects like physics adherence, with large-scale human labels and an automated judger. Weaknesses: limited to video generation models and does not evaluate interactive or real-time decision-making.

Read-first score

Read-first score 64.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 55.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Topical relevance 42%

78.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=github

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 293.

Keyword Scores

world model

video world model

generative world model

world dynamics prediction

world simulator

model-based reinforcement learning world model

interactive world model

Deep Analysis

Innovations

WorldModelBench, a benchmark designed to evaluate world modeling capabilities of video generation models in application-driven domains, focusing on instruction-following and physics adherence.
Detection of subtle world modeling violations such as irregular changes in object size that breach mass conservation law, which prior benchmarks overlook.
Crowd-sourcing 67K human labels to evaluate 14 frontier models, enabling accurate measurement of world modeling violations.
Fine-tuning a 2B-parameter judger that achieves 8.6% higher average accuracy than GPT-4o in predicting world modeling violations.
Demonstration that training to align human annotations by maximizing rewards from the judger noticeably improves world modeling capability.

Methodology

WorldModelBench incorporates instruction-following and physics-adherence dimensions to evaluate video generation models. The authors crowd-sourced 67K human labels to assess 14 frontier models, then fine-tuned a 2B-parameter judger using these labels to automate evaluation, achieving higher accuracy than GPT-4o.

Key Results

The fine-tuned judger achieves 8.6% higher average accuracy in predicting world modeling violations than GPT-4o with 2B parameters. Additionally, training to align human annotations by maximizing rewards from the judger noticeably improves world modeling capability.

Resources

Paper Project