EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models

arXiv 25.5 2025 71.3 benchmark, system

TLDR

Proposes EWMBench, a benchmark evaluating embodied world models on scene consistency, motion correctness, and semantic alignment using a curated dataset and multi-dimensional toolkit.

Reasoning

The paper addresses a critical gap in evaluating embodied world models by introducing a dedicated benchmark with a curated dataset and multi-dimensional evaluation tools. Strengths include practical utility and public availability, but weaknesses include lack of novel model contributions and limited detail on empirical results in the abstract.

Read-first score

Read-first score 71.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 40.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=dataset,github

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,dataset,evaluation,metric

Topical relevance 42%

57.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

FrontierMethodology anchorReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 226.

Keyword Scores

world model

video world model

generative world model

world dynamics prediction

interactive world model

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Proposal of EWMBench, a dedicated benchmark for evaluating embodied world models (EWMs) on three key aspects: visual scene consistency, motion correctness, and semantic alignment.
A meticulously curated dataset encompassing diverse scenes and motion patterns for embodied AI evaluation.
A comprehensive multi-dimensional evaluation toolkit designed to assess and compare candidate EWMs.

Methodology

The methodology involves constructing a curated dataset of diverse scenes and motion patterns, and developing a multi-dimensional evaluation toolkit that measures visual scene consistency, motion correctness, and semantic alignment. The benchmark is used to assess and compare existing video generation models as embodied world models.

Key Results

The benchmark identifies limitations of existing video generation models in meeting the unique requirements of embodied tasks, particularly in physical grounding and action-consistency, and provides insights to guide future advancements.

Resources

Paper Code