RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

arXiv 2026 39.1 method

TLDR

RoboGaze is a training-free multi-agent VLM framework for structured, interpretable evaluation of robot-generated manipulation videos, with a taxonomy and human-validated dataset.

Reasoning

The paper introduces a novel evaluation framework and dataset, showing strong gains over baselines and addressing false positives. Its main limitation is that it evaluates world models rather than proposing or improving them, and performance depends on underlying VLM backbones.

Read-first score

Read-first score 39.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 35.

Methodology quality 15%

100

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,baseline,benchmark,dataset,evaluation,metric

Recency 5%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Topical relevance 25%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 15%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Graph prestige 15%

5.7

PageRank-style prestige over the local citation graph built from available OpenAlex references. local_pagerank=0.0016

Citation impact 15%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 10%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 716.

Keyword Scores

world model

generative world model

video world model

interactive world model

world dynamics prediction

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Training-free multi-agent VLM framework (RoboGaze) for structured, interpretable evaluation of generated robot-manipulation videos.
Three-stage evaluation pipeline: task-scene grounding, dimension-specific specialist routing, and critic-based verification.
Novel 6-dimension, 30-type robotics-specific taxonomy producing temporally localized glitch reports.
Human-validated benchmark dataset of 382 clips spanning simulated and real-world multi-view manipulation.
Critic verifier designed to reduce false-positive 'cry-wolf' errors in standard VLM judges.

Methodology

RoboGaze is a training-free, multi-agent VLM framework that evaluates robot-manipulation videos from a task instruction and video. It uses a three-stage pipeline consisting of task-scene grounding, dimension-specific specialist routing, and critic-based verification to output temporally localized glitch reports under a 6-dimension, 30-type taxonomy. Evaluation uses a human-validated dataset of 382 simulated and real-world multi-view manipulation clips and compares eight open-source and proprietary VLM backbones against zero-shot baselines.

Key Results

RoboGaze improves description-F1 by up to +43 points and temporal alignment (F1 x IoU) by up to +37 points over zero-shot baselines, closing about 85% of the gap to the human ceiling. Its critic verifier raises clean-clip accuracy from under 25% to over 80%.

Resources

Paper