How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment

arXiv 25.11 2025 58.5 benchmark, application

TLDR

A pilot study evaluating zero-shot surgical video generation, revealing a plausibility gap between visual quality and causal understanding.

Reasoning

The paper introduces a novel benchmark (SurgVeo) and evaluation framework (SPP) for surgical video generation, with expert surgeon assessment. Its strengths include a systematic evaluation and quantitative evidence of limitations, but weaknesses are the pilot nature, limited scope (two procedures), and lack of proposed model improvements.

Read-first score

Read-first score 58.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 40.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation,result

Topical relevance 42%

57.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 212.

Keyword Scores

world model

video world model

generative world model

world simulator

world dynamics prediction

interactive world model

model-based reinforcement learning world model

Deep Analysis

Innovations

SurgVeo: the first expert-curated benchmark for video generation model evaluation in surgery
Surgical Plausibility Pyramid (SPP): a novel four-tiered framework for assessing surgical video plausibility from basic appearance to complex surgical strategy
First quantitative evidence of the 'plausibility gap' between visually convincing mimicry and causal understanding in surgical AI

Methodology

The authors introduce SurgVeo, an expert-curated benchmark of surgical clips from laparoscopic and neurosurgical procedures, and the Surgical Plausibility Pyramid (SPP), a four-tiered evaluation framework (Visual Perceptual, Instrument Operation, Environment Feedback, Surgical Intent). They task the Veo-3 foundation model with zero-shot prediction on these clips, and a panel of four board-certified surgeons evaluates the generated videos according to the SPP tiers.

Key Results

Veo-3 achieves exceptional Visual Perceptual Plausibility but fails critically at higher SPP levels (Instrument Operation, Environment Feedback, and Surgical Intent Plausibility), revealing a distinct 'plausibility gap' between visual mimicry and causal understanding.

Limitations

Pilot study with limited scope (only two surgical domains: laparoscopic and neurosurgical)
Only one foundation model (Veo-3) evaluated, no comparison with other video generation models
Zero-shot setting may not reflect performance of fine-tuned or domain-adapted models
Small expert panel of four board-certified surgeons, potentially limiting generalizability of assessments
No explicit discussion of inter-rater reliability or statistical significance of the plausibility gap

Resources

Paper