SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

arXiv 2026 53 benchmark

TLDR

SIMMER benchmarks latent failures in LLM planning using a symbolic world model, revealing high failure rates and showing counterfactual simulation reduces them.

Reasoning

The paper addresses an overlooked failure type (latent failures) with a well-defined symbolic world model grounded in real-world cooking scripts, providing clear quantitative results across six LLMs. However, the benchmark is limited to a single domain (kitchen) and uses a symbolic rather than generative or video-based world model, which may limit generalizability.

Read-first score

Read-first score 53, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 31.

Recency 6%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Citation impact 18%

93.1

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. citation_normalized_percentile=0.93108829

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,experiment

Topical relevance 29%

44.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FoundationFrontierBridgeMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 383.

Keyword Scores

world model

world dynamics prediction

world simulator

interactive world model

model-based reinforcement learning world model

generative world model

video world model

Deep Analysis

Innovations

Introduces SIMMER, the first benchmark specifically designed to evaluate latent failures in LLM planning, which are failures that do not immediately halt execution but silently compromise goal achievement.
Defines a human-curated symbolic world model grounded in the kitchen domain with 77 actions, 262 unique objects, and approximately 46,800 semantically realistic interactions derived from real-world cooking scripts.
Employs a state machine executor that validates plans against the world model and detects three types of failures: immediate precondition violations, latent hazards, and irreversible failures.
Demonstrates that explicit state reasoning via counterfactual foresight simulation can reduce latent failures by up to 72% and irreversible cases by up to 75%, offering a promising direction for more robust LLM planners.

Methodology

SIMMER constructs a human-curated symbolic world model for the kitchen domain, comprising 77 actions, 262 unique objects, and approximately 46,800 possible interactions derived from real-world cooking scripts. A state machine executor validates LLM-generated plans against this world model, detecting immediate precondition violations, latent hazards, and irreversible failures. The benchmark evaluates six LLMs on their ability to produce error-free plans and measures the prevalence of latent failures.

Key Results

Even frontier models achieve at most 17% error-free plans, and up to 56% of plans contain latent failures, the majority of which lead to irreversible consequences. Counterfactual foresight simulation reduces latent failures by up to 72% and irreversible cases by up to 75%.

Limitations

The benchmark is limited to the kitchen domain, which may not generalize to other household or real-world environments.
The world model is symbolic and human-curated, potentially missing nuances of real-world physics and unmodeled interactions.

Resources

Paper