Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents

arXiv 2026 58.5 method

TLDR

ADWM uses a latent diffusion world model to simulate environment responses for off-policy evaluation of LLM agents without online interaction.

Reasoning

The paper introduces a novel combination of autoregressive and diffusion models to address compounding errors in world modeling for LLM agent evaluation. Strengths include a principled approach to handling discrete text actions and policy-conditioned score functions. Weaknesses are the lack of detailed empirical comparisons and potential limitations in scalability, though the abstract claims strong empirical results.

Read-first score

Read-first score 58.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 48.

Recency 6%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Citation impact 18%

93.6

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. citation_normalized_percentile=0.93640389

Topical relevance 29%

68.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FoundationFrontierBridge

Rank sensitivity

Stability: volatile; rank range: 462.

Keyword Scores

world model

generative world model

world simulator

world dynamics prediction

interactive world model

model-based reinforcement learning world model

video world model

Deep Analysis

Innovations

Proposes ADWM, a latent diffusion world model for off-policy evaluation of LLM agents without real environment interaction.
Models each transition as an independent denoising process to avoid compounding errors inherent in autoregressive world models.
Introduces a policy-conditioned score function that allows the evaluation policy to directly guide diffusion generation at each step.

Methodology

ADWM learns a latent diffusion world model from pre-collected trajectories. It performs step-by-step rollouts where the world model and the LLM agent alternate in causal order, with each transition modeled as an independent denoising process. The evaluation policy directly guides the diffusion generation via a policy-conditioned score function, ensuring simulated trajectories reflect the agent's decision-making patterns.

Key Results

ADWM achieves accurate value estimates and evaluation reliability across diverse multi-turn agent tasks, demonstrating its promise as a practical framework for offline LLM agent evaluation.

Limitations

Relies on the availability and coverage of pre-collected trajectories, which may limit applicability in environments with sparse or biased offline data.

Resources

Paper