Diffusion Transformer World-Action Model for AV Scene Prediction

arXiv 2026 70 method, benchmark, application

TLDR

A compact latent Diffusion Transformer world model for autonomous vehicle scene prediction, conditioned on ego-actions, outperforms regression on perceptual metrics.

Reasoning

The paper clearly identifies the failure of standard distortion metrics for action-conditioned world models and proposes a latent Diffusion Transformer with four key ingredients, showing significant improvements in perceptual metrics like FID/KID. However, the evaluation is limited to a single dataset (nuScenes) and only front-camera scenes, and the abstract cuts off before presenting full results or limitations.

Read-first score

Read-first score 70, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 58.

Methodology quality 18%

100

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis,benchmark,evaluation,experiment,metric

Recency 6%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Citation impact 18%

87.4

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. citation_normalized_percentile=0.87422897

Topical relevance 29%

82.9

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FoundationFrontierBridgeMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 374.

Keyword Scores

world model

generative world model

world dynamics prediction

interactive world model

video world model

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Latent world model that predicts future scene latents from present front-camera latent and ego-actions, decoded to 256×256 frames up to 8 seconds ahead
Benchmarking six frozen encoders across four representation families, showing V-JEPA2 with temporal context reduces steering RMSE by 40% over the best single-frame encoder
Identification of four essential ingredients for latent Diffusion Transformer: spatial tokens, x0 objective, residual anchoring, and sampling matched to target uncertainty
Exposure of the perception-distortion frontier: distortion metrics (cosine similarity, SSIM) favor blurry regression mean, while Inception-based FID/KID reveal diffusion is far closer to real frame distribution (KID 0.078 vs 0.375, 4.8× better)
Action-controllable prediction with steering driving scene displacement (Spearman ρ=0.81 vs -0.18 for regression)
Compact 1.7M-parameter 'jump' model that recovers full ground-truth motion magnitude (1.02× GT), overcoming limited single-pass motion

Methodology

The paper proposes a compact latent world model that takes the present front-camera latent and a sequence of ego-actions to predict future scene latents, which are decoded by a frozen decoder to 256×256 frames up to 8 seconds ahead. It first benchmarks six frozen encoders (spanning four representation families) on steering RMSE, then trains a latent Diffusion Transformer (DiT) with four identified ingredients. Evaluation is performed on 150 held-out nuScenes scenes using distortion metrics (cosine similarity, SSIM) and perception metrics (FID, KID), along with a controllability analysis and a motion magnitude recovery experiment.

Key Results

The diffusion model achieves KID 0.078 versus 0.375 for regression (4.8× better), and steering controllability yields Spearman ρ=0.81 compared to -0.18 for regression. The compact 1.7M-parameter jump model recovers 1.02× ground-truth motion magnitude, whereas single-pass models capture less than half.

Limitations

Single-pass models capture less than half of ground-truth motion magnitude due to a shared-present anchor
Standard distortion metrics (cosine similarity, SSIM) actively mislead by rewarding blurry regression means over realistic predictions
The model's limited single-pass motion is traced to the shared-present anchor, requiring a dedicated jump model to recover full motion

Resources

Paper