Generating Multimodal Driving Scenes via Next-Scene Prediction

CVPR 25 2025 62.4 method, application

TLDR

Multimodal driving scene generation via autoregressive next-scene prediction with map and ego-action modalities.

Reasoning

The paper introduces a novel multimodal generation framework for driving scenes, incorporating map modality and two-stage autoregressive prediction. Strengths include handling multiple modalities and temporal coherence, but weaknesses are lack of explicit real-world validation and limited scope to autonomous driving.

Read-first score

Read-first score 62.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 41.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=github

Topical relevance 42%

58.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation

Field roles

FrontierReproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 213.

Keyword Scores

generative world model

world dynamics prediction

world simulator

world model

video world model

interactive world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Multimodal generation framework incorporating four major data modalities, including a novel addition of map modality
Two-stage autoregressive approach: Temporal AutoRegressive (TAR) for inter-frame dynamics and Ordered AutoRegressive (OAR) for intra-scene modality alignment
Action-aware Map Alignment (AMA) module to maintain coherence between map and ego-action modalities

Methodology

The framework tokenizes multiple modalities and autoregressively predicts each scene in a sequence. It uses a two-stage approach: Temporal AutoRegressive (TAR) captures inter-frame dynamics per modality, and Ordered AutoRegressive (OAR) aligns modalities within a scene by sequentially predicting tokens in a fixed order. Additionally, the Action-aware Map Alignment (AMA) module applies a transformation based on ego-action to maintain coherence between map and ego-action modalities.

Key Results

The framework effectively generates complex, realistic driving scenes over extended sequences, ensuring multimodal consistency and offering fine-grained control over scene elements.

Resources

Paper Project Code