DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model

ECCV 24 2024 40.7 method

TLDR

DrivingDiffusion generates realistic multi-view driving videos from 3D layouts using a cascaded diffusion model ensuring cross-view and cross-frame consistency.

Reasoning

The paper addresses a practical need for synthetic multi-view driving data with a novel cascaded diffusion framework. Strengths include explicit handling of cross-view and cross-frame consistency. Weaknesses: no real-world evaluation or comparison to existing methods is mentioned in the abstract, and the approach is limited to layout-guided generation without dynamics or interaction.

Read-first score

Read-first score 40.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 7.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,github

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=none

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 139.

Keyword Scores

video world model

generative world model

world model

world simulator

interactive world model

world dynamics prediction

model-based reinforcement learning world model

Deep Analysis

Innovations

Cross-view consistency ensured by information exchange between adjacent cameras in the multi-view model
Cross-frame consistency achieved by querying information from the first frame's multi-view images in the temporal model
Local prompt introduced to improve the quality of generated instances
Temporal sliding window algorithm in post-processing to enhance cross-view consistency of subsequent frames and extend video length

Methodology

DrivingDiffusion cascades three stages: multi-view single-frame image generation with information exchange between adjacent cameras for cross-view consistency, single-view video generation shared by multiple cameras that queries the first frame's multi-view images for cross-frame consistency, and post-processing using a temporal sliding window to enhance consistency and handle long video generation. Local prompts are used to improve instance quality.

Key Results

The model can generate large-scale realistic multi-camera driving videos in complex urban scenes without extra cost, fueling downstream driving tasks.

Resources

Paper Code