WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation

ECCV 24 2024 67 method, system, application

TLDR

WoVoGen uses 4D world volume to generate consistent multi-camera driving videos from vehicle control inputs.

Reasoning

The paper introduces a novel two-phase diffusion framework leveraging explicit 4D world volume for intra-world and inter-sensor consistency. Strengths include addressing key consistency challenges in multi-camera generation, while weaknesses are the lack of explicit real-world evaluation or benchmark results in the abstract.

Read-first score

Read-first score 67, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 47.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=dataset,github

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

67.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 163.

Keyword Scores

generative world model

video world model

world model

world dynamics prediction

world simulator

interactive world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Combining an explicit 4D world volume as a foundational element for multi-camera driving scene video generation
Two-phase generation pipeline: first envisioning future 4D temporal world volume from vehicle control sequences, then generating multi-camera videos conditioned on that volume and sensor interconnectivity
Controllable generation via vehicle control inputs and support for scene editing tasks

Methodology

WoVoGen operates in two phases: (i) it envisions a future 4D temporal world volume based on vehicle control sequences, and (ii) it generates multi-camera videos informed by this envisioned 4D world volume and sensor interconnectivity. The model leverages diffusion-based generation and incorporates an explicit world volume representation to ensure intra-world consistency and inter-sensor coherence.

Key Results

The system generates high-quality street-view videos in response to vehicle control inputs and facilitates scene editing tasks, demonstrating the effectiveness of the 4D world volume approach.

Resources

Paper Code