Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation

arXiv 24.6 2024 42.1 method, application

TLDR

Delphi generates controllable long videos (up to 40 frames) to improve planning in end-to-end autonomous driving.

Reasoning

The paper presents a novel diffusion-based method for long video generation with spatial and temporal consistency, and a failure-case sampling policy to boost planning performance. Strengths include addressing data scarcity for planning tasks and achieving longer consistent videos than prior work. Weaknesses are the narrow focus on autonomous driving and lack of explicit connection to world model concepts.

Read-first score

Read-first score 42.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 1.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=dataset,github

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset,experiment

Topical relevance 42%

1.4

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 349.

Keyword Scores

video world model

world model

world simulator

generative world model

interactive world model

world dynamics prediction

model-based reinforcement learning world model

Deep Analysis

Innovations

Shared noise modeling mechanism across multi-views to increase spatial consistency in long video generation
Feature-aligned module for precise controllability and temporal consistency
Generation of up to 40 frames of video without loss of consistency, approximately 5 times longer than state-of-the-art methods
Sampling policy using pre-trained visual language models to generate data similar to failure cases, improving sample efficiency
First framework to boost planning performance of end-to-end autonomous driving models by 25% using only 4% of training dataset size

Methodology

Delphi is a diffusion-based long video generation method that employs a shared noise modeling mechanism across multiple camera views to enhance spatial consistency and a feature-aligned module to achieve precise controllability and temporal consistency. It generates up to 40 frames of video and uses a failure-case-driven sampling policy, leveraging pre-trained visual language models, to select and generate data similar to failure cases for improved sample efficiency.

Key Results

Delphi generates higher quality long videos surpassing previous state-of-the-art methods, and for the first time boosts the planning performance of an end-to-end autonomous driving model by 25% while generating only 4% of the training dataset size.

Resources

Paper Code