DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving

arXiv 24.8 2024 32.5 method, application

TLDR

DriveGenVLM generates driving videos using diffusion models and uses VLMs for scene understanding to enhance autonomous driving.

Reasoning

The paper integrates video generation with VLMs, using the Waymo dataset and FVD evaluation, which are strengths. However, it lacks novelty and does not explicitly address world models or interactive aspects, limiting its scope.

Read-first score

Read-first score 32.5, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 3.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset,result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Topical relevance 42%

4.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Candidate

Rank sensitivity

Stability: volatile; rank range: 69.

Keyword Scores

world dynamics prediction

video world model

world model

world simulator

generative world model

interactive world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Integration of video generation with vision language models (VLMs) for autonomous driving
Use of denoising diffusion probabilistic models (DDPM) to generate real-world driving video sequences
Application of pre-trained EILEV model to provide narrations for generated driving videos

Methodology

The DriveGenVLM framework employs a denoising diffusion probabilistic model (DDPM) for video generation, trained on the Waymo open dataset. Generated videos are evaluated using the Fréchet Video Distance (FVD) score. A pre-trained model, Efficient In-context Learning on Egocentric Videos (EILEV), is then used to produce corresponding narrations for the generated videos.

Key Results

The paper reports that the diffusion model is evaluated using the FVD score to ensure quality and realism, but no specific numerical results are provided in the abstract. The narrations from EILEV are claimed to enhance traffic scene understanding, navigation, and planning.

Resources

Paper