⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving

arXiv 24.12 2024 37.9 method, application

TLDR

HoloDrive jointly generates camera images and LiDAR point clouds for autonomous driving using 2D-3D transforms and temporal prediction.

Reasoning

The paper introduces a novel framework for multi-modal street scene generation, addressing a gap in joint 2D-3D generation. Strengths include the use of BEV-to-Camera and Camera-to-BEV transforms and depth disambiguation. Weaknesses are that it focuses only on generation metrics and does not explicitly claim to be a world model, limiting relevance to the specified keywords.

Read-first score

Read-first score 37.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 7.

Methodology quality 25%
80

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,experiment,metric,result

Recency 8%
75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Reproducibility 25%
30

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%
10

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Methodology anchor

Rank sensitivity

Stability: volatile; rank range: 178.

Keyword Scores

generative world model
3
world model
2
video world model
1
world dynamics prediction
1
world simulator
0
interactive world model
0
model-based reinforcement learning world model
0

Deep Analysis

Innovations

  • Joint 2D-3D multi-modal generation of camera images and LiDAR point clouds for autonomous driving
  • BEV-to-Camera and Camera-to-BEV transform modules between heterogeneous generative models
  • Depth prediction branch in the 2D generative model to disambiguate un-projecting from image space to BEV space
  • Extension to future prediction via temporal structure and progressive training

Methodology

HoloDrive employs BEV-to-Camera and Camera-to-BEV transform modules to bridge heterogeneous generative models for joint generation of camera images and LiDAR point clouds. A depth prediction branch is introduced in the 2D generative model to resolve ambiguity when un-projecting from image space to BEV space. The framework is extended to future prediction by adding temporal structure and using carefully designed progressive training.

Key Results

Experiments on single frame generation and world model benchmarks demonstrate that HoloDrive achieves significant performance gains over state-of-the-art methods in terms of generation metrics.

Resources

Paper

Tags