⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion

ICLR 24 2024 62.8 method

TLDR

Copilot4D uses VQVAE tokenization and discrete diffusion to learn unsupervised world models for autonomous driving, achieving significant improvements in point cloud prediction.

Reasoning

The paper presents a novel combination of VQVAE and discrete diffusion for world modeling, with strong empirical results on multiple real-world datasets. However, it focuses on point cloud observations and short-term prediction, and lacks explicit interaction or reinforcement learning components.

Read-first score

Read-first score 62.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 42.

Methodology quality 25%
80

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset,metric,result

Recency 8%
75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%
60

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%
46

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,dataset

Field roles

Methodology anchor

Rank sensitivity

Stability: volatile; rank range: 116.

Keyword Scores

world model
10
generative world model
9
world dynamics prediction
8
world simulator
6
model-based reinforcement learning world model
4
video world model
3
interactive world model
2

Deep Analysis

Innovations

  • Tokenizing sensor observations with VQVAE for world modeling in autonomous driving
  • Recasting Masked Generative Image Transformer as discrete diffusion with enhancements for efficient parallel decoding
  • Achieving significant reduction in Chamfer distance (over 65% for 1s, over 50% for 3s) compared to prior SOTA on multiple datasets
  • Demonstrating that discrete diffusion on tokenized agent experience can unlock GPT-like unsupervised learning for robotics

Methodology

Copilot4D first tokenizes sensor observations (point clouds) using a VQVAE, then models future prediction as a discrete diffusion process. It recasts the Masked Generative Image Transformer as discrete diffusion and introduces simple enhancements to improve efficiency. The model is trained and evaluated on NuScenes, KITTI Odometry, and Argoverse2 datasets, using Chamfer distance as the metric for 1s and 3s prediction horizons.

Key Results

Copilot4D reduces prior state-of-the-art Chamfer distance by more than 65% for 1s prediction and more than 50% for 3s prediction across NuScenes, KITTI Odometry, and Argoverse2 datasets.

Limitations

  • Only evaluated on point cloud observations, not other sensor modalities such as cameras or radar.
  • Prediction horizons are limited to 1s and 3s; long-term prediction performance is not assessed.

Resources

Paper

Tags