Doe-1: Closed-Loop Autonomous Driving with Large World Model

arXiv 24.12 2024 69.9 method, application

TLDR

Doe-1 is a closed-loop autonomous driving framework using a large world model that unifies perception, prediction, and planning via next-token generation.

Reasoning

Strengths include a novel unified formulation as next-token generation and multi-modal tokens for closed-loop driving. Weaknesses are limited evaluation to the nuScenes dataset and lack of real-world deployment or comparison to state-of-the-art in the abstract.

Read-first score

Read-first score 69.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 46.

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=True; dataset=False; markers=code,dataset,github

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Topical relevance 42%

65.7

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset,experiment

Field roles

Reproducibility anchor

Rank sensitivity

Stability: volatile; rank range: 127.

Keyword Scores

world model

world dynamics prediction

video world model

generative world model

interactive world model

world simulator

model-based reinforcement learning world model

Deep Analysis

Innovations

Closed-loop autonomous driving framework using a large world model (Doe-1) for unified perception, prediction, and planning.
Formulating autonomous driving as a next-token generation problem with multi-modal tokens.
Using free-form text (scene descriptions) for perception and generating future predictions directly in RGB space with image tokens.
Position-aware tokenizer to encode action into discrete tokens for planning.
End-to-end multi-modal transformer that autoregressively generates perception, prediction, and planning tokens.

Methodology

Doe-1 employs a multi-modal transformer trained on the nuScenes dataset. It tokenizes perception as free-form text, prediction as RGB image tokens, and planning via a position-aware tokenizer, then autoregressively generates these tokens in a unified end-to-end manner.

Key Results

Experiments on nuScenes demonstrate effectiveness in visual question-answering, action-conditioned video generation, and motion planning.

Resources

Paper Code