Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

arXiv 24.7 2024 28 method, application

TLDR

TOKEN tokenizes world into object-level knowledge using MM-LLM to improve autonomous driving planning in long-tail events, reducing L2 error by 27% and collision rates by 39%.

Reasoning

Strengths: novel integration of end-to-end models with LLM reasoning, explicit alignment training, and strong empirical results on long-tail scenarios. Weaknesses: limited scope to long-tail events, no comparison to world model approaches, and abstract lacks details on generalization or real-world dataset specifics.

Read-first score

Read-first score 28, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 3.

Recency 8%

75.1

Uses a gentle age decay so recent papers surface without erasing older foundations. 2024

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

4.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Field roles

Candidate

Rank sensitivity

Stability: volatile; rank range: 48.

Keyword Scores

world model

world dynamics prediction

world simulator

generative world model

interactive world model

video world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Proposes TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge to enhance autonomous vehicle planning in long-tail scenarios.
Leverages a traditional end-to-end driving model to produce condensed and semantically enriched representations optimized for LLM planning through deliberate representation and reasoning alignment training stages.
Addresses data scarcity and inefficient tokenization by using object-level tokenization.

Methodology

TOKEN uses a traditional end-to-end driving model to generate condensed, semantically enriched scene representations. These representations are then aligned with an LLM through two training stages: representation alignment and reasoning alignment, enabling the LLM to perform planning in long-tail scenarios.

Key Results

TOKEN achieves a 27% reduction in trajectory L2 error and a 39% decrease in collision rates in long-tail scenarios, outperforming existing frameworks in grounding, reasoning, and planning capabilities.

Resources

Paper