Vega: Learning to Drive with Natural Language Instructions
TLDR
Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process.
Reasoning
Fallback reasoning generated from available title and abstract metadata: Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized...
Read-first score
Read-first score 53.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals.
Field roles
Rank sensitivity
Stability: volatile; rank range: 600.
Deep Analysis
Innovations
- Construction of a large-scale driving dataset (InstructScene) with around 100,000 scenes annotated with diverse driving instructions and corresponding trajectories
- Proposal of a unified Vision-Language-World-Action model (Vega) that combines autoregressive paradigm for vision and language with diffusion paradigm for world modeling and action
- Use of joint attention for modality interactions and individual projection layers for different modalities to enhance capabilities
Methodology
The authors first constructed the InstructScene dataset containing approximately 100,000 driving scenes annotated with diverse instructions and trajectories. They then proposed the Vega model, which employs an autoregressive paradigm to process visual inputs and language instructions, and a diffusion paradigm to generate future predictions (world modeling) and trajectories (action). Joint attention enables interactions between modalities, and individual projection layers are used for different modalities to improve capabilities.
Key Results
Extensive experiments demonstrate that Vega achieves superior planning performance and exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.