Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

arXiv 26.3 2026 63.3 method

TLDR

DILLO distills a language-action world model from a VLM to an LLM for proactive agent steering, achieving 14x speedup and improved success rates.

Reasoning

The paper presents a novel method that bypasses visual simulation by using latent state and actions to predict outcomes via text-only inference, demonstrating significant speedup and performance gains on simulated benchmarks. However, it relies on a privileged teacher during training and is only evaluated in simulation, limiting real-world applicability.

Read-first score

Read-first score 63.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 47.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,benchmark,experiment

Topical relevance 42%

67.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 216.

Keyword Scores

world model

world dynamics prediction

model-based reinforcement learning world model

world simulator

interactive world model

generative world model

video world model

Deep Analysis

Innovations

Challenging the assumption that visual processing is necessary for failure prevention; showing that a trained policy's latent state and planned actions encode sufficient information to anticipate action outcomes, making visual simulation redundant.
Introducing DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from 'simulate-then-act' to 'describe-then-act'.
Cross-modal distillation where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes, creating a text-only inference path that bypasses heavy visual generation.
Achieving a 14x speedup over baselines by eliminating visual simulation.

Methodology

DILLO is trained via cross-modal distillation: a privileged Vision Language Model (VLM) teacher annotates offline trajectories with semantic descriptions, and a latent-conditioned Large Language Model (LLM) student learns to predict these semantic outcomes from the policy's latent state and planned actions. This creates a text-only inference path that bypasses visual generation entirely. The method is evaluated on MetaWorld and LIBERO benchmarks, comparing against baselines that use visual simulation.

Key Results

DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 percentage points and 9.3 percentage points on average across tasks on MetaWorld and LIBERO.

Resources

Paper