Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

arXiv 26.3 2026 41.3 method

TLDR

DILLO distills a language-action world model from a VLM to an LLM for proactive agent steering, achieving 14x speedup and improved success rates.

Reasoning

The paper presents a novel method that bypasses visual simulation by using latent state and actions to predict outcomes via text-only inference, demonstrating significant speedup and performance gains on simulated benchmarks. However, it relies on a privileged teacher during training and is only evaluated in simulation, limiting real-world applicability.

Read-first score

Read-first score 41.3, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 47.

Recency 5%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 15%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,benchmark,experiment

Topical relevance 25%

67.1

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 15%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Citation impact 15%

16.2

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. citation_normalized_percentile=0.16205449

Graph prestige 15%

5.7

PageRank-style prestige over the local citation graph built from available OpenAlex references. local_pagerank=0.0016

Citation velocity 10%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 362.

Keyword Scores

world model

world dynamics prediction

model-based reinforcement learning world model

world simulator

interactive world model

generative world model

video world model

Deep Analysis

Innovations

Challenging the assumption that visual processing is necessary for failure prevention; showing that a trained policy's latent state and planned actions encode sufficient information to anticipate action outcomes, making visual simulation redundant.
Introducing DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from 'simulate-then-act' to 'describe-then-act'.
Cross-modal distillation where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes, creating a text-only inference path that bypasses heavy visual generation.
Achieving a 14x speedup over baselines by eliminating visual simulation.

Methodology

DILLO is trained via cross-modal distillation: a privileged Vision Language Model (VLM) teacher annotates offline trajectories with semantic descriptions, and a latent-conditioned Large Language Model (LLM) student learns to predict these semantic outcomes from the policy's latent state and planned actions. This creates a text-only inference path that bypasses visual generation entirely. The method is evaluated on MetaWorld and LIBERO benchmarks, comparing against baselines that use visual simulation.

Key Results

DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 percentage points and 9.3 percentage points on average across tasks on MetaWorld and LIBERO.

Resources

Paper