What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
TLDR
Temporal video pretraining, not pixel reconstruction, drives action-relevant structure in video world model latents, with self-supervised encoders achieving best trade-off.
Reasoning
Strengths: Clear experimental design with unified probe evaluation across diverse encoder families, identifying temporal prediction as key driver over reconstruction. Weaknesses: Limited to inverse-dynamics probing; static environments can mask temporal importance, potentially limiting generalizability.
Read-first score
Read-first score 58.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 39.
Field roles
Rank sensitivity
Stability: volatile; rank range: 314.
Keyword Scores
Deep Analysis
Innovations
- Identifies temporal predictive structure (prediction) as the primary ingredient for action-relevant video representations, not pixel reconstruction fidelity.
- Proposes a unified probe-based evaluation framework across diverse encoder families (image-only self-supervision, video pretraining, reconstruction autoencoders, diffusion models, shortcut-forcing dynamics models) to isolate the source of action-relevant structure.
- Demonstrates that video-pretrained self-supervised encoders achieve the best Pareto trade-off between visual fidelity and action prediction, while reconstruction-focused models can have near-zero action recoverability.
- Shows that inverse-dynamics supervision improves robustness to visual corruption, indicating action-aware objectives regularize latent geometry beyond clean-setting performance.
Methodology
The study uses a common inverse-dynamics probing objective to evaluate action-relevant structure in latent spaces of various video world models. It compares encoders from image-only self-supervision, video pretraining (with and without latent prediction), reconstruction-based autoencoders, diffusion models, and shortcut-forcing dynamics models across robotic benchmarks including CALVIN. The evaluation measures action recoverability and visual fidelity to identify which pretraining signals induce action-relevant latents.
Key Results
Models with strong pixel decoding quality can exhibit near-zero action recoverability, while video-pretrained self-supervised encoders consistently achieve the best Pareto trade-off between visual fidelity and action prediction. Temporal context from natural videos provides most gains; latent prediction offers a smaller additional benefit. Inverse-dynamics supervision substantially improves robustness to visual corruption.
Limitations
- Static-environment tasks (e.g., CALVIN) can partially mask the importance of temporal structure because strong image priors suffice, limiting the generality of the conclusion in such settings.
- The evaluation is limited to a single inverse-dynamics probing objective and may not capture all aspects of action-relevance or downstream task performance.
- The study focuses on robotic benchmarks; transferability to other domains (e.g., autonomous driving, video games) is not explicitly tested.