⌁ Awesome World Model Hub Papers · Datasets · Projects
中文 GitHub
← Back to papers

Learning Latent Action World Models In The Wild

arXiv 26.1 2026 60.8 method

TLDR

Learns latent action world models from in-the-wild videos without action labels, using continuous constrained latent actions to capture complex dynamics.

Reasoning

The paper expands world model learning to diverse real-world videos, addressing challenges like environmental noise and lack of common embodiment. Strengths include novel latent action representation and transferability; weaknesses are limited spatial localization and absence of full embodiment generalization.

Read-first score

Read-first score 60.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 42.

Recency 8%
100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%
80

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,evaluation,experiment

Topical relevance 42%
60

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%
30

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 267.

Keyword Scores

world model
9
video world model
8
world dynamics prediction
7
generative world model
6
interactive world model
5
world simulator
4
model-based reinforcement learning world model
3

Deep Analysis

Innovations

  • Learning latent action world models on in-the-wild videos, expanding beyond simple robotics simulations, video games, or manipulation data
  • Using continuous but constrained latent actions instead of vector quantization to capture the complexity of actions from diverse videos
  • Demonstrating transfer of agent-induced environmental changes (e.g., humans entering a room) across videos
  • Training a controller that maps known actions to latent ones, enabling latent actions to serve as a universal interface for planning tasks

Methodology

The paper proposes a world model that learns latent actions from unlabeled in-the-wild videos by enforcing properties such as continuity and constraints on the latent action space. Architectural choices are discussed to handle video diversity, environmental noise, and lack of common embodiment. A controller is trained to map known actions to latent actions, allowing the world model to be used for planning tasks and compared against action-conditioned baselines.

Key Results

Continuous constrained latent actions outperform vector quantization in capturing the complexity of in-the-wild actions, and agent-induced changes can be transferred across videos. The controller-based planning achieves similar performance to action-conditioned baselines.

Limitations

  • Lack of a common embodiment across videos causes latent actions to become localized in space relative to the camera
  • Environmental noise and video diversity introduce challenges that are only partially addressed
  • The approach is mainly able to learn spatially localized latent actions, which may limit generalization to other types of actions or embodiments

Resources

Paper

Tags