Keeping JEPA World Models Plannable When Little of the Frame Moves

arXiv 2026 39.2 method, benchmark

TLDR

JEPA world models fail to plan when few objects move; an inverse-dynamics auxiliary loss restores action sensitivity and planning on a new language-pushing benchmark.

Reasoning

Strengths: clear problem identification via probing, a simple auxiliary loss fixes planning, and an action-sensitivity metric offers a necessary condition. Weaknesses: evaluations are in simulation only, and success rates remain moderate (e.g., 0.35 overall), limiting real-world claims.

Read-first score

Read-first score 39.2, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 48.

Recency 5%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 15%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=benchmark,evaluation

Topical relevance 25%

68.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 15%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code

Graph prestige 15%

5.7

PageRank-style prestige over the local citation graph built from available OpenAlex references. local_pagerank=0.0016

Citation impact 15%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 10%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 454.

Keyword Scores

world model

world dynamics prediction

model-based reinforcement learning world model

interactive world model

world simulator

video world model

generative world model

Deep Analysis

Innovations

Introduces SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes.
Diagnoses failure of a LeWM/JEPA world model on SLIM via probes showing encoder latents are nearly action-insensitive and cannot decode pusher/object positions.
Proposes an inverse-dynamics auxiliary loss applied to encoder latents and predicted latents through a shared head discarded at test time.
Provides a cheap action-sensitivity probe computable without environment access as an empirical necessary condition for planning.
Shows a small language-goal head on the repaired latent can plan from sentences without retraining the world model.

Methodology

The paper evaluates a LeWM world model that solves PushT on SLIM, a pushing benchmark with paired visual and language goals. It uses decoding and rollout probes to locate failures in latent action sensitivity, then trains with an inverse-dynamics auxiliary loss on encoder and predicted latents via a shared head discarded at test time. Evaluation includes planning success, PushT horizon generalization, controls/response sweeps, action-sensitivity thresholds, and language-goal head planning from sentences.

Key Results

The vanilla LeWM succeeds on under 1% of SLIM trials; the inverse-dynamics loss restores probes and raises success from 0.003 to 0.35 (0.16 hard pushing) and improves PushT at twice the trained horizon. A language-goal head reaches 0.84 on navigation (visual-goal oracle 1.00), while staged sentence goals raise medium/hard pushing success from 0.04 to 0.25, on par with the goal-frame oracle.

Limitations

A single goal sentence rarely completes a push; staged sentence sequences are needed for pushing tasks.
Repaired success on SLIM remains 0.35 overall and 0.16 on the hard tier, far from the scripted controller's perfect tier performance.
The action-sensitivity probe is an empirical necessary condition, not a sufficiency guarantee.
Language-goal navigation remains below the visual-goal oracle (0.84 vs 1.00).

Resources

Paper