ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

arXiv 26.3 2026 48.8 method

TLDR

ThinkJEPA integrates a VLM with a JEPA-style latent world model for long-horizon video prediction, outperforming baselines on hand-manipulation trajectory prediction.

Reasoning

The paper presents a novel dual-temporal framework combining dense JEPA and sparse VLM branches, with a hierarchical pyramid module for guidance. Strengths include addressing limitations of both dense prediction and VLMs, but the evaluation is limited to a single domain (hand-manipulation) and lacks broader real-world benchmarks or interactive settings.

Read-first score

Read-first score 48.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 37.

Recency 6%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,dataset,experiment

Topical relevance 29%

52.9

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Citation impact 18%

37.2

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. citation_normalized_percentile=0.37244898

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 330.

Keyword Scores

world model

video world model

world dynamics prediction

generative world model

interactive world model

model-based reinforcement learning world model

world simulator

Deep Analysis

Innovations

VLM-guided JEPA-style latent world modeling framework combining dense-frame dynamics with long-horizon semantic guidance via a dual-temporal pathway
Dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with larger temporal stride for knowledge-rich guidance
Hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction

Methodology

The proposed ThinkJEPA framework integrates a dense JEPA branch for fine-grained motion and interaction cues with a uniformly sampled VLM thinker branch that provides long-horizon semantic guidance. A hierarchical pyramid representation extraction module transfers progressive reasoning signals from the VLM into guidance features for latent prediction. The model is evaluated on hand-manipulation trajectory prediction, compared against a VLM-only baseline and a JEPA-predictor baseline.

Key Results

ThinkJEPA outperforms both a strong VLM-only baseline and a JEPA-predictor baseline on hand-manipulation trajectory prediction, and yields more robust long-horizon rollout behavior.

Limitations

Dense prediction from a short observation window limits temporal context and biases predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics
VLMs are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets

Resources

Paper