Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model

arXiv 2026 36.8 method

TLDR

OMR is an inference-time method that injects gradients from a frozen V-JEPA world model into video generator sampling to improve physical consistency.

Reasoning

The paper proposes a novel, efficient inference-time refinement that avoids extra denoising loops or generator retraining, with clear gains on a video physical-commonsense benchmark. The main limitation is that the adapter still requires a small training stage and evaluation is only on a fixed 400-prompt subset.

Read-first score

Read-first score 36.8, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 38.

Recency 5%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 15%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation,metric

Topical relevance 25%

54.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 15%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,github

Graph prestige 15%

5.7

PageRank-style prestige over the local citation graph built from available OpenAlex references. local_pagerank=0.0016

Citation impact 15%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 10%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 454.

Keyword Scores

world model

video world model

world dynamics prediction

generative world model

world simulator

interactive world model

model-based reinforcement learning world model

Deep Analysis

Innovations

Off-Manifold Refinement (OMR): an inference-time method that injects world-model feedback directly into a single sampling trajectory, avoiding candidate selection, repeated decoding/re-encoding, generator-internal perturbation/re-denoising loops, and post-training optimization.
Augments the generator velocity during scheduled middle ODE steps with the gradient of an adapter-space V-JEPA 2.1 surprise energy.
Uses a small trained latent-to-embedding adapter to make the gradient tractable at inference while keeping both the video generator and the world model frozen.
Shifts the latent away from the uncorrected sampling trajectory toward regions ranked as more physically plausible by the frozen predictor, after which the generator continues rendering from the corrected state.

Methodology

OMR is an inference-time method that adds world-model feedback to a single sampling trajectory by augmenting generator velocity with the gradient of an adapter-space V-JEPA 2.1 surprise energy during scheduled middle ODE steps. A small trained latent-to-embedding adapter maps generator latents to the frozen world model embeddings, keeping gradients tractable while both the video generator and world model remain frozen. Evaluation uses a fixed 400-prompt VideoPhy-2 detailed subset and a fixed 50-prompt efficiency subset with the base Wan2.2-T2V-A14B sampler.

Key Results

On the fixed 400-prompt VideoPhy-2 detailed subset, OMR improves the joint Semantic-Adherence-and-Physical-Commonsense metric from 47.0% to 52.0%, an absolute gain of +5.0pp and a relative gain of +10.6% over the base Wan2.2-T2V-A14B sampler. On a fixed 50-prompt efficiency subset, OMR requires 1.71× the base runtime rather than the multiplicative cost of reward/search alternatives.

Limitations

Adds 1.71× base runtime overhead compared with the base sampler.
Requires a small trained latent-to-embedding adapter, so it is not fully training-free.
Reported evaluation is limited to fixed subsets of VideoPhy-2 and the Wan2.2-T2V-A14B base sampler.

Resources

Paper