RLVR-World: Training World Models with Reinforcement Learning

arXiv 25.5 2025 70.4 method

TLDR

RLVR-World uses reinforcement learning with verifiable rewards to optimize world models for transition prediction metrics, improving language and video models across diverse domains.

Reasoning

The paper presents a novel framework (RLVR) that directly optimizes world models for task-specific metrics like accuracy and perceptual quality, moving beyond standard MLE. Strengths include clear methodology and demonstrated gains in multiple domains (text games, web navigation, robot manipulation). Weaknesses include reliance on tokenized sequences and verifiable rewards, which may limit applicability to continuous or unverifiable settings.

Read-first score

Read-first score 70.4, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 60.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Topical relevance 42%

85.7

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=dataset,metric

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=code,dataset,github

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 179.

Keyword Scores

world model

video world model

world dynamics prediction

model-based reinforcement learning world model

generative world model

interactive world model

world simulator

Deep Analysis

Innovations

Using reinforcement learning with verifiable rewards (RLVR) to directly optimize world models for task-specific metrics like accuracy or perceptual quality, instead of standard MLE.
Formulating world modeling as autoregressive prediction of tokenized sequences and evaluating metrics of decoded predictions as verifiable rewards.
Unified framework applicable to both language- and video-based world models across diverse domains.

Methodology

RLVR-World uses reinforcement learning with verifiable rewards to train world models. The world model is formulated as an autoregressive predictor of tokenized sequences, and rewards are computed based on metrics (e.g., accuracy, perceptual quality) of the decoded predictions. The framework is evaluated on language- and video-based world models across text games, web navigation, and robot manipulation, comparing against standard MLE training.

Key Results

RLVR-World achieves substantial performance gains on both language- and video-based world models across domains including text games, web navigation, and robot manipulation.

Resources

Paper Project