World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation

arXiv 25.9 2025 69.7 method, system, application

TLDR

Proposes diffusion-based world models as high-fidelity simulators to refine robotic manipulation policies via reinforcement learning in imagined environments.

Reasoning

The paper presents a novel framework combining diffusion world models with policy refinement, supported by extensive simulation and real-world experiments. Strengths include addressing sim-to-real gap and enabling safe policy optimization; weaknesses may include reliance on pre-trained policies and potential limitations of frozen world models.

Read-first score

Read-first score 69.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 52.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,dataset,experiment

Topical relevance 42%

74.3

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset,github

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 44.

Keyword Scores

world model

world simulator

generative world model

model-based reinforcement learning world model

world dynamics prediction

interactive world model

video world model

Deep Analysis

Innovations

Framework employing diffusion-based world models as high-fidelity simulators to refine pre-trained policies entirely in imagined environments for robotic manipulation
Direct end-to-end policy optimization within a frozen world model, unlike prior works that use world models primarily for planning
Two-hot action encoding scheme tailored for robotic manipulation
Adoption of diffusion backbones to improve modeling fidelity

Methodology

World4RL pre-trains a diffusion world model on multi-task datasets to capture diverse dynamics. It then refines pre-trained policies entirely within this frozen world model using reinforcement learning, without online real-world interactions. The framework uses a two-hot action encoding scheme and diffusion backbones to enhance modeling fidelity.

Key Results

Extensive simulation and real-world experiments demonstrate that World4RL provides high-fidelity environment modeling and enables consistent policy refinement, yielding significantly higher success rates compared to imitation learning and other baselines.

Limitations

Reliance on pre-trained policies that may be suboptimal, potentially limiting refinement quality
Potential distribution shift between imagined and real environments due to the frozen world model, which may not generalize to unseen dynamics

Resources

Paper Project