Model Predictive Control with Differentiable World Models for Offline Reinforcement Learning

arXiv 26.3 2026 58.7 method

TLDR

Offline RL with differentiable world models enables inference-time policy optimization via gradient-based MPC, improving performance on D4RL benchmarks.

Reasoning

The paper introduces a novel inference-time adaptation framework using differentiable world models for offline RL, showing consistent gains on D4RL benchmarks. However, the approach is computationally expensive, though a tilted MeanFlow sampler reduces cost.

Read-first score

Read-first score 58.7, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 35.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,benchmark,dataset

Topical relevance 42%

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=dataset

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 406.

Keyword Scores

world model

model-based reinforcement learning world model

world dynamics prediction

generative world model

world simulator

interactive world model

video world model

Deep Analysis

Innovations

Inference-time policy optimization via a Differentiable World Model (DWM) that enables end-to-end gradient computation through imagined rollouts
Model Predictive Control (MPC) inspired framework for offline RL with inference-time adaptation
Tilted one-step MeanFlow sampler to reduce computational cost while retaining most of the performance gains

Methodology

The paper proposes a Differentiable World Model (DWM) pipeline that allows end-to-end gradient computation through imagined rollouts for inference-time policy optimization (ITPO). It uses a pretrained policy and a learned world model, and at inference time optimizes policy parameters via gradients from the DWM. The method is evaluated on D4RL continuous-control benchmarks (MuJoCo locomotion tasks and AntMaze) against strong offline RL baselines.

Key Results

The method yields consistent gains over strong offline RL baselines on D4RL benchmarks. A tilted version of one-step MeanFlow sampler recovers much of the gains at a fraction of the computational cost.

Limitations

Inference-time adaptation is expensive: rollout generation and backpropagation dominate per-step compute
The tradeoff between performance and computational cost is studied explicitly, indicating that full adaptation may not be practical for real-time applications

Resources

Paper