Learning from World Feedback: Why Model Uncertainty Fails as a Risk Signal in Model-Based RL

arXiv 2026 36.1 method, application

TLDR

Model uncertainty fails as a safety risk signal in MBRL; world feedback signals reduce collisions from 26% to 1-14%.

Reasoning

The paper provides strong empirical evidence across multiple world-model architectures, showing that model uncertainty is anti-correlated with safety. Its proposed world-feedback signals are effective, but the abstract does not specify the domain or environment, limiting generalizability claims.

Read-first score

Read-first score 36.1, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 34.

Recency 6%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=analysis

Topical relevance 29%

48.6

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Citation impact 18%

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. cited_by_count=0

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

Frontier

Rank sensitivity

Stability: volatile; rank range: 162.

Keyword Scores

model-based reinforcement learning world model

world model

world dynamics prediction

world simulator

generative world model

interactive world model

video world model

Deep Analysis

Innovations

Proposes the RLxF programme with three principles: ground risk in world outcomes, validate proxies before deployment, and substitute outcome-trained feedback models when direct world signals are unavailable.
Demonstrates that dynamics-based uncertainty penalties are anti-correlated with safety, increasing collision rates from 26% to 34% in safe model-based RL.
Introduces three world-feedback signals (sensor-derived margin via minimum lidar, time-to-collision, and an outcome-supervised feedback model g_psi trained on collision labels) that reduce collisions to 1–14% without retraining the world model or planner.
Identifies a structural mismatch: model uncertainty has support over state-prediction space while task risk has support over constraint boundaries, with empirical correlation r ≈ 0.15.
Draws an analogy between the outcome-supervised feedback model and outcome-trained reward models in RLHF, extending principles to LLM alignment.

Methodology

The study compares MPC planning with and without dynamics-based uncertainty penalties across four world-model architectures spanning a 2x MSE range, using TOST equivalence testing (n=200). It then replaces the uncertainty proxy with three world-feedback signals (lidar margin, time-to-collision, and a learned g_psi trained on prior collision labels) and measures collision rates, while also computing the empirical correlation between model uncertainty and task risk.

Key Results

Dynamics-based uncertainty penalties increase collision rates from 26% to 34%, while replacing them with world-feedback signals reduces collisions to 1–14% without retraining the world model or planner. The correlation between model uncertainty and task risk is only r ≈ 0.15.

Limitations

Findings are demonstrated in a specific safe model-based control regime and may not generalize to other RL settings or tasks.
The outcome-supervised feedback model g_psi requires prior collision labels, which may not be available in all scenarios.
The analysis of correlation (r ≈ 0.15) is based on the tested architectures and environments; the structural mismatch may vary in other contexts.

Resources

Paper