Imagine to Ensure Safety in Hierarchical Reinforcement Learning

arXiv 2026 52.9 method

TLDR

Hierarchical RL with a learnable world model ensures safety via high-level subgoals and low-level imagined rollouts, outperforming baselines on long-horizon tasks.

Reasoning

The paper presents a novel integration of hierarchical policies with a world model for safe exploration, showing strong empirical results on challenging navigation and manipulation tasks. However, it lacks real-world validation and does not detail the world model architecture or training process.

Read-first score

Read-first score 52.9, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 30.

Recency 6%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Citation impact 18%

95.2

Uses OpenAlex-shaped citation metadata as a bibliometric attention signal, separate from paper quality. citation_normalized_percentile=0.95238224

Methodology quality 18%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,metric

Topical relevance 29%

42.9

Uses existing LLM keyword relevance scores normalized to 0-100. world model,world simulator,generative world model,interactive world model,video world model,world dynamics prediction,model-based reinforcement learning world model

Reproducibility 18%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Citation velocity 12%

Citation velocity estimates citations per publication-year to reduce old-paper bias. velocity=0.00

Field roles

FoundationFrontierBridgeMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 408.

Keyword Scores

world model

model-based reinforcement learning world model

world dynamics prediction

world simulator

generative world model

interactive world model

video world model

Deep Analysis

Innovations

Combines a learnable world model with hierarchical policies (high-level and low-level) to promote safety at both levels
High-level policy generates intermediate subgoals that bias exploration toward safe regions
Low-level policy uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching subgoals

Methodology

The proposed method integrates a learnable world model with two complementary policies: a high-level policy that produces intermediate subgoals to guide exploration towards safe regions, and a low-level policy that performs imagined rollouts in the learned world model to minimize unsafe actions when executing those subgoals. The approach is evaluated on long-horizon navigation and manipulation tasks with high-dimensional action spaces, comparing against existing Safe RL baselines using success rate and constraint satisfaction metrics.

Key Results

The method significantly outperforms existing Safe RL baselines in both success rate and strong empirical constraint satisfaction, consistently meeting the prescribed safety budget across seeds, while prior approaches fail to effectively solve these complex long-horizon scenarios.

Limitations

Relies on an accurate learned world model, which may be difficult to obtain in complex or partially observable environments
Hierarchical structure introduces additional training complexity and potential for suboptimal subgoal generation

Resources

Paper