Official Review · ICLR 2025

The core innovation of this paper is the proposed method called ITES (Imagine To Ensure Safety in Hierarchical Reinforcement Learning). This method combines model-driven reinforcement learning and hierarchical reinforcement learning to ensure safety. It introduces a high-level policy and a low-level policy to generate safe intermediate subgoals and guide the agent toward the final goal. Through trajectory imagination, the low-level policy learns to reach these subgoals safely. Key contributions include: Using hierarchical reinforcement learning for enhanced safety and performance. Employing a world model to verify safety in imagination and a cost model for generating safe subgoals. Demonstrating superior performance on SafetyGym and SafeAntMaze benchmarks compared to state-of-the-art methods, especially in satisfying safety constraints. Compared to existing methods, ITES (Imagine To Ensure Safety in Hierarchical Reinforcement Learning) achieves better performance quality while maintaining comparable safety violations. Additionally, the paper explores an alternative implementation of safety in hierarchical reinforcement learning using Lagrange multipliers and demonstrates that ITES provides better safety while maintaining comparable performance to traditional safe policies in a custom long-horizon environment called SafeAntMaze. 1.The paper claims novelty in integrating safety with hierarchical reinforcement learning (HRL) and a model-based approach, but similar methods have been explored previously. For instance, Safe HIRO [1] and IAHRL [2] also focuses on hierarchical approaches for safe exploration. Provide a clearer distinction between ITES and existing approaches. Explicitly state the unique aspects of ITES, such as how the world model or subgoal generation differs from previous methods. [1] Roza F S, Roscher K, Günnemann S. Safe and efficient operation with constrained hierarchical reinforcement learning[C]//Sixteenth European Workshop on Reinforcement Learning, 2023. [2] Lee S H, Jung Y, Seo S W. Imagination-augmented hierarchical reinforcement learning for safe and interactive autonomous driving in urban environments[J]. IEEE Transactions on Intelligent Transportation Systems, 2024. 2.The use of a world model for safety verification in imagination depends on the accuracy of the learned model. If the model is imperfect, it may not accurately detect safety violations, potentially leading to unsafe behavior in the real environment. This paper does not address the limitations of model inaccuracies or present any strategies to handle model uncertainty. 1.How exactly is the safe region for subgoal generation defined, and how is it updated during the training process? Is it a static boundary, or does it adapt based on the agent's learning progress? 2.Provide a clearer distinction between ITES and existing approaches, such as [1] and [2]. [1] Roza F S, Roscher K, Günnemann S. Safe and efficient operation with constrained hierarchical reinfo