Official Review · ICLR 2025

This paper introduces DSPO, a method for offline safe reinforcement learning (RL) that learns safe policies without requiring explicit cost labels, using only a small set of safe demonstrations. Unlike prior approaches that depend on predefined Markovian cost functions, DSPO overcomes this limitation through a two-stage process: (1) training a discriminator to extract safety signals from trajectories, and (2) using a conditional diffusion model to generate high-return, safety-compliant trajectories, from which a policy is learned via behavior cloning. Experiments on SafetyGym, BulletGym, and MetaDrive show that DSPO achieves safer policies with better returns than the tested baselines. 1. **Interesting topic:** The authors explore an engaging and important topic in offline RL—cost-free safe reinforcement learning. Advancing research in this direction holds significant potential for improving learning from safe demonstrations. 1. **Limited Contribution:** While the authors claim three major contributions—a more practical problem setup, a novel approach combining a safety-transformer with a diffusion model data generator, and the construction of an offline dataset suite—these contributions seem limited in significance. 1.1. **Problem Setup:** Similar problem formulations, which assume the absence of ground-truth cost functions or rely on learning preferences from a few safe demonstrations, have been explored in prior works on inverse safe RL [1, 2] and meta-safe RL [3]. 1.2. **Methodological Novelty:** The safety-transformer has been widely explored in prior research [4], and classifier-free diffusion models in (safe) RL / RL dataset curation have also been extensively studied [5, 6, 7]. The authors do not sufficiently clarify how their approach differs from previous work, nor do they provide a comprehensive comparison with related diffusion model-based safe RL approaches, such as FISOR [9] and Safe Decision Diffuser [10]. 1.3. **Dataset Suite:** The proposed dataset suite is a modified version of the OSRL benchmark [8], created by selecting safe demonstrations and masking cost information. 2. **Vague Experimental Validation:** The main experimental results, presented in Table 1, focus only on reward returns. For safe RL tasks, it is crucial to also report detailed cost/safety metrics to ensure thorough validation. 3. **Additional Comments:** Please refer to my specific questions for further points. Reference: [1] Schlaginhaufen, Andreas, and Maryam Kamgarpour. "Identifiability and generalizability in constrained inverse reinforcement learning." In International Conference on Machine Learning, pp. 30224-30251. PMLR, 2023. [2] Kim, Konwoo, Gokul Swamy, Zuxin Liu, Ding Zhao, Sanjiban Choudhury, and Steven Z. Wu. "Learning shared safety constraints from multi-task demonstrations." Advances in Neural Information Processing Systems 36 (2024). [3] Lin, Qian, Zongkai Liu, Danying Mo, and Chao Yu. "An Offline Adaptation Framew