Official Review · ICLR 2024

The paper introduces Relaxed State-Adversarial Offline Reinforcement Learning (RAORL), a model-free approach for offline RL. RAORL reframes the policy problem as a state-adversarial optimization challenge, eliminating the need for explicit environmental modeling and addressing model uncertainty issues. The method guarantees policy robustness and adaptability to dynamic transitions. Empirical evaluations on the D4RL benchmark demonstrate RAORL's superiority over baseline methods in continuous-control tasks, highlighting its potential for risk-sensitive applications. 1. The paper introduces an intriguing concept by integrating adversarial robust reinforcement learning with offline model-free reinforcement learning within a state-adversarial optimization framework. 2. The paper's solid theoretical foundation and validation of the reality gap contribute to its strength. 1. The paper's structure is not well-organized, making it challenging to grasp the primary contributions and follow the narrative. Furthermore, the writing is convoluted, containing multiple unclear sentences that hinder comprehension. 2. The experimental results lack persuasiveness. Firstly, the authors do not compare their work with state-of-the-art algorithms such as SAC-N, EDAC [1], and LB-SAC [2]. Even when implementing the algorithms based on https://github.com/tinkoff-ai/CORL, this paper's results are inferior to previous methods in most Mujoco and Adroit environments. Hence, I believe there are significant limitations in the paper's empirical contributions. 3. Figure 2 should clarify the color codes representing different algorithms and indicate the corresponding perturbation magnitudes. The paper should explain how optimal state perturbations were determined in the robustness analysis. The connection between the state perturbations in the experiments and the real-world environments, as claimed in the paper, seems tenuous. I have reservations about the practical significance of this paper. 4. While the paper focuses on model-free offline RL, I find it puzzling that model-based offline RL is included in the related work. Robust RL works like [3, 4] (and others) dealing with state adversaries seem more relevant, yet the authors do not discuss them in the related work section. [1] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. NeurIPS 2021. [2] Q-Ensemble for Offline RL: Don’t Scale the Ensemble, Scale the Batch Size. Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, Sergey Kolesnikov. [3] Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. Huan Zhang, Hongge Chen, Duane Boning, Cho-Jui Hsieh. ICLR 2021. [4] Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning. Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Furong Huang. Neurips 2022. 1. How does RAORL effectively bridge the performance gap in r