Official Review · ICLR 2025

This paper tackles the challenge of continual offline reinforcement learning for goal-conditioned navigation, where agents must adapt to new tasks using only pre-collected data, without forgetting previous skills. The authors introduce Hierarchical LOW-Rank Subspaces of Policies (HILOW), a hierarchical framework that uses low-rank subspaces for scalable, memory-efficient policy adaptation. Experiments demonstrate HILOW’s superior scalability and adaptability compared to existing continual reinforcement learning methods on standard benchmarks. 1. HILOW provides an solution for goal-conditioned offline continual reinforcement learning, excelling in knowledge retention and data efficiency. 1. This paper focuses on continual offline reinforcement learning with an emphasis on goal-conditioned navigation tasks, but this specific problem is not adequately highlighted throughout. The title, as well as the related work and preliminaries sections, do not mention goal-conditioned learning explicitly. Additionally, the paper does not clearly define the continual offline reinforcement learning problem with goal-conditioned navigation tasks or explain its importance, which weakens the motivation behind this work. 2. Each section of the paper reads as a list of points, with minimal cohesion or natural progression between parts. For example, there is no clear logical connection between sections 4.3 and 4.4. It is unclear why subspace exploration is needed immediately after subspace extension. This disjointed organization makes the overall flow of the paper difficult to follow and requires significant revision to improve coherence. 3. The choice to sample anchor weights from a Dirichlet distribution lacks adequate explanation. While Dirichlet distributions are commonly used for weight sampling, relying solely on this approach may not be sufficient for finding the optimal weight combination in complex tasks. A stronger justification or alternative exploration method could improve this aspect. 4. The propose method is complex and may bring additional implementation and computational costs. 1. The baselines used in the experiments are designed for continual reinforcement learning. Is it fair to compare these with the proposed method in goal-conditioned offline continual reinforcement learning? Could you explain why these baselines were chosen and how well they address the challenges specific to goal-conditioned offline continual reinforcement learning tasks? 2. Could you provide details on the run-time and computational efficiency of the proposed algorithm? Given the added complexity from subspace extension, anchor sampling, and hierarchical imitation learning, how does HILOW’s efficiency compare to that of the baselines?