Official Review · ICLR 2025

This work builds upon previous work in contrastive reinforcement learning. What's unique to this work is how their learning algorithm conditions on a single goal; this has implications on the distribution of data within the replay buffer which ultimately impacts the learning dynamics and exploration behaviour of the agent. The authors argue that this approach results in emergent skill learning without explicitly having to define a reward function or learning curriculum for the task being solved. This argument is supported with simulated experiments on robotic manipulation and maze traversing benchmarks. The performance of their approach is compared to contrastive reinforcement learning using a range of human-specified goal difficulties and existing approaches (SAC+HER and RIS). The authors justify their claim with empirical results, leading to the insight that contrastive reinforcement learning with conditioning on a single goal demonstrates advantageous exploration and skill learning behaviours relative to existing approaches that seek to learn effective exploration and skill learning strategies. - The authors provide valuable insights into the effectiveness of training a policy conditioned on a single goal with contrastive reinforcement learning in the single-task setting. - The authors contribute to the discussion on skill learning through empirically demonstrating the emergence of skill learning with contrastive objective functions. - The authors validate each of their claims and provide discussions on counter arguments (e.g. overfitting single-task) that support their empirical results. - The authors provide sufficient information and high-quality materials to reproduce their results. - There could be more analysis of emergent skill learning, in the current draft the authors mention that fixing a checkpoint and examining the qualitative behaviour of the policy they see evidence of skill learning, further justification of this claim is required in my opinion. - The authors highlight an awareness of this point but further analysis on why this strategy works would be useful. - It would be interesting to see if this approach results in exploration strategies that are capable of solving more complex tasks. For example tasks that encounter Sussman's anomaly, where naive subgoal design may not work well. I am unsure how this approach would perform but it would be promising if it did well and might further justify the strength of this more general approach to learning to solve tasks with reinforcement learning. Also tasks with significant bottlenecks in which naive exploration strategies are often incredible sample inefficient, does this approach help improve performance on such tasks? - While the authors address the concern of overfitting to the individual task being solved, they do not explore how well the learnt representations generalise. While there are examples of solving tasks with minor perturbations there isn't a discussion of m