Official Review · ICLR 2024

Curriculum reinforcement learning is to train the agent on a sequence of simpler, related tasks in order to gradually progress the learning towards a difficult, often sparse-reward target task. The hope is that if there are common optimal decisions among these simpler intermediate tasks and the target task, the agent will learn transferrable skills during the process and accelerate the learning process. However, this assumption may not hold if there are unobserved confounders in the environment. This paper delves into this problem from a causal perspective, defining conditions under which the optimal decisions remain consistent. It also introduces a method to create a curriculum that aligns causally. The method is tested in two grid-world environments with unseen confounders to validate the effectiveness of the proposed algorithms. The structure and flow of the paper are easy to follow. Several simplified examples are provided to support the argument and illustration. To my knowledge, the perspective of confounders in curriculum reinforcement learning is novel. * Toyish experiments. The experiments conducted are limited to grid-world environments, which significantly narrows the scope of application. For the conclusions to be generalized, the experiments should ideally be extended to a more diverse set of environments. * Scalability. The paper does not adequately address how the proposed method scales to continuous environment variables. This is a significant concern as many practical applications in continuous or high-dimensional environment variable space. By not tackling this challenge, the authors leave a gap in understanding the full potential and limitations of their method. * Missing related work: * Hu, Xing, et al. "Causality-driven Hierarchical Structure Discovery for Reinforcement Learning." Advances in Neural Information Processing Systems 35 (2022): 20064-20076. * Cho, Daesol, Seungjae Lee, and H. Jin Kim. "Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation." arXiv preprint arXiv:2301.11741 (2023). * Huang, Peide, et al. "Curriculum reinforcement learning using optimal transport via gradual domain adaptation." Advances in Neural Information Processing Systems 35 (2022): 10656-10670. * In Algorithm 3, how is the actions sequence $\{X_1, \ldots, X_H\}$ obtained in the first place? If we have this optimal action sequence before the learning starts, why do we need RL? * In Theorem 3, one of the conditions is that: For every $j=1, \ldots, N-1$, actions $\boldsymbol{X}^{(j)} \subseteq \boldsymbol{X}^{(j+1)}$. Does this mean that the transition must be deterministic and there only exists one unique optimal solution?