Official Review · ICLR 2024

This paper advocates for the sequential learning of options to bolster the stability of Hierarchical Reinforcement Learning, drawing inspiration from supervised learning methodologies. However, it falls short in terms of presentation quality and experimental validation. Notably, the proposed algorithm fails to demonstrate significant improvement over weak baselines, leading me to recommend rejection of this paper. The algorithm design is well-conceived, aiming to enhance the stability of Hierarchical Reinforcement Learning (HRL) by employing a boosting technique derived from supervised learning. Additionally, the proposed HRL algorithm holds promise for multi-task reinforcement learning. (a) For the related work part, the authors should put an emphasis on Hierarchical Reinforcement Learning and provide comparisons with them to show the novelty. Works on option discovery (a.k.a., skill discovery) should also be included. (b) Sequentially learning options and restricting them as weak policies could potentially lower the sample efficiency and option usability. (c) The criterion design in this algorithm is complicated and includes multiple hyperparameters, which may be hard to be fine-tuned. (d) The paper needs quite some polish, such as the layout of Figure 1 and quality of Figure 2. (e) The baselines are too weak to be supportive. More recent and advanced HRL algorithms should be included, given the extensive research in this area. (f) The results in Section 5 cannot show the advance of the proposed algorithm. (a) $Q^{\mu, \Omega}$ is not defined in Section 2. (b) The definition of the weak policy space should better be presented in the main context. Also, its design is domain-specific, which may deter the generality of the proposed algorithm. (c) The algorithm design involves multiple key hyperparameters shown in Eq. (1)-(3). Sensitivity analysis regarding them can be beneficial to show the robustness of the proposed algorithm. (d) Fixing options learned from early training stages can potentially make them under-performed and less prefered and thus never be used in the late training stages.