Official Review · ICLR 2024

This paper studies the problem of adaptation in Reinforcement Learning (RL) and proposes an adaptation strategy to improve training efficiency. This method is compatible with pre-trained neural networks and rule-based agents. The authors provide experiments in the nanoRTS environment to demonstrate the strength of their method. (1) Interesting topic: Adaptation is an interesting and motivating area for reinforcement learning. (1) Lack of related works discussion. The authors state that adaptation for RL is a largely unexplored area, while I would recommend doing a further literature review on similar topics. Some examples contain [1, 2, 3] (2) Insufficient experiments. The provided experiment results are limited. I would recommend doing more experiments with diverse tasks. For example, if the domain is focused on Reinforcement learning, you may try other tasks in the OpenAI gym [4]. (3) Lack of README file to run the code. I appreciate the authors for providing the code. However, it would further benefit the reviewers, and potential users with a README file to guide how to run the code. Reference: [1] Nagabandi, Anusha, et al. "Learning to adapt in dynamic, real-world environments through meta-reinforcement learning." arXiv preprint arXiv:1803.11347 (2018). [2] Arndt, Karol, et al. "Meta reinforcement learning for sim-to-real domain adaptation." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020. [3] Guez, Arthur, David Silver, and Peter Dayan. "Efficient Bayes-adaptive reinforcement learning using sample-based search." Advances in neural information processing systems 25 (2012). [4] Brockman, Greg, et al. "Openai gym." arXiv preprint arXiv:1606.01540 (2016). (1) What is the relationship between the proposed method with adaptive RL such as [5, 6, 7] Reference: [1] Nagabandi, Anusha, et al. "Learning to adapt in dynamic, real-world environments through meta-reinforcement learning." arXiv preprint arXiv:1803.11347 (2018). [2] Arndt, Karol, et al. "Meta reinforcement learning for sim-to-real domain adaptation." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020. [3] Guez, Arthur, David Silver, and Peter Dayan. "Efficient Bayes-adaptive reinforcement learning using sample-based search." Advances in neural information processing systems 25 (2012). [4] Brockman, Greg, et al. "Openai gym." arXiv preprint arXiv:1606.01540 (2016). [5] Khan, Said G., et al. "Reinforcement learning and optimal adaptive control: An overview and implementation examples." Annual reviews in control 36.1 (2012): 42-59. [6] Rigter, Marc, Bruno Lacerda, and Nick Hawes. "Risk-averse bayes-adaptive reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 1142-1154. [7] Eghbal-zadeh, Hamid, Florian Henkel, and Gerhard Widmer. "Context-adaptive reinforcement learning using unsupervised learning of context variables." NeurIPS 2020 Workshop on Pre-registration in Machine Learning. PMLR, 202