Official Review · ICLR 2025

This paper explores the potential of foundation models (large language models and large vision-language models) for enhancing the exploration ability of reinforcement learning agents. The authors propose Foundation Model Exploration (FME), which integrates the foundation models into the reinforcement learning framework for intelligent exploration behavior. To be specific, the authors let foundation models execute $H$ step actions with a fixed probability. The authors conduct experiments on tasks like Gridworld, and Atari to show the effectiveness of their proposed method. ## Pros - the studied topic is interesting, i.e., exploring the possibility of foundation models in enhancing the exploration ability of the reinforcement learning algorithms - this paper is easy to read and easy to follow, the idea is simple - the authors provide detailed prompt information in the paper and the appendix ## Cons Despite the aforementioned advantages, this paper has the following flaws: - **Overclaim.** This paper is not the first work that utilizes VLMs to enhance the exploration efficiency of reinforcement learning agents. Some of the previous works [1, 2] have already explored that (but using different approaches). This paper only provides a heuristic method that integrates VLMs/LLMs into the exploration process of the reinforcement learning agents. Furthermore, the authors claim that they provide *the first fundamental study of the capacity of foundation models in traditional reinforcement learning exploration challenges*, but they only conduct experiments on very limited tasks and base algorithms, as well as a limited number of LLMs/VLMs. This cannot be claimed as the fundamental study. - **The structure of this paper can be improved.** It is somewhat strange that the authors first set their focus on LLMs in bandit problems and then turn to VLMs. Section 3 and Section 4 are less connected, making it unclear what the most important insight of the paper is. If my understanding is correct, the authors would like to stress results under the VLMs, but they spend too much space on LLM results. I strongly believe that the entire experimental section in this paper can be significantly improved. - **Lacking experiments and baselines.** This paper is also weak in that it considers too few baselines and evaluation tasks. To be specific, - the considered environments are too few. The authors only consider two tasks from Atari, which is not sufficient to show the effectiveness of their method. The authors should consider some hard exploration environments like Montezuma's revenge instead of some simple tasks. The authors are also encouraged to include more robotics tasks. If the authors decide to focus on the Atari benchmark, please add results on more tasks. I know it can be expensive, but they are essential to show the effectiveness of the proposed method - the authors should run longer environmental steps in the Pong task, as the algorithms do not converge wi