Official Review · NEURIPS 2023

This paper examines the possibility of taking advantage of the ability of in-context learning to use large language models to implement policy iteration so that large language models can be used to solve reinforcement learning problems without querying the gradients. Experiments have been conducted on 5 artificial environments showing that the proposed method ICPI can indeed achieve decent performance across all considered environments. Given the success of in-context learning of large language models, applying in-context learning to policy iteration is a well-motivated idea. The proposed ICPI is sufficiently different from the original policy iteration and the other recent attempts to combine the success of large language models with reinforcement learning techniques. In the experiments, 5 artificial environments are proposed and ICPI achieves better results than baselines in all environments. Detailed ablations are also provided to justify the importance of each component of ICPI. The applications of ICPI are limited. Right now it can only be applied to handcrafted small environments like chain and mini catch. It is unclear if large language models can do complex planning in more practical tasks such as Atari and mujoco tasks. Additionally, the current formulation of ICPI can only handle discrete state and action space. Still, it would be interesting to see if large language models can be used to handle realistic continuous-action robotics tasks. It also seems that a curated hint is required for more complicated tasks like mini invaders and point mass. Comparisons to other more recent methods are missing. To justify the advantage of ICPI, it would be interesting to see how ICPI compares to popular deep RL baselines (either state-based or text-based) such as PPO [1], SAC [2], and CQL [3]/IQL [4] since some recent works [5, 6] have shown that they can be used for large language models too. Some of the arguments are not supported. For example, in line 8-10, "All gradient techniques are inherently slow, sacrificing the 'few-shot' quality that makes in-context learning attractive to begin with" is not supported. ICPI also involves inferencing the entire trajectory to calculate the q function for each action, so it seems unclear whether it is faster than gradient-based techniques. A comparison of ICPI and gradient-based methods in terms of computation time is needed. The writing can be improved. The related works and methodology sections can be more concise. There are some references typos and the paragraph from 160 - 164 is repeated. [1] Proximal Policy Optimization Algorithms, https://arxiv.org/abs/1707.06347 [2] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, https://arxiv.org/abs/1801.01290 [3] Conservative Q-Learning for Offline Reinforcement Learning, https://arxiv.org/abs/2006.04779 [4] Offline Reinforcement Learning with Implicit Q-Learning, https://arxiv.org/abs/2110.06169 [5] Is