Official Review · NEURIPS 2024

The authors built on the recent work of Lin et. al 2023, extending their ICLR framework so that instead of being for one agent, it is for multi-agent systems; at the same time, they also analyze and provide evidence of "ICGP" (in-context game-playing) capabilities in transformers. The paper analyzes zero-sum Markov games for two players and explores learning the Nash Equilibrium. In these games, two setups are studied: one of decentralized learning (where the Nash Equilibrium must be reached without each player observing the opponent's actions) and another of centralized learning (where a transformer controls the actions of both players). The authors find important theoretical results related to transformers in decision-making, e.g., that pre-trained transformers can realize both model-based and model-free designs, and the in-context learning capability of transformers in playing normal-form games 1. **Soundness of the Claims** a) **Theoretical Grounding** The paper does a great job of formalizing and mathematically demonstrating decentralized and centralized training of transformers to achieve ICGP, and I think it is a high-impact contribution. One could argue that it is also necessary to provide more empirical evidence (which is scarce in this paper) of the dynamics being modeled, but results like Theorem 4.1 are, in my opinion, a sufficient contribution. Among other things, this work shows the capability of transformers to not only implement existing algorithms (not new in the literature) but also to adapt them to the needs of model-free designs (as far as I know, new), demonstrated by the transformer’s ability to perform exact and approximate implementations, which is important as we move to real-world applications. b) **Empirical Evaluation** The methodology followed by the authors ("J. Details of the Experiments") seems to be solid. They use the EXP3 algorithm to collect pre-training data to ensure that the data is reflective of competitive strategies and behaviors. They also customize GPT-2, with only 2 layers and 4 attention heads, tailored to match the action dimensionality in its output layer, which directly supports computing policies from the model outputs (Lines 1002-1008). The Nash Equilibrium (NE) gap is calculated by comparing the expected rewards of the max player's policy against the min player's policy over time. The gap measures how close the transformer’s induced policy is to an ideal Nash Equilibrium. Since the area of applied game theory for LLMs/multi-agent LLMs is still largely unexplored, providing such detailed information about the experiments and the code is likely going to be useful and impactful for the community. 2. **Significance** In general, I think that ICL work has a lot of impact; at this point, it is folk knowledge that ICL > fine-tuning, so papers like this one have a lot of significance for the advancement of the state of the art. In particular, Theorems 3.4 and 4.1 can prove to