Official Review · NEURIPS 2024

* The authors theoretically study the capacity for (nonlinear) transformer models to perform in-context learning with a low-complexity function class. * The set-up involves pretraining transformers on sequences encoding input-output examples from a reduced-rank Gaussian single-index model. * The use of Gaussian single index models gives the transformer a chance to implement a more intricate (but still analytically tractable) in-context learning algorithm than in-context linear regression for linear data. * The restriction of the in-context target functions to a low-rank subspace gives the transformer an opportunity to internalise (during pretraining) a 'prior' that it can rely on during in-context learning for a sample efficiency advantage. * Within this setting the authors prove their main theorem, characterising the in-context sample efficiency of a transformer trained under certain conditions. * Importantly, the efficiency scales with the dimension of the low-dimensional model subspace rather than the ambient dimension. * This suggests that the pretrained transformer model is capable not only of performing ICL of Gaussian single-index models, but also of encoding the 'prior' and gaining a corresponding sample efficiency advantage from doing so. I thank the authors for their contributions in this important area of research. In my opinion the work shows at least the following strengths. 1. The work is well motivated by an important research goal within the field: advancing our theoretical understanding of more complex forms of ICL exhibited by transformers. This helps to bridge the gap between the nuanced ICL capabilities of LLMs and the simpler abilities of in-context linear regression transformers, for example. 2. The setting proposed by the authors, of learning Gaussian single-index models, appears to provide a neat example of a non-linear in-context learning capability making non-trivial use of a transformer's MLP layers, while still being analytically tractable and therefore amenable to informative comparison with baselines from the statistics and ML literature, as the authors have demonstrated. 3. The main theoretical result by the paper clearly demonstrates that pretrained transformers are capable of learning ICL for this nonlinear function class and that they are able to take advantage of the special low-dimensional structure of the function class in-context to improve their sample complexity over an ICL method that considers arbitrary directions. 4. The paper is reasonably well-organised and the writing is clear. Though for me the technical content is outside of my background and zone, with careful study I was able to understand the setting and theorem statement thanks to the detailed technical description and accompanying discussion provided by the authors. I lack the background to evaluate the soundness of the main theorem, but the high-level