Official Review · ICLR 2026

This paper performs an empirical analysis of transformers trained in a supervised manner to perform linear algebra tasks. The authors design detailed experiments to identify the bottlenecks of transformers in performing such operations. Specifically, they find that transformers tend to learn properties of the input data distribution rather than the underlying matrix operation algorithms. To address this, the authors propose a new training recipe and a novel combination of architectural components to mitigate the identified limitations. Finally, they evaluate the improved architecture on three downstream applications. * The paper conducts a detailed and comprehensive analysis of the limitations of transformer-based architectures in performing matrix operations. It further proposes a new model architecture and training recipe, and evaluates them on a wide range of tasks to demonstrate their effectiveness. * The paper provides thorough details of all experimental settings, which is good for future reproducibility. * The work uses simple linear algebra tasks as a testbed to reveal that transformers may primarily learn the statistical properties of the data distribution rather than the underlying algorithmic procedures. * Overall, the paper is a solid empirical study. However, my main concern is in the motivation for investigating the transformer’s ability to perform such simple linear algebra tasks. As the authors repeatedly emphasize, (e.g., line 86: “We note that the goal of our work is in understanding the potential capabilities and limitations of transformers for linear algebra…”; line 472: “This research area is in its early stages… our goal is not to compete with purpose-built classical algorithms…”), the stated goal is primarily exploratory. However, it remains unclear why understanding transformers’ capabilities on these tasks is important, and what future directions this understanding enables. Does this line of research aim to eventually develop a single transformer that outperforms classical linear algebra solvers? Why is a new transformer architecture needed to solve problems that classical solvers already handle perfectly? It is also not evident whether the insights obtained from these experiments will translate to a better understanding of transformer usefulness in real-world analytical settings. * Many experiments in the main text lack multi-run results with different random seeds. While the appendix provides details about whether hyperparameter optimization was performed, the plots in the main text appear to come from single runs. Including multiple seeds would strengthen the statistical reliability of the results. Since the models are relatively small and trained for fewer than 10,000 iterations, adding multi-run averages and std should be feasible. * Some aspects of the writing logic could be improved. For example, in line 297 -line 300, the authors state "We now construct, step-by-step, our RangeFormer architecture, and intr