Official Review · ICLR 2026

This paper considers transformer's in-context learning (ICL) ability for categorical outputs. They claim that attention-only transformers are insufficient in this case and the MLP layers in traditional transformers are necessary. They claim they develop a sparse transformer parameterization that trains more efficiently. They also consider ICL in multi-question image analysis in the surgical contexts. 1. The problem considered is interesting and may potentially deepen our understanding of ICL ability of transformers. 1. The paper is very poorly written. It is very difficult to understand the mains results, both theoretical and empirical. For the empirical results, the experimental setup is described in great detail but not easy to understand. It is unclear what questions they are meant to answer, and why they can answer the questions. There is a list of empirical findings in Section 3.3, but they are not properly discussed and it is unclear if they are well supported by the experiments. For the theoretical analysis, the results are scattered in the informal discussions. It is hard to see what are the main results and their implications, why they are interesting and important. 2. The claimed sparse transformer parameterization is not properly introduced. 3. The purpose surgical example is unclear. It seems unrelated to the main theme of the paper other than it is an example of ICL. See weakness.