Official Review · ICLR 2026

The paper introduces COGITAO (COmpositional Generalization In Transformations And Objects), a dataset for evaluating compositional generalization in abstract visual domains. Inspired by the ARC-AGI benchmark, COGITAO aims to provide a large-scale, controllable environment that enables scientific study of compositional generalization. The dataset consists of input–output grid pairs, where the output is generated by applying a sequence of transformations (e.g., rotation, translation) to the input. The dataset defines two generalization regimes: (1) compositional generalization, which tests generalization to novel transformation sequences, and (2) environmental generalization, which varies object and grid properties. The authors evaluate four architectures: Vanilla Transformer, Grid Transformer, Pondering Looped Transformer, and LLaDA and showed that they fail to generalize. 1. Originality: The dataset is based on prior datasets such as ARC and PGM, but introduced more diverse transformations (28 atomic total) and objects. 2. Quality: The authors clearly defined the two generalizations (compositional and environmental) and carefully designed the transformations and generalization splits. The authors also tested diverse architectures. 3. Clarity: The paper is well-organized and clearly written 4. Significance: COGITAO provides a generative framework for analyzing and improving compositional generalization in vision models. It offers a large-scale benchmark that can guide the development of models for challenging tasks such as ARC. 1. Motivation Clarity: The motivation for why this new benchmark is necessary could be improved. The manuscript argues that ARC’s dataset is too small and heterogeneous for systematic study. However, ARC was intentionally designed to be small to probe generalization without overfitting through large-scale training, and related benchmarks such as PGM (Hill et al., 2019) already operate at million-scale sample sizes. Thus, dataset size alone does not fully justify the need for COGITAO. To clarify its unique contribution, the authors could provide concrete examples of the new scientific questions or analyses that COGITAO enables, but prior benchmarks cannot. 2. Connection to Broader Context: The paper deliberately focuses on training models (e.g., transformers) via supervised learning and omits large pretrained models or in-context learning, arguing that training details for foundation models are opaque. While this focus is understandable from a scientific control perspective, it leaves unclear how COGITAO relates to the current frontier of model capabilities. Recent progress, such as reports of strong performance by O3 on ARC, raises the question of whether large-scale pretrained systems may already solve the generalization task. e.g. recent report on ARC by O3? If so, what specific research questions remain unresolved that make COGITAO valuable? Does the dataset primarily expose limitations of small architectures train