Official Review · ICLR 2026

The proposed AtomWorld is a benchmark designed to evaluate spatial reasoning capabilities of LLMs in the domain of crystalline materials. The authors assume that LLMs possess the capability to understand and manipulate 3D atomic structures for crystal design. They designed a series of cognitive tasks in materials discovery and benchmarked several state-of-the-art LLMs, including Gemini 2.5 Pro, GPT-o3, GPT-o4-mini, Deepseek Chat, Llama-3 70B, and Qwen-3. All benchmark tasks uses CIF format as textual input of the LLMs along with a natural language action prompt, requiring certain actions to edit the CIF in order to output a valid modified CIF structure. The benchmark results suggest high success rates on simple operations like adding or changing single atoms, while failures on rotations and atom swapping. The work try to disentangle different failure modes, but the analysis suggests LLMs are tackling the tasks as pattern-matching rather than physics-informed reasoning. 1. The work aims at building a comprehensive benchmark that assesses whether LLMs can handle mechanical operations and higher-level cognitive tasks. It is working at building the foundation for future material discovery with LLM capabilities. 2. Multiple LLMs across different model families are benchmarked in this work. And it reported an interesting observation on how all LLMs consistently fail at complex spatial reasoning tasks. 1. The benchmarking data are oversimplified. This work evaluates LLMs exclusively on simple prototype crystals with at most three elements, which raises questions about whether the findings generalize to multi-component systems or more complex lattice structures. 2. In addition, the benchmark rely on extremely small sample sizes often tens or hundreds. E.g. CIF-Repair was tested on 22 samples, CIF-Gen on 20 samples, and only 10 samples ran DFT in StructProp. Therefore the conclusions drawn from these tests about model capabilities or reasoning quality are not reliable. 3. The StructProp benchmark design is intuitive but fundamentally flawed, as it assumes LLMs understand the property concepts being tested without validating this assumption. Interpretation of the StructProp results can only be regarded as bias from memorized patterns. 1. All demonstrated examples use relatively simple crystals with at most three elements. Have you benchmarked LLMs on materials with higher compositional complexity or complex crystal systems, e.g. from the Materials Project database? 2. The CIF format contains many redundant information and can produce lengthy token sequences. Have you evaluated the LLMs' performance on token length of the CIFs? How does the token counts distribute for each task? How does performance correlate with the length of the input CIF file? Does the context window limitations affect performance of the LLMs? 3. Specifically when atom coordinates involved, have you tested whether generating a structure where the target atom appears early or late