Official Review · ICLR 2026

This paper studies LM agent benchmarks and proposes: (A) a taxonomy of benchmark failures, (B) a three-stage debugging pipeline (rule-based checks, LLM-as-a-judge, and a final "quality" filter that removes tasks that most models solve or that show low discriminative ability), and (C) trimmed versions of six existing benchmarks (compressed to $\approx 45\\%$ of original size). The intended outcome is a cleaner, more discriminative suite of tasks and improved cross-model separability (measured by non-overlapping confidence intervals). - **Originality.** - The paper attempts to systematize common failure modes in LM agent benchmarks and to operationalize them into a practical debugging pipeline. - **Quality.** - The authors conduct a multi-benchmark pass and surface concrete issues; several examples appear to be genuinely new (or at least under-documented) and would be useful to benchmark maintainers. - Automating checks via an LLM-judge could reduce maintainer overhead if validated on unseen benchmarks. This paper is really problematic and I apologise for the harsh tone in advance. 1) **Circular validation akin to training on the test set.** The paper derives a taxonomy from the six target benchmarks, turns that taxonomy into a long, highly specific LLM prompt, and then re-applies it to the same six benchmarks. This design is training on the test set. The Gemini LLM-judge is primed with bug patterns that are specific to the evaluation set, so its success is not strong evidence of generality. **Actionable fix:** Evaluate on genuinely unseen benchmarks not used in taxonomy construction, say Terminal Bench; preregister the taxonomy (frozen) and report zero-shot detection rates on new benchmarks. 2) **Unclear empirical value of the taxonomy.** What is the point of yet another taxonomy? What's the point of doing this rather than the ABC (Zhu et al NIPS2025)? Is the taxonomy useful in any way? It is not shown that the taxonomy increases bug-finding power beyond a simpler prompt or generic critique instructions. The taxonomy is also fairly coarse, which is tautologically exhaustive but not deeply diagnostic. **Actionable fix:** Provide ablations that compare (i) generic critique prompts vs. taxonomy-guided prompts, (ii) short vs. long taxonomy prompts, (iii) ABC prompt vs. prompt in this paper. 3) **Problematic pipeline design and selection-induced inflation of separability.** The final “quality” filter removes tasks where the frontier models can reliably solve (i.e., the easy tasks), then reports improved separability vs. randomly sub-sampled subsets of the original benchmark. Discarding all easy tasks just to artificially inflate separation is **bad practice**. To begin with, several benchmarks (e.g., GAIA) already come with difficulty tags, and I want the authors to explain this: why did the developers of GAIA not only keep the hard tasks? Wouldn't that be better for the separability of their benc