Official Review · ICLR 2026

This paper presents a new benchmark ACADREASON. - This benchmark is designed to be a challenging, reasoning-intensive benchmark for academic domains, including CS, Law, Econ, Math and Philosophy. Each domain contains 10 questions. - The benchmark is heavily human expert curated. The questions and answers are extracted and formulated by human experts from latest publications, along with hints and scoring checklist. - The paper also benchmarked the performance of latest Large Reasoning Model and Tool-used agents on ACADREASON. The results show the benchmark is very challenging. - The proposed benchmarks is a good contribution to the community. It has detailed and careful human expert annotations, which is a good effort. - The benchmark is well-positioned, as it focuses on challenging reasoning and academic domains. - The paper has benchmarked a wide range of latest LLMs and reasoning paradigms. - The benchmark is relatively small, 50 questions in total. Although the scoring hints can give a bit more fine-grained signals, but in general the size is limited. - It would be better to include a comparison section against other relevant benchmarks. - As one aspect of the contributions is the difficulty, would it be possible to quantify the difficulty and compare with other benchmark in a more explicit way?