Official Review · ICLR 2026

This paper introduces a new benchmark: GeoHOP, for remote-sensing-based geometrical reasoning tasks. The authors also proposed a baseline approach: GeoPlanner, combining LLMs with tool use. The authors have benchmarked several LLMs in the new benchmark with different settings. - The benchmark is useful and fills in one gap compared to existing reasoning benchmarks for LLM (which are mostly indoor / not remote-sensing oriented). - The paper is well written. - The authors have benchmarked a wide range of LLMs from different companies. **Experimental Evaluation Can be Stronger** The analysis in the experiments is not super extensive/impressive. I would suggest the following as a benchmark paper: - What are the failure modes that are common across all LLMs? The authors have shown the Tool execution success/failures, but more importantly, to solve the problems in the benchmark, what is the current most challenging open problem? Currently, the paper has only a few such kinds of discussions (open problems and future directions) but this is very critical for a benchmark paper. - What (fine-grained) capabilities can this benchmark evaluate? Right now, it is a bit abstract that GeoHOP can evaluate "Geometric Reasoning in RS images", we would like some discussions on fine-grained difficulties that can be quantified. For example, the benchmark can have a subset of tasks where semantic perception/grounding is especially hard, a subset of tasks where geometric grounding / relational grounding is especially hard, and a subset of tasks where multi-step long-horizon reasoning/planning is especially hard, so that readers can develop methods to address each of these challenges. **Related Works should be more extensive** Though this paper is mainly focused on geospatial reasoning + perception, I encourage the authors to discuss a broader range of reasoning benchmarks for LLMs in related works. E.g., FOLIO [1] and LogiCity [2], they are also closely related literature that is trying to evaluate similar capabilities of LLMs (especially in LogiCity the visual track requires both perception/grounding and reasoning). The authors should highlight the key challenge in geospatial reasoning compared to the reasoning problems in those benchmarks. [1] Han, Simeng, et al. "Folio: Natural language reasoning with first-order logic." arXiv preprint arXiv:2209.00840 (2022). [2] Li, Bowen, et al. "LogiCity: Advancing neuro-symbolic ai with abstract urban simulation." Advances in Neural Information Processing Systems 37 (2024): 69840-69864. See weaknesses. I will consider raising the score if the authors address my concerns during the rebuttal phase.