Official Review · ICLR 2026

This paper introduces a new benchmark for web agents, primarily focusing on information retrieval / deep research tasks. The authors claim that their benchmark is novel and introduces features not previously seen in other benchmarks. 1. Long, multi-question information retrieval - Each sample spans multiple questions that test both breadth and depth of reasoning, requiring agents to perform multi-hop information retrieval while maintaining contextual coherence across web pages. 2. Intermediate step verifiability - A large task broken into several, intermediate subtasks, each of which has verifiable, fixed ground truth answers. The authors provide detailed analysis of benchmark’s data statistics and discuss the evaluation metrics used to prove the uniqueness and complexity of their benchmark. The authors use these metrics to evaluate several agentic frameworks and frontier models on their benchmark, showing generally low scores across metrics like Success Rate (SR) and Completion Rate (CR). They also provide a comparative analysis of how models and agent paradigms perform across different domains and across breadth vs depth-oriented tasks. 1. The proposed benchmark introduces long-horizon, multi-question challenge to deep research agents with verifiable subtasks, where the multi-question consistency feature is an important extension to existing benchmarks that largely focus on single-fact tasks or subjective report generation. 2. Empirical result shows that the benchmark is difficult for most advanced agentic systems and frontiner models, which provides a a potential new direction the research community can make measurable progress towards. 3. The experiments and analysis are relative extensive, and covers a good representation of agent paradigms and frontier models to make sure that the observations of benchmark difficulty is not overfitting particular agent designs or models. 1. The main evaluation metrics this paper use are based on LLM as a judge, but there is no study of how well the LLM judge matches human assessment of answer quality on this dataset. This is especially important since this dataset claims to expose reliable subtask-level assessment of agent quality. If the evaluation itself is not stable enough, it puts into question all of the evaluation results presented in this work. One potential alternative, for instance, taken by the GAIA benchmark, is to specify the output format with task instructions and use stricter evaluator functions like string match. 2. One of the main contributions of this benchmark concerns subtask-level verification, which is important and useful for complex information retrieval tasks that is oft-neglected in many benchmarks. However, the paper fails to discuss this contribution in relation to previous work in the literature. For one, HotpotQA (which is cited by this paper) introduced supporting fact evaluation for reasoning, which is a proxy to checking the necessary reasoning steps are reached to