Official Review · ICLR 2026

This paper introduces DeepShop, an online benchmark for evaluating web shopping agents on complex, realistic shopping tasks. Starting from seed queries drawn from existing live-web benchmarks, the authors use LLM-driven diversity and complexity evolution to produce queries spanning five domains (Books, Electronics, Home, Fashion, Sports) and layered constraints (attributes, filters, sorting). Evaluation is both fine-grained (check attributes / filters / sorting separately) and holistic (task success only if all required aspects pass). Experiments on simple RAG, various web agents, and deep research systems show that all current methods struggle significantly, especially with applying filters and sorting preferences, resulting in low overall success rates and highlighting the benchmark's difficulty. 1. The paper identifies a significant limitation in existing web agent benchmarks. The focus on simple, deterministic tasks does not adequately test an agent's ability to handle the complex, multi-constraint queries that are common in real-world e-commerce. The proposed benchmark is on the right track of briging this gap of evaluating more advanced agent that has real-world application values. 2. The proposed fine-grained evaluation framework is a key strength. It allows researchers to move beyond a simple binary success metric and diagnose why an agent failed (e.g., it found the right product attributes but failed to apply the sorting preference). This is well-demonstrated in the error analysis (Section 5.4) , which identifies critical failure points like grounding, replanning, and limited action spaces. 1. The paper lack crutial details on environment and reproducibility. While offline benchmarks are static and simpler, they offer a unique advantage in controllability and reproducibility over online enviroments. A core challenge of an online benchmark is the dynamic nature of websites (e.g., prices change, review counts increase, items go out of stock). The paper does not adequately address how the benchmark's ground truth is maintained. How can an agent's success on a query like "at least 300 reviews" or "check for the lowest price" be reliably evaluated over time? This lack of clarity raises significant concerns about the benchmark's long-term utility and reproducibility. This has been a primiary issue that is discussed in previous work of online benchmarks (e.g. WebCanvas). However, the paper does not provide much details nor discussions about how this is managed in DeepShop. 2. The paper is vague about the specific e-commerce sites used. Screenshots and a prompt example suggest Amazon is a primary target, but this is not explicitly stated or confirmed as the only target. This may cause concerns in bias and robustness of the performance evaluation using the proposed benchmark. 3. The benchmark's focus on single-turn, complex queries may not be representative of realistic shopping scenariors, which is often an interactive, multi-turn process w