Decoupling Search from Reasoning: A Vendor-Agnostic Grounding Architecture for LLM Agents Emmanuel Aboah Boateng, Kyle MacDonald, Amardeep Kumar, Siddharth Kodwani, Sudeep Das DoorDash, Inc. {e.aboahboateng, kyle.macdonald, amardeep.kumar, siddharth.kodwani, sudeep.das2}@doordash.com Abstract
arXiv:2606.18947v1 [cs.AI] 17 Jun 2026
Production LLM agents increasingly depend on real-time search, yet native search grounding bundles retrieval policy, provider choice, evidence injection, cost, latency, and generation behavior behind a single model-provider boundary. This coupling makes grounding hard to inspect, tune, reuse, or port, and can trigger Search-Induced Verbosity that breaks strict output contracts. We present Decoupled Search Grounding (DSG), a vendor-agnostic boundary that moves grounding outside the reasoning model through an MCP-compatible gateway, exposing provider routing, source-aware context rendering, configured fallback, retrievaldepth control, and exact plus semantic caching as first-class controls. Across five frontier models on SimpleQA, FreshQA, and HotpotQA, native search leads on recency-sensitive FreshQA, but DSG exposes a stronger frontier when control matters: on SimpleQA it nearly matches native accuracy (86.1% vs. 87.7%) at 91% lower search cost, preserves concise answer contracts, and reaches a 99.4% warm-cache hit rate with 68% lower latency. Deployed as a shared production grounding layer for large-scale agentic workloads with interchangeable models, DSG matches or slightly exceeds native-search accuracy on an e-commerce query-understanding (QIU) workload while cutting search cost by over 98%. Real-time grounding is best treated as an optimizable interface boundary, not a fixed model feature.
1
Introduction
The deployment of Large Language Models (LLMs) in production increasingly centers on agentic tasks: multi-step workflows in which a model reasons over intermediate state, decides when to call external tools, and often uses search to ground actions in fresh evidence. These systems must satisfy strict cost, latency, reliability, and outputcontract constraints (Schick et al., 2023; Mialon et al., 2023; Yao et al., 2023; Li et al., 2023; Qin
Figure 1: Decoupled Search Grounding (DSG) separates retrieval from model-native generation, making provider choice, caching, and output control explicit.
et al., 2024). As static Retrieval-Augmented Generation (RAG) pipelines evolve into dynamic, toolusing workflows (Lewis et al., 2020; Karpukhin et al., 2020; Izacard and Grave, 2021; Jiang et al., 2023; Asai et al., 2024), the interface between the reasoning model and real-time search becomes a first-order systems decision. Deployed NLP systems already show that retrieval design, evaluation, latency, and serving cost can dominate model choice in practice (Murtaza et al., 2025; Chen et al., 2025; González Juclà et al., 2026; Li et al., 2025). A common integration path is native search grounding: a model-provider feature in which the model API invokes a provider-managed search stack and injects retrieved evidence into generation. Native integrations are convenient and often strong, especially on recency-sensitive questions. Teams can instead integrate external providers directly, but their APIs, pricing, result formats, ranking, and failure surfaces differ substantially, which pushes engineers toward the default native path even when production systems need explicit control over provider choice and retrieval behavior. This coupling hides
decisions that production teams commonly need to control: which provider is queried, how much context is retrieved, how costs are bounded, whether repeated queries can be cached, and how tool outputs interact with downstream schema constraints. We study this interface boundary directly. Prior work links verbose LLM responses to uncertainty and studies over-searching in search-augmented systems (Zhang et al., 2025; Xie et al., 2026); here we isolate an interface-specific failure mode, Search-Induced Verbosity, in which native search changes response style despite strict output instructions. Strict prompts such as “Provide only the final answer entity” can yield explanatory paragraphs, which matters for production systems where LLMs frequently act as intermediate nodes whose outputs are parsed, classified, cached, or routed by downstream services. We propose a vendor-agnostic Decoupled Search Grounding (DSG) architecture implemented through an MCP-compatible gateway. Instead of treating search as an opaque model feature, DSG exposes search as a structured tool layer that can normalize provider outputs, route requests, cache repeated queries, and preserve the separation between retrieval and reasoning. The key contribution is the controllable boundary itself: DSG externalizes provider policy, cache reuse, fallback, and source-aware context rendering while keeping the reasoning model interchangeable. We evaluate DSG against native search across five frontier models (GPT-4o, GPT-4o-mini, Gemini 2.5 Flash, Gemini 2.5 Pro, and Claude Sonnet 4), public QA benchmarks (SimpleQA, FreshQA, HotpotQA), and proprietary e-commerce Query Intent Understanding (QIU) datasets. Our contributions are: 1. A systems boundary for grounding: We formalize search grounding as a decoupled, provider-agnostic layer that keeps the reasoning model interchangeable while exposing retrieval policy, caching, and provider choice as explicit system controls. 2. A search-intelligence and standardization framework: We implement DSG as an MCPcompatible gateway that gives agents a stable search interface, normalizes heterogeneous provider outputs into source-aware context, and supports cache-gated retrieval with configured provider fallback across search backends.
3. Evidence on behavior, quality, and operations: We identify Search-Induced Verbosity as a prompt-compliance risk in native search grounding and show how DSG’s structured tool boundary helps mitigate it, while quantifying accuracy, cost, latency, retrieval-depth, and cache-reuse trade-offs across public QA benchmarks and a production workload. DSG makes grounding an interface decision: provider choice, cost policy, cache reuse, context rendering, fallback, and output-contract behavior become first-class controls while the reasoning model stays interchangeable. Native and decoupled search expose different trade-offs; the value of DSG is exposing that choice as an explicit, controllable decision.
2
Related Work
Toolformer (Schick et al., 2023), ReAct (Yao et al., 2023), API-Bank (Li et al., 2023), ToolLLM (Qin et al., 2024), and WebGPT (Nakano et al., 2021) establish that LLMs can use external tools and APIs. These works primarily study agent capability and benchmark performance; we focus instead on the production consequences of where the search interface sits in the model stack. RAG has become a standard mechanism for improving factuality (Lewis et al., 2020; Karpukhin et al., 2020; Izacard and Grave, 2021), with later work exploring dynamic retrieval and retrievalreasoning interaction, including FLARE (Jiang et al., 2023), RARR (Gao et al., 2023), Self-RAG (Asai et al., 2024), Adaptive-RAG (Jeong et al., 2024), and IRCoT (Trivedi et al., 2023). In deployed workflows, these retrieval choices interact with context construction, prompt optimization (Zhou et al., 2023; Boateng et al., 2026a), and strong-to-weak adaptation of smaller reasoning models (Hsieh et al., 2023; Aboah Boateng et al., 2025; Boateng et al., 2025); DSG is complementary because it fixes the grounding interface while keeping the reasoning model interchangeable. Industry applications such as multi-category e-commerce intent understanding further motivate multi-source grounding (Boateng et al., 2026b). Our contribution is orthogonal: we study whether search is embedded as opaque provider behavior or exposed as an explicit, controllable subsystem. Our evaluation combines short-form factuality (SimpleQA; Wei et al., 2024), multi-hop stress testing (HotpotQA; Yang et al., 2018), and recency-
sensitive grounding (FreshQA/FreshLLMs; Vu et al., 2024), aligning with calls for dynamic evaluation such as Dynabench (Kiela et al., 2021). For scalable assessment, we draw on LLM-as-judge and RAG evaluation work including G-Eval (Liu et al., 2023), MT-Bench / Chatbot Arena analysis (Zheng et al., 2023), ARES (Saad-Falcon et al., 2024), RAGAs (Es et al., 2024), and agentic taskcompletion evaluation (Bhonsle et al., 2025). Industry-track work increasingly documents deployed retrieval trade-offs in enterprise support, dynamic vector stores, and latency-constrained search (Murtaza et al., 2025; González Juclà et al., 2026; Chen et al., 2025; Li et al., 2025). MCP has emerged as a practical standard for connecting models to tools and data; early work studies tool-description quality and production design patterns (Hasan et al., 2026; Srinivasan, 2026). We use MCP as the connection substrate for DSG, and evaluate decoupled search grounding against proprietary native search. Distinct from these lines of work, we treat real-time search grounding itself as a controllable, vendor-agnostic interface and measure the cost, latency, caching, and outputcontract trade-offs it exposes against proprietary native search across multiple frontier models and a production workload.
3
No Search
13
DSG
13
(b) Format drift DSG 0%
≈ 27× longer
Native 10
1
10
2
78.1%
Native
353 10
3
Median response length (chars, log scale)
0
20
40
60
80
Explanation-style outputs (%)
Figure 2: Prompt-compliance diagnostic on HotpotQA. Native search can shift concise answer extraction into explanatory output, while DSG preserves the answeronly interface required by downstream parsers.
We observe that native search can shift the model from concise extraction to explanatory summarization, even when the final-answer instruction is explicit. The answer may be present, but extra prose can break exact-match evaluation or downstream parsers. Example: Prompt-Compliance Failure Instruction: “Provide ONLY the final answer entity.” Question: “Is the building at 200 West Street taller than 888 7th Avenue?” Native search output: “Based on the search results, I can compare the heights of both buildings: 200 West Street is 749 feet tall... while 888 7th Avenue is 628 feet tall...” 245 chars
The Challenge of Native Search
Native search is provider-managed grounding: the model API invokes search and injects evidence through provider-specific context handling. This one-call abstraction is attractive for agents, but it moves retrieval policy inside the provider boundary. Developers lose control over provider selection, result normalization, caching, tool-output formatting, and migration paths when a better reasoning model or search provider becomes available. When the model call is an intermediate step in an automated pipeline, this coupling creates four deployment challenges: model-provider lockin, opaque latency, fixed cost structures, and instruction-following degradation that we term Search-Induced Verbosity. 3.1
Native search induces format drift (a) Output length
Search-Induced Verbosity
Production pipelines often use LLMs as intermediate reasoning nodes whose outputs must parse as JSON, boolean flags, or short entities. Prompts therefore specify strict output contracts, such as “Provide ONLY the final answer entity.”
DSG output: “Yes”
3 chars
We treat this as a deployment-risk diagnostic, not as a claim that native search always changes output style. In the evaluated cases, native search sometimes shifted models from concise extraction to explanatory summarization despite strict finalanswer instructions. In our HotpotQA diagnostics, an affected Claude Sonnet 4 native-search run produced answers beginning with “Based on the search results...” in 78.1% of predictions; among exactmatch failures, 62.3% still contained the correct answer as a substring. Table 8 reports the diagnostic summary. This matters when a semantically correct answer is unusable because it violates an exact-entity, JSON, boolean, or categorical output contract. Headline accuracy uses task-level judge/classifier scoring, while format compliance is reported separately as a diagnostic. DSG mitigates this by keeping retrieval as a structured tool response with a stable output boundary; future work with provider-
side telemetry could further isolate the internal triggers.
4
Decoupled Search Grounding for LLM Agents
We implement Decoupled Search Grounding (DSG) as an MCP-compatible gateway between agentic applications, interchangeable reasoning models, and provider-agnostic search backends (Figure 3). 4.1
Source-Aware Tool Formatting
Unlike native integrations that hide retrieval inside a provider-specific API, our architecture exposes search through explicit, configurable tool schemas. Engineers specify the query interface and provider options once at the gateway layer, so each agent inherits them without hard-coding. When the LLM emits a search tool call, DSG executes the query through its MCP-compatible interface and normalizes provider responses into a structured internal result object. The rendered tool context is consistent across providers and pairs snippets with source URLs, giving the model explicit provenance cues for which evidence to use. These URLs also give engineers an inspectable artifact for logging, debugging, source filtering, and fallback. Returning results as a discrete tool response preserves the retrieval-generation boundary, which a provider-managed context stream collapses. 4.2
Provider Abstraction and Fallback
The gateway acts as a grounding control plane where provider choice, fallback order, and cost metadata are configurable policy applied across agents. The Provider Registry normalizes agnostic search providers (e.g., Serper, BrightData, Firecrawl, Exa). Standard HTTP providers can be onboarded through a YAML adapter that specifies endpoint, method, headers, request templates, response-field mappings, capabilities, and cost metadata; richer providers use dedicated adapters behind the same interface. This lets researchers and engineers tune grounding for different applications, including freshness, cost, latency, or source coverage, without changing the reasoning model or application prompt. A provider timeout or error response advances the request along the configured fallback chain, so transient provider failures do not interrupt the grounding context the model receives.
4.3
Search Intelligence Layer
A central advantage of decoupling is that search policy can be optimized across requests, beyond the scope of a single model call. DSG introduces a search intelligence layer that applies a tiered decision policy: exact cache lookup, semantic cache reuse, and configured provider fallback for novel queries. Let q denote a query, q̄ its normalized form, C = {(qi , Ri )} a provider-isolated semantic cache, e(·) an embedding function, and p1 , . . . , pk a configured provider fallback chain. The gateway decision is: j ∗ = arg max cos(e(q), e(qj )), j E(q̄) Re , D(q) = Rj ∗ , sj ∗ ≥ τ S(p, q), else
(1)
where sj ∗ = cos(e(q), e(qj ∗ )), E denotes an exact-cache hit, S executes the configured fallback chain p = (p1 , . . . , pk ), and τ is a configurable semantic-match threshold. The DSG cache architecture stores exact and semantic entries with provider-scoped keys, so cached results from one provider do not silently replace another provider’s output. Cache entries also carry provider and domain specific time-to-live bounds, so recencysensitive queries expire quickly while static factoid results persist, keeping high cache-hit rates from compromising grounding freshness. This keeps provider policy explicit: repeated queries can bypass external APIs, semantically similar queries can reuse compatible evidence, and novel queries fall through to the configured provider chain. Algorithm 1 summarizes the gateway execution path. The result is reusable search intelligence: cache policy, retrieval depth, provider selection, and fallback become measurable controls the system can observe and tune. Because grounding is intercepted at a single boundary, each request emits structured telemetry, including the selected provider, retrieval depth, cache outcome, end-to-end latency, and provider-side cost, so researchers and engineers can attribute grounding cost and latency per query and per application and detect regressions when a provider degrades. Expressing these controls as configuration rather than code lets a single grounding layer be governed consistently across many agents and reasoning models without retraining or reprompting. Because the cache sits behind a shared gateway, evidence retrieved for one appli-
Figure 3: Decoupled Search Grounding architecture. DSG separates agentic applications and interchangeable reasoning models from external search providers through a stable tool interface, cache-gated search intelligence, provider abstraction, operational controls, and normalized model-facing grounding context.
Algorithm 1: Cache-Gated Provider Execution Input: Query q, threshold τ , fallback chain p1 , . . . , p k Output: Rendered tool context T q̄ ← Normalize(q); R ← ExactCache.Get(q̄); if R ̸= ∅ then return RenderContext(R); end z ← Embed(q); (qj , Rj , sj ) ← NearestSemanticCache(z); if sj ≥ τ then return RenderContext(Rj ); end for p ∈ (p1 , . . . , pk ) do if p is unavailable then continue; end Rraw ← ExecuteSearch(p, q); if Rraw has results then R ← NormalizeProviderResult(Rraw ); ExactCache.Set(q̄, R); SemanticCache.Set(q, z, R); return RenderContext(R); end end return RenderContext(∅);
cation can be reused by another, so warm-cache hit rates compound as more workloads route through DSG. In Section 5, a repeated-query replay reaches a 99.4% cache hit rate, reducing benchmark latency by 68% and driving marginal search cost near zero. Appendix I summarizes the resulting deployment control surfaces, including provider policy, repeated-query reuse, source inspectability, and output-contract reliability.
5
Empirical Evaluation
We evaluate DSG as a systems choice: accuracy retained, cost paid, and controls exposed when search moves outside the model-provider boundary. Benchmarks span static factuality (SimpleQA; Wei et al., 2024), recency-sensitive knowledge (FreshQA; Vu et al., 2024), multi-hop QA (HotpotQA; Yang et al., 2018), and productionstyle e-commerce grounding. Public QA uses fixed samples, fixed prompts, and task-level scoring. SimpleQA and FreshQA use GPT-4.1 judging, with a 3-judge SimpleQA validation ablation showing 97.78% mean unanimous agreement (Appendix G.3); HotpotQA uses deterministic EM/F1. Prompt-compliance diagnostics are reported separately from headline accuracy. For QIU, we follow the multi-category marketplace intent-routing protocol of Boateng et al. (2026b): Retail (N = 7,988) covers general non-food retail queries, and Tail (Synthetic) (N = 2,335) covers rare long-tail queries. Costs are provider-side search costs per 1K queries, excluding model inference cost; latencies are benchmark latencies from evaluation outputs. 5.1
Cost-Accuracy Pareto Optimality
Native integrations package retrieval quality with fixed provider pricing. Table 1 and Figure 5 show that DSG routing creates a broader operating frontier. On SimpleQA, DSG+BrightData reaches 86.1% mean accuracy, close to native search at 87.7%, while reducing average searchprovider cost from $20.00 to $1.80 per 1K queries.
A. Retrieval Depth Is Tunable
B. Latency
84 82 80
practical sweet spot
78
GPT-4o GPT-4o-mini Claude S4
76 74
2
4
6
10
15
Average latency (ms)
SimpleQA accuracy (%)
86
C. Marginal Cost
4,570 ms
4000
0.8 68% lower
3000 2000
1,465 ms
$0.739 6.2% hit
0.6
0.4
0.2
1000 0
Search cost ($)
5000
88
$0.0045 99.4% hit
Cold
Warm
0.0
Cold
Warm
Retrieved results
Figure 4: Operational controls enabled by DSG: retrieval-depth tuning and semantic caching reduce warm-cache latency from 4,570ms to 1,465ms and nearly eliminate marginal search cost. Task
Scope
No Search
Native
DSG+BrightData
DSG+Serper
SimpleQA FreshQA QIU Retail QIU Tail (Synthetic)
5-model mean 5-model mean Gemini Flash Gemini Flash
30.8 / $0 56.4 / $0 91.10 / $0 83.08 / $0
87.7 / $20.00 72.6 / $20.00 93.40 / $7.90 87.62 / $10.37
86.1 / $1.80 68.0 / $1.86 93.81 / $0.238 86.61 / $0.386
83.3 / $0.67 67.3 / $0.66 93.90 / $0.110 87.79 / $0.146
Table 1: Main results: accuracy / search cost per 1K queries. Academic rows average five models; QIU uses Gemini Flash.
Mean SimpleQA accuracy (%)
DSG+Serper 83.3% | $0.67/1K Native Search 87.7% | $20/1K
80 ~11x lower cost at near parity
DSG+BrightData 86.1% | $1.8/1K
5.2
60
40 No Search 30.8% | $0/1K
20
0
$0.01
$0.1
$1
general-purpose providers when cost, inspection, portability, or caching matter more.
$10
Search cost per 1K queries (USD)
Figure 5: SimpleQA cost-accuracy frontier, averaged across five models. DSG+BrightData nearly matches native accuracy at roughly 11x lower search cost, while DSG+Serper provides the lowest-cost high-accuracy operating point.
DSG+Serper provides a lower-cost operating point at 83.3% accuracy and $0.67 per 1K queries. Bootstrap confidence intervals are reported in Appendix G.1. The trade-off is task-dependent. On FreshQA, native search leads on recency-sensitive questions, consistent with tightly integrated proprietary freshness or indexing advantages. This reinforces the case for decoupling: choosing the grounding backend becomes a routing decision the team controls, with native search available as one provider within the same interface when freshness dominates and
Real-World Impact: E-commerce QIU
QIU classifies ambiguous marketplace queries using catalog and web evidence, with accuracy measured on the final resolved intent (Boateng et al., 2026b); DSG serves as the shared production grounding layer for multiple applications, including the QIU model evaluated here. The QIU rows in Table 1 show the production-relevant cost-quality story: on Retail, DSG+Serper reaches 93.90% versus 93.40% for native search while reducing search cost from $7.90 to $0.110 per 1K queries; on Tail (Synthetic), it reaches 87.79% versus 87.62% for native search at $0.146 per 1K queries, over 98% below native search. This is striking because our native baseline, Gemini Flash, natively supports Google Search grounding (Gemini Team, 2025; Google AI for Developers, 2026); matching or exceeding it with general-purpose providers suggests externalized control can offset tightly coupled native search. Full provider tables are in Appendix F. 5.3
Operational Controls: Caching and Tuning
Because DSG intercepts structured tool calls, it exposes controls unavailable in black-box search. A
SimpleQA max_results sweep shows quality rising sharply from 2 to 4–6 results before saturating around 6–10 (Figure 4A), turning grounding into a tunable budget knob. In a repeated-query cache experiment with GPT-4o and DSG+BrightData, the warm pass reaches a 99.4% hit rate, reduces average latency from 4,570ms to 1,465ms, and nearly eliminates marginal search-provider cost (Figure 4B–C).
6
Conclusion
We argue that real-time grounding for LLM agents should be an optimizable systems boundary that teams control directly. By separating retrieval from reasoning, DSG keeps the reasoning model interchangeable while exposing provider policy, sourceaware rendering, retrieval-depth tuning, fallback, and warm-cache reuse as explicit controls. Across five frontier models and a production workload, it approaches native-search accuracy on public QA, matches or slightly exceeds it in deployment at far lower cost, and holds strict output contracts where native search drifts into prose. The most consequential open problem we surface is Search-Induced Verbosity, where a correct answer wrapped in prose still breaks downstream parsers; future work should measure it across models with provider-side telemetry, add contract-aware validators at the boundary, and learn adaptive routing, retrieval depth, and multi-hop strategies over the same interface.
Limitations While our DSG architecture demonstrates significant advantages in cost, latency, and prompt compliance, we acknowledge several limitations in our study and the proposed system. Reliance on Third-Party APIs The decoupled architecture fundamentally relies on the stability, latency, and data quality of third-party search vendors (e.g., Serper, BrightData, Firecrawl). Changes to these external APIs, including pricing adjustments, rate limits, or indexing behavior, could impact the cost-accuracy Pareto frontier presented in this work. Reported provider costs reflect pricing observed during our evaluation period and are expected to change over time; our contribution is the controllable grounding interface and the operating frontiers it exposes, not any specific price point. Multi-Hop Reasoning Constraints Our evaluation on the HotpotQA benchmark (detailed in
Appendix E) revealed that search augmentation, whether native or DSG-based, provides only modest gains for complex, multi-hop reasoning tasks compared to single-hop factoid retrieval (SimpleQA). This suggests that while our architecture effectively grounds models for direct queries, solving deep, multi-step reasoning problems likely requires more advanced retrieval-reasoning coordination, such as iterative retrieval, planning, or selfreflection (Trivedi et al., 2023; Asai et al., 2024; Jeong et al., 2024), beyond simple search decoupling. We hypothesize that DSG’s tool boundary is most effective for controllable retrieval, while complex multi-hop tasks may require tighter iterative loops with planning, reflection, and repeated evidence acquisition. Evaluation Methodology For academic benchmarks (SimpleQA, FreshQA), we utilized an LLMas-a-judge framework (GPT-4.1). While this methodology is highly scalable and increasingly well-supported in the literature (Liu et al., 2023; Zheng et al., 2023; Saad-Falcon et al., 2024; Es et al., 2024), automated evaluators can exhibit biases, particularly regarding formatting sensitivities. We mitigated this by utilizing a multi-judge majority voting ablation (Appendix G.3), but human evaluation remains the gold standard for nuanced qualitative assessment.
Ethical Considerations For semantic caching, the gateway is designed to cache only provider-permitted and ethically allowable context under applicable provider terms and data-use agreements, rather than indiscriminately storing retrieved content. The planned public artifact excludes proprietary QIU data, providerrestricted retrieved snippets, and internal deployment details. Upon acceptance, we intend to release non-proprietary prompts, plotting scripts, aggregate public-benchmark outputs, and DSG configuration templates.
Acknowledgments We thank Elyse Winer, Drishya Giri, Kaiwen Bian, Jamie Gu, Prabhjot Saini, Stephanie Poon, Sebastian Connelly, Tanvi Priya, Johny Rufus, and Tom Tang for their contributions, feedback, and support throughout this work. An early version of this work received the Engineering Excellence award at the 2026 DoorDash AI Hackathon.
References Emmanuel Aboah Boateng, D. Victor, B. Cassiano, N. Ehimwenma, A. Nabiha, M. Chyna, and 1 others. 2025. Using large generative models to improve the performance of weak language models in performing complex tasks. U.S. Patent Application US20250348745-A1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations. Oral. R. Bhonsle, R. Dutta, S. Vavilapalli, H. Seth, A. Jaye, Y. Chang, M. Rungta, and 1 others. 2025. Autoeval judge: Towards a general agentic framework for task completion evaluation. arXiv preprint arXiv:2508.05508. Emmanuel Aboah Boateng, C. O. Becker, N. Asghar, K. Walia, A. Srinivasan, E. Nosakhare, and 1 others. 2025. Concept distillation from strong to weak models via hypotheses-to-theories prompting. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Emmanuel Aboah Boateng, Z. Johnson, T. Xia, S. Zhang, A. Jay, J. Feng, A. Mate, and 1 others. 2026a. SAPO: Secure automated prompt optimization via multi-agent collaboration. In Proceedings of the ACM Conference on AI and Agentic Systems, pages 773–784. Emmanuel Aboah Boateng, Kyle MacDonald, Akshad Viswanathan, and Sudeep Das. 2026b. Agentic multisource grounding for enhanced query intent understanding: A DoorDash case study. arXiv preprint arXiv:2603.01486. Qinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi, Liangzhong Guo, Yuan Wang, Wei Wang, and Yunshi Lan. 2025. ComRAG: Retrieval-augmented generation with dynamic vector stores for real-time community question answering in industry. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 749–763. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. RARR: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477–16508.
Gemini Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Daniel González Juclà, Mohit Tuteja, Marcos Esteve Casademunt, Keshav Unnikrishnan, Yasir Usmani, and Arvind Roshaan. 2026. Retrieval enhancements for RAG: Insights from a deployed customer support chatbot. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 169–180. Google AI for Developers. 2026. Grounding with google search. https://ai.google.dev/ gemini-api/docs/google-search. Accessed 2026-06-15. Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E. Hassan. 2026. Model context protocol (MCP) tool descriptions are smelly! towards improving AI agent efficiency with augmented MCP tool descriptions. arXiv preprint arXiv:2602.14878. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park. 2024. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050. Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
Benjamin Chess, and John Schulman. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations. Spotlight. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 338–354.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36.
Xiaoyu Li, Xiao Li, Li Gao, Yiding Liu, Xiaoyang Wang, Shuaiqiang Wang, Junfeng Wang, and Dawei Yin. 2025. Proactive guidance of multi-turn conversation in industrial search. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 706–717.
Vasundra Srinivasan. 2026. Bridging protocol and production: Design patterns for deploying AI agents with model context protocol. arXiv preprint arXiv:2603.13417.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane DwivediYu, Asli Celikyilmaz, and 1 others. 2023. Augmented language models: A survey. arXiv preprint arXiv:2302.07842. Syed Shariyar Murtaza, Yifan Nie, Elias Avan, Utkarsh Soni, Wanyu Liao, Adam Carnegie, Cyril John Mathias, Junlin Jiang, and Eugene Wen. 2025. Implementing retrieval augmented generation technique on unstructured and structured data sources in a call center of a large financial institution. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 598–606. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight,
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledgeintensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014–10037. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. FreshLLMs: Refreshing large language models with search engine augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13697–13720. Jason Wei, Karina Nguyen, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368. Roy Xie, Deepak Gopinath, David Qiu, Dong Lin, Haitian Sun, Saloni Potdar, and Bhuwan Dhingra. 2026. Over-searching in search-augmented large language models. arXiv preprint arXiv:2601.05503. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations. Yusen Zhang, Sarkar Snigdha Sarathi Das, and Rui Zhang. 2025. Demystify verbosity compensation behavior of large language models. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 160–178. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.
A
Dataset Details
To ensure a comprehensive evaluation, we utilized a mix of public academic benchmarks and proprietary industry datasets. SimpleQA A benchmark designed to evaluate the factuality of LLMs on short, single-hop factoid questions. It is highly sensitive to hallucinations and requires precise entity extraction. We utilized a random subsample of N = 1, 000 questions. FreshQA A dynamic QA benchmark (N = 500) that tests a model’s ability to retrieve highly recency-sensitive knowledge (e.g., “Who won the most recent Super Bowl?”). This dataset specifically tests the latency and freshness of the underlying search indices. HotpotQA A multi-hop reasoning dataset (N = 1, 000) where answering the question requires retrieving and synthesizing information from multiple distinct sources. Proprietary QIU (Retail & Tail (Synthetic)) Query Intent Understanding (QIU) datasets sourced from a large-scale e-commerce platform. The Retail dataset (N = 7, 988) covers general non-food retail queries, while the Tail (Synthetic) dataset (N = 2, 335) covers rare long-tail queries. These datasets represent the complex, noisy inputs typical of real-world marketplace search.
B
Visual Summary of Full Results
Figures 6–8 provide compact visual summaries of the comprehensive tables below. The plots are intended for fast pattern inspection; the exact values and caveats remain in the corresponding tables.
C
Comprehensive Results: SimpleQA
Table 2 presents the comprehensive results on the SimpleQA benchmark (N = 1, 000) across all evaluated models and search providers. This expands upon the summary presented in Section 5.
D
Comprehensive Results: FreshQA
Table 3 presents the comprehensive results on the FreshQA benchmark (N = 500), which tests recency-sensitive knowledge. Native search is the accuracy leader across all five models; Exa was not run for FreshQA and is therefore omitted.
E
Comprehensive Results: HotpotQA
Table 4 presents the comprehensive results on the HotpotQA benchmark (N = 1, 000). Unlike SimpleQA, which tests single-hop factoid retrieval, HotpotQA requires multi-hop reasoning. Search augmentation provides modest and model-dependent gains, but DSG+BrightData is the strongest overall DSG configuration and is competitive with or better than native search for most models. All rows are reported with task-level EM/F1 scoring.
F
QIU Provider Tables
The QIU datasets are proprietary e-commerce intent-understanding evaluations. Table 5 reports the final provider comparison used in the main analysis.
G
Ablations and Diagnostics
G.1
SimpleQA Bootstrap Uncertainty
Table 6 reports bootstrap confidence intervals for the SimpleQA frontier summarized in Figure 5. We resample queries with replacement within each model/configuration, compute model-level accuracy, and then average across the five evaluated models for each bootstrap replicate. Configuration No Search Native DSG+BrightData DSG+Serper
Mean Acc. (%)
95% CI
30.82 87.74 86.12 83.34
[29.62, 32.06] [86.82, 88.64] [85.18, 87.06] [82.32, 84.34]
SimpleQA Accuracy Across Models and Search Providers GPT-4o
35.0
88.0
85.4
82.0
81.2
75.7
74.6
GPT-4o-mini
9.7
85.6
86.4
81.9
80.8
73.3
70.3
Gem-Flash
32.3
85.9
85.3
82.2
82.6
66.9
67.8
Gem-Pro
54.9
88.7
86.5
84.2
83.9
72.2
72.1
Claude S4
22.2
90.5
87.0
86.4
85.5
72.6
73.3
Mean
30.8
87.7
86.1
83.3
82.8
72.1
71.6
ch Sear No
Nati
Exa
ily Tav
80
ve
Data
ht Brig
l raw
er
Serp
c Fire
40
Accuracy (%)
60
20
0
Figure 6: SimpleQA accuracy heatmap across all evaluated models and search providers. Outlined cells mark the best configuration for each row; the mean row summarizes the overall provider pattern.
FreshQA Accuracy Across Models and Search Providers 60.0
74.0
66.2
66.8
65.8
66.8
75
GPT-4o-mini
45.8
62.8
61.0
59.0
58.0
59.2
70
Gem-Flash
61.0
75.2
71.4
71.0
71.2
71.2
Gem-Pro
62.6
75.6
68.2
68.8
69.0
68.4
Claude S4
52.6
75.2
73.0
70.8
71.2
70.2
Mean
56.4
72.6
68.0
67.3
67.0
67.2
Data ight
Serp
65 60 55
Accuracy (%)
GPT-4o
50 h
arc o Se
N
v Nati
e
l
er
Br
ily Tav
raw irec
F
45
Figure 7: FreshQA accuracy heatmap. Outlined cells mark the best configuration for each row. Native search leads across all evaluated models on this recency-sensitive benchmark; Exa was not run for FreshQA. Table 6: Bootstrap uncertainty for SimpleQA modelmean accuracy. Intervals use 10,000 bootstrap replicates over query-level correctness from the primary singlejudge results.
G.2
Retrieval Tuning and Cache Efficiency
Table 7 reports the two operational ablations summarized in Figure 4: retrieval-depth tuning and repeated-query caching. max_results
GPT-4o
GPT-4o-mini
Claude S4
2 4 6 10 15
76.8% 82.2% 82.4% 84.8% 83.2%
76.8% 81.8% 84.8% 85.8% 86.6%
82.4% 86.4% 87.4% 85.8% 86.8%
Pass
Hits
Hit (%)
Cost ($)
Latency (ms)
Cold Warm
31 497
6.2 99.4
0.7385 0.0045
4570 1465
Table 7: Operational ablations. Top: effect of max_results on SimpleQA accuracy with DSG+BrightData. Bottom: semantic cache replay on SimpleQA (N = 500), where the warm pass nearly eliminates marginal search cost and reduces average benchmark latency by 68%.
G.3
Multi-Judge Validation
To ensure the reliability of our single-judge LLM evaluation (GPT-4.1) on SimpleQA, we conducted a multi-judge validation ablation using a 3-judge
Retail (N=7,988)
100 Better region: ↑ accuracy ← lower cost
Better region: ↑ accuracy ← lower cost
Serper
95
Accuracy (%)
Tail (Synthetic) (N=2,335)
95
Native
90 90
Serper
BrightData
Native
No Search
85
BrightData
85 No Search
80
$0.01
$0.1
$1
$10
80
$0.01
Search cost / 1K queries (USD)
$0.1
$1
$10
Search cost / 1K queries (USD)
Figure 8: QIU cost-accuracy trade-offs for Retail and Tail (Synthetic). Exact provider results are reported in Table 5. Model
Search Config
Acc (%)
Cost ($)
Median Latency (ms)
P95 (ms)
GPT-4o
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
35.0 88.0 85.4 82.0 81.2 75.7 74.6
0.00 10.00 1.54 0.53 1.81 0.55 0.54
530 2717 3913 2119 2563 1804 2646
1012 4543 9595 3696 5566 8920 4933
GPT-4o-mini
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
9.7 85.6 86.4 81.9 80.8 73.3 70.3
0.00 10.00 2.00 0.76 22.62 0.90 0.93
526 1458 4087 2434 3201 2729 4258
978 2255 12212 6125 8397 7286 9823
Gemini 2.5 Flash
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
32.3 85.9 85.3 82.2 82.6 66.9 67.8
0.00 35.00 1.66 0.59 1.94 0.69 0.68
1610 3110 5836 3822 4307 3065 3143
3699 8049 18116 9550 13830 11621 11389
Gemini 2.5 Pro
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
54.9 88.7 86.5 84.2 83.9 72.2 72.1
0.00 35.00 1.80 0.67 2.17 0.76 0.78
5989 5063 9107 6539 7976 7853 8165
12207 13609 27629 19139 23464 23135 24551
Claude Sonnet 4
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
22.2 90.5 87.0 86.4 85.5 72.6 73.3
0.00 9.99 2.01 0.79 2.58 0.94 0.96
1294 4712 6380 4281 4702 4668 6837
2482 15472 17853 10894 12750 11248 13817
Table 2: Comprehensive SimpleQA results detailing accuracy, cost, and latency across all tested configurations.
majority verdict. The mean 3-judge unanimous agreement was 97.78% (range 95.9%–98.6%). The
3-judge majority provided a slight uplift (+1.15pp) over the single judge, confirming that our primary
Model
Search Config
Acc (%)
Cost ($)
Median Latency (ms)
P95 (ms)
GPT-4o
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Tavily
60.0 74.0 66.2 66.8 65.8 66.8
0.00 9.98 1.55 0.52 1.78 0.53
463 2558 3955 1976 1501 2225
1043 4179 10157 3172 3122 3600
GPT-4o-mini
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Tavily
45.8 62.8 61.0 59.0 58.0 59.2
0.00 10.00 2.15 0.77 2.68 0.75
468 1717 4705 2516 1866 2530
883 2913 13601 6128 6161 6795
Gemini 2.5 Flash
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Tavily
61.0 75.2 71.4 71.0 71.2 71.2
0.00 35.00 1.64 0.60 1.97 0.58
1348 2796 4837 2740 2166 2831
3323 8254 16012 7950 8324 8076
Gemini 2.5 Pro
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Tavily
62.6 75.6 68.2 68.8 69.0 68.4
0.00 35.00 1.82 0.62 2.06 0.61
5555 5531 8310 6051 5919 6598
22210 18083 31556 28749 21108 29817
Claude Sonnet 4
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Tavily
52.6 75.2 73.0 70.8 71.2 70.2
0.00 10.00 2.16 0.81 2.53 0.81
1182 5725 6336 4276 3723 4116
2867 17726 16846 12447 11037 11567
Table 3: Comprehensive FreshQA results. Evaluated using GPT-4.1 judge in RELAXED mode.
evaluation metric is highly stable and slightly conservative. G.4
Prompt-Compliance Diagnostics
Table 8 summarizes the Claude Sonnet 4 HotpotQA diagnostic shown in Figure 2. The diagnostic measures the specific format-drift pattern discussed in Section 3; it is separate from the final HotpotQA benchmark table in Appendix E, which uses tasklevel EM/F1 scoring. Mode No Search DSG Native
Med. Len.
Based... (%)
Extractable (%)
13 13 353
0.0 0.0 78.1
– – 62.3
Table 8: Prompt-compliance diagnostic for Claude Sonnet 4 on HotpotQA. Length is measured in output characters. “Starts Based...” is the fraction of predictions beginning with explanatory search-result framing; “Extractable Failures” is the fraction of exact-match failures that still contained the target answer as a substring.
We treat Search-Induced Verbosity as a deployment-risk diagnostic rather than a headline accuracy metric. Headline accuracy uses task-
level scoring, while this diagnostic isolates whether the final response obeys the requested output contract. The diagnostic instead isolates a productioninterface risk: in workflows that require exact entities, JSON, or categorical flags, an answer can be present but wrapped in prose that breaks downstream parsers. DSG mitigates this risk by keeping retrieval as a structured tool response with a stable output boundary. Additional Gemini example. The same pattern appears in Gemini 2.5 Flash with native grounding (Table 9). The model has the correct answer, but activating native search shifts the final response into explanatory prose that fails the exact-output contract.
Model
Search Config
EM (%)
F1
Cost ($)
Median Latency (ms)
P95 (ms)
GPT-4o
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
38.1 37.6 37.4 35.6 37.4 34.7 36.3
0.521 0.541 0.545 0.519 0.530 0.515 0.529
0.00 10.00 1.97 0.68 2.32 0.67 0.65
502 2999 4881 2364 2937 2088 2667
1127 5860 12339 4699 6462 4139 5110
GPT-4o-mini
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
29.3 39.6 41.0 39.8 39.0 37.6 36.8
0.404 0.523 0.566 0.548 0.539 0.527 0.521
0.00 10.00 2.51 0.89 3.04 0.86 0.85
487 1477 5811 3331 4499 2710 4033
835 2297 14773 5893 8049 6641 8368
Gemini 2.5 Flash
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
41.1 42.0 44.7 44.2 44.0 43.1 43.9
0.537 0.578 0.591 0.582 0.572 0.576 0.577
0.00 35.00 2.36 0.81 2.74 0.78 0.79
1873 3110 11482 5484 5587 5053 5475
8265 18943 25481 13570 16691 13261 13895
Gemini 2.5 Pro
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
44.7 45.1 47.7 46.4 46.4 46.1 45.6
0.596 0.610 0.627 0.610 0.608 0.606 0.602
0.00 35.00 2.43 50.86 2.85 0.78 0.79
5923 6798 12321 9263 10518 8872 9612
19550 23193 37805 29404 30059 41160 37290
Claude Sonnet 4
No Search Native DSG+BrightData DSG+Serper DSG+Firecrawl DSG+Exa DSG+Tavily
33.1 32.6 43.3 43.3 41.1 39.7 39.2
0.442 0.444 0.581 0.571 0.551 0.525 0.521
0.00 10.00 2.94 1.04 3.45 36.00 36.00
1264 6761 8777 5439 7711 4607 6657
3189 17811 19843 10621 16034 10549 12848
Table 4: Comprehensive HotpotQA results. Exact Match (EM) and F1 scores are deterministic. Results are more mixed than SimpleQA, reflecting the additional difficulty of multi-hop retrieval and exact-answer formatting. Tail (Synthetic) (N = 2,335)
Config No Search Google (Native) DSG+Tavily DSG+Serper DSG+BrightData DSG+Firecrawl
Acc (%)
∆ vs. No Search
Search Usage (%)
Cache (%)
Cost ($/1K)
83.08 87.62 87.75 87.79 86.61 85.61
– +4.54pp +4.67pp +4.71pp +3.53pp +2.53pp
0.0 29.6 29.8 29.8 26.9 26.9
0.0 0.0 1.15 1.73 3.19 4.31
0.000 10.373 2.354 0.146 0.386 2.056
Retail (N = 7,988)
Config No Search Google (Native) DSG+Tavily DSG+Serper DSG+BrightData DSG+Firecrawl
Acc (%)
∆ vs. No Search
Search Usage (%)
Cache (%)
Cost ($/1K)
91.10 93.40 93.80 93.90 93.81 92.20
– +2.30pp +2.70pp +2.80pp +2.71pp +1.10pp
0.0 22.5 26.9 26.9 23.2 23.2
0.0 0.0 14.4 14.9 25.53 26.77
0.000 7.900 1.800 0.110 0.238 0.969
Table 5: QIU provider comparisons for Tail (Synthetic) and Retail.
Field
Example
Field
Illustrative DSG tool context
Instruction Question
Provide ONLY the final answer entity. Which is larger, Asante Traditional Buildings or Gulangyu? Gulangyu “Gulangyu” (correct) “Gulangyu is larger. Gulangyu is an island with an area of approximately 1.88 sq...” (fails exact match)
Query
Which is larger, Asante Traditional Buildings or Gulangyu? Title: Kulangsu, a Historic International Settlement; Source: https://whc.unesco. org/en/list/1541/; Content: UNESCO World Heritage page for Kulangsu / Gulangyu. Title: Asante Traditional Buildings; Source: https://whc.unesco.org/en/list/35/; Content: UNESCO World Heritage page for Asante Traditional Buildings in Ghana.
Target No Search Native Search
Result 1
Result 2
Table 9: Additional Search-Induced Verbosity example for Gemini 2.5 Flash.
H
System Prompts and Reproducibility
A central observation of this paper is that, in the cases we evaluated, native search integrations can weaken adherence to strict output-format instructions. To ensure reproducibility and transparency, we provide the exact system prompt utilized across all models during the factoid retrieval (SimpleQA) and multi-hop (HotpotQA) evaluations.
Table 10: Illustrative source-aware DSG tool context.
Qualitative inspection suggests that preserving titles and URLs helps the model distinguish primarysource evidence from generic web snippets and prioritize evidence by source provenance. This source-aware rendering gives the reasoning model consistent cues about where each piece of evidence came from.
Standardized System Prompt You are a highly precise answering agent. Your task is to answer the user’s question as concisely as possible. CRITICAL INSTRUCTIONS: 1. Provide ONLY the final answer entity or short phrase. 2. DO NOT include any conversational filler (e.g., "The answer is..."). 3. DO NOT explain your reasoning. 4. If you must use a search tool to find the answer, do so, but your final output to the user must still be ONLY the exact entity.
Despite these explicit, capitalized instructions, in our runs some models using native search (e.g., Claude Sonnet 4 with web_search) often returned paragraphs of text beginning with "Based on the search results...". In contrast, DSG processes search results as a distinct tool response rendered from structured provider-normalized results, allowing the LLM to maintain its persona and adhere strictly to the formatting constraints. H.1
Source-Aware Context Rendering
DSG normalizes heterogeneous search responses into a common evidence record with fields such as title, url, and content. The model-facing rendering preserves this structure as numbered snippets with explicit source URLs, rather than merging retrieved text into an opaque context block. The public benchmark outputs report predictions and aggregate metrics, but omit raw retrieved snippets.
I
Deployment Criteria Comparison Dimension
Native search
DSG
Accuracy
Strong, especially on freshness-sensitive queries
Search cost
Fixed provider pricing
Latency
Provider-managed and opaque
Repeatedquery reuse
Not exposed
Portability / inspectability
Coupled to model provider
Outputcontract reliability
Can induce verbose outputs
Near-parity on public QA; task-dependent and stronger on the evaluated QIU production workload Tunable provider routing; lower searchprovider cost in our evaluations Task/providerdependent; cacheoptimizable warm path Exact + semantic cache; 99.4% warm replay hit rate (Table 7) Provider-agnostic registry with source-aware title/URL/snippet rendering Lower format drift in diagnostics (Table 8)
Table 11: Expanded deployment-criteria comparison between native search and DSG. Accuracy is not assumed to favor one interface universally: native search is strongest on freshness-sensitive QA, while DSG matches or exceeds native on the evaluated QIU production workload and exposes deployment controls unavailable in native integrations.