arXiv CSopen access

To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

Preprint. Under review.

To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling Qinyuan Wu1 Soumi Das1 Mahsa Amani1 Arijit Nag1 Seungeon Lee1 Krishna Gummadi1 Abhilasha Ravichander1 Muhammad Bilal Zafar2,3 1 Max Planck Institute for Software Systems 2 Ruhr University Bochum 3 UAR RC Trust Correspondence: [email protected]

arXiv:2605.00737v1 [cs.AI] 1 May 2026

Abstract Agentic AI architectures augment LLMs with external tools, unlocking strong capabilities. However, tool use is not always beneficial; some calls may be redundant or even harmful. Effective tool use, therefore, hinges on a core LLM decision: whether to call or not call a tool, when performing a task. This decision is particularly challenging for web search tools, where the benefits of external information depend on the model’s internal knowledge and its ability to integrate potentially noisy tool responses. We introduce a principled framework inspired by decision-making theory to evaluate web search tool-use decisions along three key factors: necessity, utility, and affordability. Our analysis combines two complementary lenses: a normative perspective that infers true need and utility from an optimal allocation of tool calls, and a descriptive perspective that infers the model’s self-perceived need and utility from their observed behaviors. We find that models’ perceived need and utility of tool calls are often misaligned with their true need and utility. Building on this framework, we train lightweight estimators of need and utility based on models’ hidden states. Our estimators enable simple controllers that can improve decision quality and lead to stronger task performance than the self-perceived set up across three tasks and six models.

Introduction

At their core, agentic AI architectures enable AI models, such as LLMs, to extend their functionality by calling external tools (Yao et al., 2023; Singh et al., 2025). When solving challenging tasks, LLMs can strategically augment their internal (parametric) knowledge, skills, and abilities by utilizing external tools. The crucial decision to call or not to call a tool lies with the LLM. Our focus in this paper is on how LLMs exercise their agency in making tool calling decisions. Today, we lack principled ways to assess the quality of tool calling decisions made by LLM agents. Most evaluations are coarse-grained—they test whether tool calling improves the aggregate accuracy (Yao et al., 2023; Qian et al., 2025; Feng et al., 2025)—and overlook finer-grained aspects of performance. The lack of such assessments makes it difficult to investigate principled approaches to enhance (improve) agentic decisions to call tools. Table 1 demonstrates the need for such assessments for an example task, where models can invoke a Web search tool. Across different models, we see that (a) while tool calling helps in aggregate, (b) optimal tool calling can achieve much better performance than always calling tools with significantly fewer calls, and (c) self-decisions made by models today to call tools are far from optimal in terms of accuracy. Our goal is to develop a framework to understand and improve these sub-optimal tool calling or function calling decisions by LLM agents. We use tool calling and functional calling interchangeably throughout the paper. Inspired by the theory of rational choice (Peterson, 2017; Tversky & Kahneman, 1974), we posit that the goal of tool calling should be to help the model maximize the expected improvement in task performance (utility gain) afforded by tool use, under cost constraints. Intuitively, a model should only call a tool for task instances where its performance improves, i.e., the tool call has positive utility, and under budgetary constraints, the model should prioritize instances with higher utility under the budget constraints. The key challenge for the model then is to estimate or predict the utility of a tool call for a given task instance. 1

Preprint. Under review.

We now propose a framework for assessing LLM function calling decisions along three dimensions: 1. Necessity: Does a model M require external help to solve a task T ? That is, can M not solve T satisfactorily using its own internal parametric knowledge? 2. Utility: Will a model M benefit from calling a function F , when solving a task T ? That is, to what extent will M’s performance on T improve or worsen by calling F ? 3. Affordability: Is it cost-effective for a model M to call a function F when solving a task T ? That is, does the gain in utility from calling F justify the additional cost of calling F ? Theoretically, we make two important observations about need and utility dimensions of our framework: First, a model’s utility from calling a tool on a given task is upper bounded by its need, i.e., how poorly it performs when solving the task independently. If a model can perfectly solve a task on its own, tool use cannot yield positive utility. Second, estimating need for tool calling depends only on the model and the task, while estimating utility requires additional knowledge about the tool. However, learning to predict the behavior of a complex tool can itself be highly challenging. As we will show later in Section 4.3, this distinction is important for assessing and optimizing tool calling decisions in practice. Practically, using our framework, we study LLM function calling through three lenses (Peterson, 2017; Tversky & Kahneman, 1974). Normatively, an LLM should call tools only when needed: that is, when the tool provides positive utility, and prioritize cases with the highest expected utility gain, which often correspond to more challenging instances for the model, under a given budget. Descriptively, we examine how LLMs actually behave in practice (the S ELF - DECISION setting), including suboptimal tool use and adherence to cost constraints. Prescriptively, we explore how to improve tool calling so that decisions in practice better match the normative ideals. We operationalize our framework on six open source models ranging from 3 – 120 billion parameters, including both instruction-tuned and reasoning models. We evaluate their performance at three question-answering tasks with two web search tools namely, Google 1 and Perplexity Search 2 . We propose experimental setups to measure true need and true utility that determine normative (optimal) tool calling behavior. We infer LLMs’ (self-)perceived need and (self-)perceived utility that guide their (self-)decisions to call tools. Our investigation uncovers the key source of the performance gap between the optimal (oracle) decisions and LLM (self-)decisions to call tools. We find that tool-calling is not universally beneficial—it does not always improve utility, and in some cases can even hurt. We also identify a misalignment between perceived and true quantities. Although models are internally consistent (perceived need and utility are correlated), these perceptions deviate from true signals, leading to suboptimal decisions and a gap to the oracle. Beyond misperception of need and utility, we observe that LLMs struggle to make rational decisions under affordability constraints. To address these gaps, we propose two latent estimators of true need and true utility, enabling better-aligned decisions and improved performance. Inspired by prior work, hidden states better predict LLM performance than natural language outputs (Orgad et al., 2025; Snyder et al., 2024). The estimators still fall short of the optimal setup, reinforcing our theoretical insight that oracle-level performance necessitates modeling tool behavior, which is inherently challenging. In summary, our paper makes the following three contributions: (i) Inspired by Rational Choice theory, we propose a framework to assess the optimality and efficiency of LLM tool use. The framework enables us to move beyond aggregate accuracy metrics and measure finer-grained quality criteria. (ii) Our experiments, spanning six models, show that contemporary LLMs struggle with correctly perceiving the need and utility of tools, leading to suboptimal too-calling decisions. (iii) We propose an improved tool calling mechanism that estimates the need and utility of the tool using the internal states of the underlying LLM. Our estimators lead to better performance; however, in line with our theoretical insight, filling the gap completely would require better models of the tools themselves.

Related Work

Tool-Augmented and Cost-Aware LLMs. Recent work shows that large language models (LLMs) can effectively interact with external tools such as web search, calculators, and APIs (Schick et al., 2023b; Qin et al., 2023). Early approaches integrate tool use into the reasoning process (Yao et al., 1 https://serpapi.com/ 2 https://docs.perplexity.ai/docs/getting-started/integrations/mcp-server

Preprint. Under review.

Model GPT-OSS-120B Qwen3-30B-A3B Qwen-3-30B-IT Gemma3-27B-IT Mistral3.1-24B-IT Llama3.2-3B-IT

N O T OOL

A LWAYS T OOL

S ELF - DECISION

O PTIMAL

Score

Calls

Score

Calls

Score

Calls

Score

Calls

0.61 0.70 0.68 0.60 0.72 0.56

0 0 0 0 0 0

0.78 0.80 0.82 0.80 0.81 0.80

500 500 500 500 500 500

0.73 0.81 0.82 0.80 0.79 0.74

152 342 452 462 172 340

0.83 0.88 0.87 0.85 0.85 0.87

300 252 284 297 345 249

Table 1: Performance on the entity task under different tool-use strategies. N O T OOL: no access to tools. A LWAYS T OOL: tool is always invoked. S ELF - DECISION: model autonomously decides. O PTIMAL: oracle policy selecting the best decision per instance.

Tasks 𝒯

🧠Model’s Perceptions

Entity Task

Perceived Need: Do I need help? Perceived Utility: Will the tool help? Perceived Affordability: Where will it help most?

"In a paragraph, could you tell me what you know about RL models?"

InvivoQuery Task "how does walmart 401k match work"

BFCL Task "Who is the performer of 2024 Super Bowl halftime show?"

🧠: To call or not to call….. Task Instance x

🤖

True Need: Can the model solve it alone? (measured by NO TOOL performance) True Utility: Does the tool improve results? (compare NO TOOL vs ALWAYS TOOL) True Affordability: Use tools where they help most (pick top gains from NO TOOL → ALWAYS TOOL)

Generate Response y

Model (𝓜) Tool Response r Tool Description

NO TOOL: y = 𝓜(x)

Decide to Call π(x) = 1

External Tool 𝓕

ALWAYS TOOL y = 𝓜(x,r) SELF-DECIDED: π(x) = 1, y = 𝓜(x,r); π(x) = 0, y = 𝓜(x);

Figure 1: Given input x, the model M decides π ( x ) ∈ {0, 1} to call a tool (response r) or not, producing y = M( x, r ) or y = M( x ). We compare NO TOOL, ALWAYS TOOL, and SELF-DECISION, and evaluate decisions via need (requires help), utility (performance gain), and affordability (cost vs. gain), distinguishing perceived vs. true quantities.

2023) and improve tool selection during inference (Schick et al., 2023a), while more recent methods enhance self-directed tool use through grounding, execution feedback, and reinforcement learning (Lu et al., 2024; Qiao et al., 2024; Singh et al., 2025; Qian et al., 2025; Feng et al., 2025; Jin et al., 2025; Li et al., 2025b; Wang et al., 2025). In parallel, a growing body of benchmarks evaluates aggregate tool-use performance (Qin et al., 2023; Li et al., 2023; Patil et al., 2024; Huang et al., 2023), as well as whether models make correct tool-calling decisions under different scenarios, such as access limitations (Ross et al., 2025); see (Qu et al., 2025) for a survey. Recent work also studies how LLMs decide whether and which tools to invoke, learning tool-selection policies via prompting, fine-tuning, or reinforcement learning, and developing systems for API routing (Schick et al., 2023b; Qian et al., 2025; Patil et al., 2024; Eisenstein et al., 2025). At the same time, tool use introduces non-trivial costs in latency, computation, and API usage, motivating cost-aware planning (Wu et al., 2025) and budget-constrained scaling (Liu et al., 2025). However, existing methods evaluate decisions primarily through downstream task success or global cost-performance trade-offs, without assessing whether individual tool calls are well-justified. As a result, it remains unclear whether a tool call is necessary, beneficial, redundant, or harmful, or whether it is justified by its cost. Web Search and Retrieval Augmentation. A prominent application of tool-augmented LLMs is web search and retrieval augmented generation (Lewis et al., 2020; Liu et al., 2023; Xue et al., 2025; Kirsten et al., 2025). However, search and retrieval can also introduce noise, latency, and non-determinism (Kim et al., 2025; Kirsten et al., 2025; Mallen et al., 2023), motivating adaptive retrieval strategies Jiang et al. (2023); Su et al. (2024). Most approaches rely on uncertainty estimation to decide when to retrieve, either by eliciting uncertainty through fine-tuning (Kadavath et al., 2022; Amayuelas et al., 2024; Kapoor et al., 2024; Moskvoretskii et al., 2025) or by training auxiliary predictors such as factuality classifiers (Gottesman & Geva, 2024), semantic entropy (Kossen et al., 2024), and self-assessed confidence (Chen et al., 2026). These signals guide retrieval in adaptive RAG systems (Jeong et al., 2024; Ding et al., 2025; Yao et al., 2025; Li et al., 2025a; Asai et al., 2024). However, while necessity is modeled prior to tool invocation, utility is only evaluated after retrieval Gou et al. (2023), rather than being assessed as an expected benefit before invoking the tool. 3

Preprint. Under review.

Operating Frameworks for Assessing and Optimizing Tool Calling

Problem Setup As illustrated in Figure 1, we consider a setting in which a large language model M receives a task instance x from a task T and must decide whether to rely on its parametric knowledge or invoke an external tool to perform the task. Let F denote the tool, with textual description dF specifying, for instance, its name, functionality, and applicable use cases, among potentially richer attributes. Given an input x and a tool F with description dF , the model M decides whether to generate a response independently or to call F to assist in producing the final output. In the process, the model (implicitly) follows a tool-call policy πM,F , where πM,F ( x ) = 1 when F is invoked and πM,F ( x ) = 0 when F is not invoked. Hereon, we use π ( x ) for convenience. If the model elects to call the tool, it generates a query q and receives a response from the tool r = F (q). The final output is produced as y = M( x, r ), where r = ∅ if no tool is used. However, tool invocation can also incur costs, including added latency, monetary expense, and other overheads; we denote this cost by C . To assess the quality of a model’s tool-call policy πM,F , we introduce two additional reference policies, N O T OOL and A LWAYS T OOL (see Figure 1). The model’s performance with these reference policies is needed to measure the intrinsic need for tool use and the intrinsic utility of the tool. We formally define the experimental setups for the three policies below. 1. S ELF - DECISION: The model M is given the task input x ∼ T and tool description dF . It autonomously decides whether to invoke the tool, i.e., π ( x ) ∈ {0, 1}, and generates its response. This is the paradigm typically used in modern tool-augmented LLMs Yao et al. (2023); Schick et al. (2023b). 2. N O T OOL: The model M is provided only with the task input x ∼ T and is not given access to the tool or its description. In this setup, the model relies entirely on its parametric knowledge, leading to the tool calling decision π ( x ) = 0. 3. A LWAYS T OOL: The model M is given the task input x ∼ T and tool description dF , but is required to invoke the tool. Thus, π ( x ) = 1 for all inputs.

Evaluating Tool Calling Decisions Thus far, there has been no principled way to evaluate and enhance the quality of the decision π ( x ), as outcomes in the self-decided setting are entangled with factors such as task need and tool utility. To address this limitation, we study tool calling through the normative, descriptive, and prescriptive lenses of decision theory (Peterson, 2017; Tversky & Kahneman, 1974). The normative view defines what ought to happen (oracle decisions), while the descriptive view captures what models actually do. Finally, we take a prescriptive view and propose a controller framework that closes this gap by learning predictors that guide tool-calling decisions toward normative behavior. 1. Normative lens: Optimal Tool Calling Decisions. From a normative perspective, optimal tool calling should occur only for instances in which the model truly requires assistance and can benefit from it. Therefore, we define ground-truth measures of need and utility. We obtain these ground truths by comparing the model’s performance with and without tool access. Let s NT ( x ) and s AT ( x ), normalized within [0, 1], denote the performance scores of the model’s responses under N O T OOL and A LWAYS T OOL, respectively. We define three oracle quantities: True Need: This quantity measures whether the model needs a tool call for a given instance x. Intuitively, if the model performs sufficiently well without tools, then a tool call is unnecessary; otherwise, it is required. More generally, this notion can be defined with respect to any taskspecific quality metric (in our case, a factuality score). Formally, we denote this binary label by N⋆ ( x ) ∈ {0, 1}, where N⋆ ( x ) = 1 indicates that the score from N O T OOL response (s NT ( x )) falls below a predefined quality threshold. True Utility: This quantity measures whether a tool call improves performance. We compare the scores under A LWAYS T OOL and N O T OOL (s AT ( x ) and s NT ( x )), and assign a ternary label U⋆ ( x ) ∈ {−1, 0, +1}, where +1 indicates that the tool call improves performance (positive) over no tool, 0 indicates no effect (neutral), and −1 indicates that it degrades performance (negative). True Affordability: This quantity captures which high-utility instances should receive tool calls under a budget constraint. We define the marginal utility gain from a tool call as ∆⋆ ( x ) = s AT ( x ) − s NT ( x ). Given a budget that allows at most K tool calls over a dataset D , the optimal allocation selects the 4

Preprint. Under review.

top K instances with the largest ∆⋆ ( x ). We denote this set by SK⋆ and define the total utility gain as Gain⋆K = ∑ x∈S ⋆ ∆⋆ ( x ). This provides a criterion for determining which tool calls are worth making. K

2. Descriptive lens: Model’s (Self) Tool Calling Decisions. From the descriptive view, we observe the model’s actual tool-calling behavior, reflecting what occurs in practice when the model autonomously decides whether to invoke a tool. Perceived Need: Under N O T OOL, we prompt the model to indicate whether a tool call is needed for a given instance x. Unlike true need, which is defined based on performance, perceived need is b ( x ) ∈ {0, 1}. obtained directly from the model’s response. This yields a binary label N Perceived Utility: We treat the model’s tool-calling decision under S ELF - DECISION as a proxy for b ( x ) = 1; otherwise, U b ( x ) = 0. perceived utility. If the model decides to invoke the tool, then U Perceived Affordability: Under S ELF - DECISION with a budget of K tool calls, we define perceived affordability by the first K instances selected by the model for tool-calling. We compute the resulting [ K , using the true marginal gains ∆⋆ ( x ) over the K instances. gain, Gain 3. Prescriptive lens: Improving Tool Calling Decisions. If there is misalignment between descriptive observations and normative ground-truth labels, the resulting decisions are suboptimal. We therefore need a controller framework that better approximates the normative/true labels. For each input x, let h( x ) denote the hidden-state representation of the final token that we use to train lightweight multi-layer perceptron (MLP) classifiers. The goal of these classifiers is to estimate the true need and utility for any given input. Latent Need Estimator (LNE): This estimator predicts true need N⋆ ( x ) from hidden states h( x ) , e ( x ) ∈ {0, 1} that predicts the need/no-need obtained under N O T OOL. The output is a binary label N for calling the tool. Latent Utility Estimator (LUE): This estimator predicts whether a tool call will be beneficial, using either the hidden state h( x ) under N O T OOL (LUEx ) or under A LWAYS T OOL with the tool description included (LUEx,dF ). We train these models on a binarized version of true utility, where helpful cases (U⋆ ( x ) = +1) are treated as positive, and neutral/negative cases are treated as negative. We do not train a separate affordability model. Instead, we rank instances by the confidence scores of the utility estimator (LUE) and allocate the budget to the top-K instances predicted to be helpful. ] K , using This induces a budget-aware tool-calling policy. We then compute the resulting gain, Gain ⋆ the true marginal gains ∆ ( x ) of the top-K instances. This decomposition into three lenses highlights three distinct questions: when tools are truly needed, whether the model’s behavior reflects this need, and whether simple predictors/estimators can yield improved tool-calling decisions that better align with true need.

Experiments and Findings

In this section, we first describe the experimental setup, and then analyze need, utility, and affordability from both normative (ideal) and descriptive perspectives. Our analysis reveals a consistent misalignment between these views, helping to explain the patterns observed in Table 1, particularly the gap between O PTIMAL and S ELF - DECISION, which indicates inefficiencies in tool use. Finally, we show that a simple binary classifier based on internal representations improves alignment with the normative framework, leading to better overall performance. Models and Tool. We evaluate six open-source models from five families, spanning 3B to 120B parameters and including both instruction-tuned and reasoning variants. Table 3 lists the models along with their corresponding Hugging Face links. We focus on web search as the tool of interest for three reasons. First, it is widely used and central to real-world LLM applications. Second, its utility is highly instance-dependent: some queries can be resolved using parametric knowledge alone, while others require external retrieval. Third, each search call incurs a non-trivial financial cost, since web search operations are expensive, making this setting well-suited for studying affordability. Together, these properties make web search a representative and challenging testbed for analyzing tool use. Web search is implemented via a FastMCP server backed by Google Search (SerpApi), 5

Preprint. Under review.

returning up to five results (title, snippet, URL), which are appended to the task instance as input. We additionally replicate the Entity Task using Perplexity search, Brave Search, and Tavily Search, observing consistent results across providers. Results for other search tools appear in Appendix G. Datasets and Performance Metrics. We evaluate on three datasets. Entity task consists of entitycentric prompts in which the model generates a descriptive paragraph about a given entity extracted from real-world chat logs (Karnam et al., 2026). InVivoQuery task comprises factual questions derived from real-world user requests (Karnam et al., 2026). These queries may benefit from external tool use, enabling the study of tool-selection behavior in realistic settings. For both open-ended tasks (Entity and InVivoQuery), we follow prior work on long-form factuality (Song et al., 2024) and use an LLM-as-judge to extract and verify claims. Three annotators endorsed 99.7%, 99.9%, and 92.1% of the LLM’s verdicts. We show the details of the human annotation in the Appendix B. BFCL task (Berkeley Function Calling Leaderboard) evaluates function-calling accuracy and tooluse capabilities (Patil et al., 2025). Given ground-truth answers, we use an LLM-as-a-judge for comparison; since answers are short and unambiguous, no human annotation is required (Zheng et al., 2023). Dataset construction details and examples are provided in Appendix A.2.1, and implementation details of the scoring pipeline are provided in Appendix A.3. Experimental Setup. Models are served via vLLM (Kwon et al., 2023), with a maximum of 512 generation tokens and a temperature of 0. We use a two-stage framework: the model first decides whether to invoke web search, then generates a response conditioned on search results when applicable. Details are provided in Appendix A.

Normative Lens: Measure True Need and Utility. We operationalize True Need and True Utility by comparing model performance under the N O T OOL and A LWAYS T OOL. True Need corresponds to instances where N O T OOL performance is Low or Mid, indicating the necessity of external tool (Figure 2). True Positive Utility captures performance improvements from no tool to with tool use (e.g., Low → Mid/High), while True Negative Utility reflects performance degradation. Instances with no change are categorized as neutral utility.

7 67 20 2 162 90 High 1 51 100 Low Mid

Neutral

Positive

True Need

Performance with tool Low Mid High

Need and Utility

Performance without tool

4.1

Negative

Figure 2: True need and true positive utility are correlated, but not perfectly aligned. Rows are grouped by the model’s (GPTOSS-120B) factuality scores under N O T OOL (parametric knowledge), while columns show scores under A LWAYS T OOL. Scores are bucketed into low (0–0.1), mid (0.1–0.9), and high (0.9–1). Cells above the diagonal indicate positive utility, while those below indicate negative utility. The bracket highlights the true need, where low N O T OOL scores reflect insufficient parametric knowledge.

Tool calls can hurt performance when there is no true need. Across models and tasks, we observe three consistent patterns (Figure 2). First, tool call is highly effective in the True Need regime, where the model cannot solve the task using parametric knowledge alone. In this region, 51% of instances show performance gains (177/(177 + 2 + 169)). Second, tool calls can degrade performance when the model already performs well without them. When N O T OOL performance is High, incorporating tools leads to negative utility in 34% of cases (52/(52 + 152)), indicating unnecessary or harmful intervention. Third, most instances lie on the diagonal, suggesting that tool call is often redundant and does not affect outcomes. These patterns generalize across all models and tasks (Figures 10, 23, and 38). Detailed score distributions for all datasets are shown in Figures 9, 22, and 37. Overall, true need and utility define a normative ideal for tool use: a model should invoke tools only when they yield positive utility. We provide an example where web search has negative utility in Appendix C. Descriptive Lens: Measure Perceived Need and Utility. To estimate perceived need, we test whether the model requires external assistance under the N O T OOL setting. We design three prompt variants: (1) a structured JSON response, (2) a direct question (“Do you know the answer?”), and (3) “Do you need help?” We then analyze the responses to measure perceived need. All prompts exclude tool-related information, isolating the model’s knowledge of the tool. We measure perceived utility through tool-calling behavior under the S ELF - DECISION setting, where the model is provided with both the tool description and the task input. A tool invocation is interpreted as an indication of perceived positive utility. Full prompts are provided in Appendix A. 6

Preprint. Under review.

(a) GPT-OSS

(b) Qwen3-A3B

No Call

301 27

(d) Gemma3-IT

(e) Mistral-IT

No Need

Call

169

35 305 11 149

Tool Calling

Call

443 19

Perceived

Need

No Call

No Need

Tool Calling

Need

Tool Calling

Call

Need

Perceived

No Need

No Call

Perceived

No Need

435 18

Tool Calling

32 126

Need

No Call

Call

70 278

Perceived

No Need

293 49

Tool Calling

Perceived

Need

No Call

Call

149

Tool Calling

No Need

No Call

Perceived

Need

(f) Llama3.2-IT

Figure 3: Perceived need is only partially aligned with tool call (perceived utility). The x-axis shows the model’s perceived need, and the y-axis shows perceived utility / tool-call decisions.

(a) GPT-OSS

Tool Calling

329

177

140 22

Perceived

No Need

269 12 201 18

Need

True

303 14

Perceived

Need

True

No Need

No Need Need

Perceived

Need

No Need Need

Need

No Need

True

69 106

Need

No Need

38 338 8

116

Tool Calling

Call

No Call

Call

No Call

Call

No Call

Call

105 24

144

178

138 53

(g) GPT-OSS

(h) Qwen3-A3B

(i) Qwen3-IT

94 264

True Utility

284 32

True Utility

309 40

True Utility

237 134

True Utility

70 253

Other Positive

No Call

Other Positive

Call

Other Positive

No Call

Other Positive

Call

Other Positive

True Utility

Other Positive

256 69

(b) Qwen3-A3B

Tool Calling

Perceived

No Need No Need Need

14 138

True

205 143

Need No Need Need

True

No Need Need

Perceived

No Need

No Need Need

Perceived

Need

No Call

202 107

(j) Gemma-3-IT (k) Mistral3.1-IT (l) Llama3.2-IT

Figure 5: The perceived need and utility are not aligned with the true need and utility. Entity Task; Top: true vs perceived need. Bottom: true vs. perceived utility across models. Most models are self-consistent: perceived utility (tool calling) correlates with perceived need, with two models showing substantial deviations. In the absence of tool descriptions, model behavior is driven solely by perceived need— i.e. whether external assistance is required. Introducing tool descriptions adds a second layer: decisions depend on both perceived need and the model’s assessment of tool utility. Although tool call generally aligns when models report a need for external help (Figure 3), the relationship is imperfect. Models sometimes invoke tools despite reporting no perceived need, and fail to do so even when the perceived need is recognized. Thus, tool call does not follow directly from perceived need; instead, tool availability introduces an additional decision layer, creating systematic gaps between belief and action. Perceived need also varies substantially across models, indicating that it is model-dependent rather than task-intrinsic. Some models like MistralIT and Llama3.2-IT consistently over- or under-estimate need respectively, leading to divergent judgments of true need. These patterns persist across tasks (Figures 25 and 40). Even as models’ perceptions are self-consistent, their perceptions do not align with true need and utility. As illustrated in 1 63 Figure 4, there is a clear mismatch between perceived need/utility and true positive utility. Consequently, the model’s perceptions 1 are insufficient for optimal decision-making. The same observation is held for other models and tasks in Figures 11, 24, and 39. Figure 4: Perceived signals only We provide a breakdown of the misalignment in Figure 5 for the partially align with true utility. Entity task, Figure 26 for the InVivoQuery task, and Figure 41 for Venn diagrams of True Positive the BFCL task. Across all settings, the confusion matrices reveal Utility, Perceived Need, and Per- consistent misalignment between perceived and true need, as well ceived Utility for GPT-OSS-120B as between perceived utility (i.e., tool-calling decisions) and true on the entity task. Ideally, Per- utility (i.e., positive, other (negative or neutral)). While larger ceived Utility ⊆ Perceived Need models (e.g., GPT-OSS-120B, Gemma-3-27B-IT) exhibit slightly ⊆ True Positive Utility, but this better calibration, substantial errors remain. Smaller models, by nesting is violated, explaining the contrast, show strong systematic biases, either overestimating suboptimal performance of S ELF - their need for tools or failing to recognize it altogether. This misDECISION . calibration is even more pronounced in utility decisions, where models frequently overuse or underuse tools. Overall, these findings indicate that LLMs are unreliable judges of when tool call is necessary or beneficial. 64

33 79

True Positive Utility Perceived Need Perceived Utility

Preprint. Under review.

4.2

Cost and Affordability

Next, we study how tool calls should be allocated under explicit budget constraints. Ideally, a rational model should allocate tool calls to instances that yield the highest utility gain. We measure the utility [ K ) lenses (Section 3), gain over N O T OOL under both the normative (Gain⋆K ) and descriptive (Gain enabling a direct comparison between ideal/true and realized/perceived gains. We vary the fraction of instances allowed to invoke tools, K ∈ [1, 100]%. For a dataset of 500 instances, K = 100% allows tool use on all instances, while K = 80% permits at most 400 calls. We describe the budget limitation and the current state of cost in the prompt; the details can be found in the Appendix A. We fix a total budget B = $10,000. Assuming uniform cost per call, each K implies a per-call cost of B /K (e.g., K = 80% ⇒ $25 per call). For each K, we compute: (a) the utility gain over N O T OOL for the selected set of K instances and (b) the total number of tool calls made by the model. All models exhibit a consistent gap between true (ideal) and perceived utility gain, indicating suboptimal tool-call decisions. Figure 6 summarizes the affordability results. We compare true [ K , dashed lines). Boxes denote utility gain (Gain⋆K , solid lines) with perceived utility gain (Gain perceived utility gain under the no cost description setting (C ). As expected, true utility gain plateaus as K increases, reflecting diminishing returns. In contrast, perceived utility gain varies substantially across models: Gemma and GPT-OSS achieve relatively high gains, while Llama and Mistral show weaker improvements despite increased tool calls, indicating inefficient budget utilization.

Under budget constraints, models struggle to prioritize tool calls. Figure 14 measures alignment between tool calls and utility rankings using NDCG Järvelin & Kekäläinen (2002). Alignment is weak, indicating poor prioritization of high-utility queries. Under tight budgets (e.g., a single call), some models (GPT-OSS, Llama, Mistral) achieve high NDCG, suggesting they can identify the top-utility instance. However, this does not scale: as the budget increases, NDCG drops sharply, revealing inconsistent ranking and poor allocation across decisions between true and perceived setup. Details of NDCG computation are in Appendix D.2.1.

Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT

Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description

0.25

Utility Gain over No-Tool

Cost information has uneven effects. Without cost descriptions, most models (except Mistral and Llama) achieve slightly higher utility gains than in the cost-aware setting. For Mistral and Llama, the no-cost condition yields substantially larger gains, suggesting that these models do not effectively incorporate cost information into their tool-use decisions.

0.20 0.15 0.10 0.05 0.00

20%

40%

60%

80%

% Queries with Tool Calls

100%

Figure 6: Utility gain over the N O T OOL under varying cost constraints. Solid lines show optimal allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotModels consistently violate budget constraints. Figure 13 plots ted lines indicate always-calling. tool calls against per-call cost under an explicit budget setting (Prompt A.4.1). While most models follow the expected trend, they systematically exceed the budget. Qwen3-30B-A3B adheres most closely, followed by GPT-OSS-120B, whereas Llama and Mistral show weak cost sensitivity. Gemma and Qwen3-30B-IT perform the worst, frequently violating constraints even at high costs (e.g., > $50 per call). In the implicit cost setting (Prompt A.4.1), where models must infer allowable calls, all models fail to regulate usage (Figure 15), indicating a lack of internal budget tracking. Although explicit instructions partially improve behavior (Figure 6), overall performance remains suboptimal, highlighting limitations in cost-aware decision-making. 4.3

The Controller Framework

Prior results reveal a consistent misalignment between perceived and true need and utility, leading to suboptimal tool-call decisions, particularly under budget constraints. At the same time, prior works (Snyder et al., 2024; Orgad et al., 2025) show that models encode useful signals about whether they know the answer, but fail to reliably express them in their outputs. Motivated by this gap, we propose a prescriptive or control mechanism that operates directly on latent representations to guide tool-call decisions. Our approach requires no finetuning and is applicable to any pretrained LLM. We train lightweight predictors on hidden states using supervision from our normative framework—specifically, binary MLP classifiers for True Need and True Utility. For each model, we extract 8

Preprint. Under review.

Task

Entity Task

InvivoQuery Task

BFCL Task

Model GPT-OSS-120B Qwen3-30B-A3B Qwen-3-30B-IT Gemma3-27B-IT Mistral3.1-24B-IT Llama3.2-3B-IT GPT-OSS-120B Qwen3-30B-A3B Qwen-3-30B-IT Gemma3-27B-IT Mistral3.1-24B-IT Llama3.2-3B-IT GPT-OSS-120B Qwen3-30B-A3B Qwen-3-30B-IT Gemma3-27B-IT Mistral3.1-24B-IT Llama3.2-3B-IT

No Tool 0.61 (0) 0.70 (0) 0.68 (0) 0.60 (0) 0.72 (0) 0.56 (0) 0.53 (0) 0.58 (0) 0.46 (0) 0.50 (0) 0.55 (0) 0.45 (0) 0.36 (0) 0.20 (0) 0.29 (0) 0.42 (0) 0.30 (0) 0.17 (0)

Always Tool 0.78 (500) 0.80 (500) 0.82 (500) 0.80 (500) 0.81 (500) 0.80 (500) 0.59(500) 0.65 (500) 0.62 (500) 0.59 (500) 0.58 (500) 0.50 (500) 0.73 (328) 0.60 (328) 0.71 (328) 0.66 (328) 0.66 (328) 0.63 (328)

Optimized 0.83 (300) 0.88 (252) 0.87 (284) 0.85 (297) 0.85 (345) 0.87 (249) 0.71 (235) 0.78 (239) 0.71 (313) 0.67 (259) 0.72 (238) 0.61 (251) 0.76 (129) 0.64 (146) 0.74 (148) 0.73 (103) 0.71 (135) 0.66 (160)

Self-decided 0.73 (152) 0.81 (342) 0.82 (452) 0.80 (462) 0.79 (172) 0.74 (340) 0.56 (433) 0.41 (235) 0.26 (458) 0.42 (326) 0.54 (270) 0.44 (1376) 0.66 (196) 0.52 (261) 0.67 (351) 0.62 (360) 0.60 (302) 0.58 (346)

LNE 0.78 (351) 0.81 (329) 0.81 (339) 0.75 (292) 0.78 (279) 0.79 (429) 0.59 (442) 0.63 (371) 0.61 (465) 0.58 (469) 0.59 (478) 0.50 (465) 0.65 (207) 0.56 (264) 0.58 (206) 0.61 (218) 0.57 (240) 0.58 (263)

LUEx 0.77 (341) 0.79 (249) 0.79 (306) 0.75 (284) 0.77 (267) 0.78 (383) 0.56 (164) 0.60 (176) 0.55 (269) 0.56 (358) 0.54 (112) 0.48 (237) 0.55 (130) 0.42 (156) 0.51 (138) 0.50 (83) 0.46 (143) 0.42 (153)

LUEx,d 0.75 (293) 0.78 (248) 0.81 (271) 0.78 (308) 0.80 (263) 0.78 (354) 0.57 (163) 0.59 (177) 0.54 (250) 0.56 (344) 0.55 (93) 0.48 (211) 0.12 (123) 0.15 (154) 0.13 (134) 0.07 (67) 0.13 (132) 0.16 (161)

Table 2: Task performance across experimental setups. Values outside parentheses report factuality scores, while those in parentheses indicate the number of tool calls. representations across layers and select the one with the best predictive performance. Using five-fold cross-validation over the full dataset (Table 2), we find that latent estimators consistently outperform the model’s explicit decisions in predicting both need and utility across all models. Implementation details are provided in Appendix A.6. Latent Need Estimator (LNE). The latent need estimator (LNE) uses the model’s final-token representation to predict true need via an MLP. As shown in Figure 7, LNE improves need estimation across models, with larger gains for smaller models (e.g., Llama3.2-3B). Latent predictors consistently outperform explicit decisions and improve downstream performance (Table 2), yielding higher factuality and fewer unnecessary tool calls. 1.00

Accuracy

Perceived Need

Latent Utility Estimator (LUE). The latent utility estimator (LUE) predicts true utility. Following Section 3, we consider two variants: LUEx and LUEx,dF . As shown in Figure 18, both outperform perceived utility, indicating models encode signals of tool usefulness not reflected in decisions. However, adding tool descriptions yields limited and inconsistent gains: LUEx,dF is often comparable to or worse than LUEx , suggesting current descriptions are noisy and unreliable for capturing true utility. Taken together, latent predictors better recover true need and utility than model perceived decisions. Need is easier to estimate and more directly improves performance, while utility prediction is harder and sensitive to tool descriptions.

LNE

0.80 0.60 0.40 0.20 0.00

27B 20B 30B B-IT 24B -3B ma3- SS-1 en3- 3-30 l3.1- a3.2 Gem GPT-O Qw Qwen Mistra Llam

Figure 7: LNE improves true-need prediction across models, with larger gains for smaller models. Confusion matrices appear in Figure 17. The same pattern holds for InvivoQuery (Figure 31) and BFCL (Figure 46).

Improving utility under budget via external estimators. Latent estimators enable more effective tool-call control under budget constraints by selecting top-k instances based on predicted probabilities. As shown in Figures 21, 36, and 51, this consistently yields higher utility under the same budget, with especially strong gains for smaller models. Notably, Llama3.2-3B achieves decision quality comparable to much larger models (e.g., Gemma3-27B, GPT-OSS-120B). These results do not imply that LNE and LUE fully solve tool-use decisions, but highlight the value of external controllers that better exploit latent signals. Even simple controllers improve decision quality without modifying the base model, while more advanced ones could leverage predicted need and utility to rank actions and make principled, budget-aware decisions.

Concluding Discussion

We study when a language model should call an external tool in this paper. We propose a framework that decomposes tool-calling decisions into necessity, utility, and affordability, and analyzes them through normative, descriptive, and prescriptive lenses. Our results reveal a consistent misalignment between models’ perceived signals and normative/true need and utility, leading to suboptimal tool use. To address this, we introduce lightweight estimators based on latent representations that better approximate true need and utility. These estimators improve decision quality and overall performance without modifying the base model, suggesting that effective agent behavior may require explicit control mechanisms beyond the model’s native decision process. However, our approach has 9

Preprint. Under review.

limitations. The estimators are not perfect and may introduce errors. Future works may investigate why tool calls can lead to negative utility, how tool descriptions influence both perceived and latent utility, and develop more accurate predictors and more effective controllers for tool-calling decisions.

Disclosure of LLM use in Research

We used generative AI tools, including GitHub Copilot 3 and Claude Code 4 , to assist with selected aspects of this work. Specifically, these tools were used to support code development by generating implementation details based on high-level designs specified by the authors, and to help draft portions of the experimental setup descriptions from code written by the authors. We also employ LLM-based evaluators (LLMs-as-a-judge) to assess model outputs. To ensure reliability, we conduct human evaluation on a randomly sampled subset of instances and verify that the automated judgments are well-aligned with human annotations. For manuscript preparation, generative AI tools were used in a limited capacity for editing purposes, including shortening paragraphs and correcting grammar and spelling. All core ideas, methodological design, experimental decisions, and interpretations of results were developed and verified by the authors. All AI-assisted outputs were carefully reviewed and validated by the authors to ensure correctness, originality, and alignment with the intended scientific contributions. The authors take full responsibility for the content of this paper.

Contribution The overall research direction and experimental design were conceived and discussed collectively by all co-authors. Specific contributions are as follows: Q. Wu & S. Das led the core framework design. Q. Wu also implemented the primary codebase and conducted the main entity-task experiments. M. Amani was responsible for the InvivoQuery task dataset construction and its corresponding experiments. A. Nag handled the BFCL task dataset construction and experiments. S. Lee conducted the Perplexity, Brave, and Tavily Search results for the entity task. K. Gummadi, A. Ravichander, and B. Zafar provided senior guidance and feedback on framework design, experimental methodology, data analysis, and manuscript writing.

References Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Yang Wang. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 6416–6432, 2024. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=hSyW5go0v8. Lihu Chen, Gerard de Melo, Fabian M Suchanek, and Gaël Varoquaux. Query-level uncertainty in large language models. In ICLR, 2026. Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. Rowen: Adaptive retrievalaugmented generation for hallucination mitigation in llms. In Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp. 12–21, 2025. Jacob Eisenstein, Reza Aghajani, Adam Fisch, Dheeru Dua, Fantine Huot, Mirella Lapata, Vicky Zayats, and Jonathan Berant. Don’t lie to your friends: Learning what you know from collaborative self-play. arXiv preprint arXiv:2503.14481, 2025. 3 https://github.com/features/copilot 4 https://github.com/anthropics/claude-code

Preprint. Under review.

Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https://arxiv.org/abs/2504.11536. Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without generating a single token. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3994–4019, 2024. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738, 2023. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023. Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7036– 7050, 2024. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 7969–7992, 2023. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew G Wilson. Large language models must be taught to know what they don’t know. Advances in Neural Information Processing Systems, 37: 85932–85972, 2024. Sai Keerthana Karnam, Abhisek Dash, Krishna P. Gummadi, Animesh Mukherjee, Ingmar Weber, and Savvas Zannettou. Bowling with chatgpt: On the evolving user interactions with conversational ai systems. In ACM Web Conference, 2026. Hanna Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin, and Kimin Lee. When {LLMs} go online: The emerging threat of {Web-Enabled}{LLMs}. In 34th USENIX Security Symposium (USENIX Security 25), pp. 1729–1748, 2025. Elisabeth Kirsten, Jost Grosse Perdekamp, Mihir Upadhyay, Krishna P Gummadi, and Muhammad Bilal Zafar. Characterizing web search in the age of generative ai. arXiv preprint arXiv:2510.11560, 2025. Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927, 2024. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 11

Preprint. Under review.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459–9474, 2020. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 3102–3116, 2023. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025a. Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025b. Tengxiao Liu, Zifeng Wang, Jin Miao, I Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, et al. Budget-aware tool-use enables effective agent scaling. arXiv preprint arXiv:2511.17006, 2025. Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. Webglm: towards an efficient web-enhanced question answering system with human preferences. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pp. 4549–4560, 2023. Yining Lu, Haoping Yu, and Daniel Khashabi. GEAR: Augmenting language models with generalizable and efficient tool resolution. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 112–138, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.eacl-long.7. URL https://aclanthology. org/2024.eacl-long.7/. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.546. URL https://aclanthology.org/2023.acl-long. 546/. Viktor Moskvoretskii, Maria Marina, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, and Alexander Panchenko. Adaptive retrieval without self-knowledge? bringing uncertainty back home. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6355–6384, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.319. URL https://aclanthology. org/2025.acl-long.319/. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=KRnsX5Em3W. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544–126565, 2024. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. 12

Preprint. Under review.

Martin Peterson. An introduction to decision theory. Cambridge University Press, 2017. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025. Shuofei Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3550–3568, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.195. URL https://aclanthology.org/ 2024.naacl-long.195/. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025. Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshi Suhara. When2call: When (not) to call tools. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3391–3409, 2025. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı́, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: language models can teach themselves to use tools. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023a. Curran Associates Inc. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023b. Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning. arXiv preprint arXiv:2505.01441, 2025. Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. On early detection of hallucinations in factual question answering. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2721–2732, 2024. Yixiao Song, Yekyung Kim, and Mohit Iyyer. VeriScore: Evaluating the factuality of verifiable claims in long-form text generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9447–9474, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.552. URL https://aclanthology.org/2024. findings-emnlp.552/. Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12991–13013, 2024. Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124–1131, 1974. Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv e-prints, pp. arXiv–2504, 2025. 13

Preprint. Under review.

Duo Wu, Jinghe Wang, Yuan Meng, Yanning Zhang, Le Sun, and Zhi Wang. Catp-llm: Empowering large language models for cost-aware tool planning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8699–8709, 2025. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Liu Weichuan, Lei Hou, and Juanzi Li. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27022–27043, 2025. Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, et al. Wildhallucinations: Evaluating long-form factuality in llms with real-world entity queries. arXiv preprint arXiv:2407.17468, 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023.

Experimental Setup

A.1

Models

Open-source models. Locally-hosted models are served with vLLM Kwon et al. (2023) using a single-pass generation pipeline. The context window is set to 4,096 tokens, GPU memory utilisation to 90%, and tensor parallelism is configurable via --tensor-parallel-size. Unless a modelfamily override is specified, all open-source models are generated with a maximum of 512 output tokens, temperature τ = 1.0, and top-p = 1.0. Name in paper

Shorten for space

HuggingFace model ID

Link

GPT-OSS-120B Qwen3-30B-A3B Qwen3-30B-IT Gemma3-27B-IT Mistral3.1-24B-IT Llama3.2-3B-IT

GPT-OSS Qwen3-A3B Qwen3-IT Gemma3-IT Mistral3.1-IT Llama3.2-IT

openai/gpt-oss-120b Qwen/Qwen3-30B-A3B Qwen/Qwen3-30B-A3B-Instruct-2507 google/gemma-3-27b-it mistralai/Mistral-Small-3.1-24B-Instruct-2503 meta-llama/Llama-3.2-3B-Instruct

https://huggingface.co/openai/gpt-oss-120b https://huggingface.co/Qwen/Qwen3-30B-A3B https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507 https://huggingface.co/google/gemma-3-27b-it https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct

Table 3: Model names used in this paper and their corresponding identifiers on Hugging Face. A.2

Task and Dataset

We introduce three datasets used in our evaluation. A.2.1

Main: Entity Task Dataset Construction

In real-world applications, search is typically associated with entity-centric factual questions. Successfully using search requires the model to determine when external information is needed, how to formulate an effective query, and how to incorporate retrieved results into its final answer. Therefore, we focus on the Entity Task, where we evaluate model factuality on an entity-centric questionanswering benchmark. For each entity e in the dataset, the model is given the prompt: “In a paragraph, could you tell me what you know about {entity}?” 14

Num of Google Search Results

Preprint. Under review.

300 200 100 0

Figure 8: Distribution of the number of Google search results across all entities in the entity dataset. The violin depicts the kernel density estimate of the result count distribution. The horizontal line indicates the median. Extreme outliers, defined as values beyond 1.5× the interquartile range, are excluded for visual clarity. and is asked to generate a free-form paragraph. We adopt this entity-centered question format, inspired by prior work (Zhao et al., 2024) that evaluates LLMs’ knowledge of individual entities and shows that LLMs perform poorly on long-tail entities. To construct a realistic set of entities, we employ a two-stage, claim-based extraction pipeline using GPT-4o on the InVivoGPT dataset Karnam et al. (2026), which consists of real-world ChatGPT conversation logs. In the first stage, GPT-4o (temperature=1) extracts all named entities from user and assistant messages and classifies them according to the Wolfram Language Entity Types, a taxonomy covering 21 high-level categories (e.g., geographic entities, people, computing-related entities, and medical entities) with fine-grained subtypes. Each entity is returned as a structured JSON object containing its text span, specific type (e.g., City, ProgrammingLanguage), and high-level category. In the second stage, an optional verification step checks whether each extracted entity exists in the real world (on the internet) and whether its assigned category is correct, correcting it if necessary. To scale this process efficiently, we use the OpenAI Batch API to process up to 25,000 requests per batch, storing results as JSON files per message. From the extracted entities, we randomly sampled 500 entities for this study. Among them, 262 entities have corresponding Wikipedia pages, while 238 do not. Figure 8 shows the distribution of the number of Google search results for these entities. The majority of entities have relatively low search result counts. We list the examples of the entity task dataset as following: “In a paragraph, could you tell me what you know about Adréa?” “In a paragraph, could you tell me what you know about National Guards?” “In a paragraph, could you tell me what you know about Damodaran?” “In a paragraph, could you tell me what you know about CDA01?”

InvivoQuery Task Dataset. We further construct evaluation queries based on the same source dataset. Specifically, we select entity-centered user prompts (i.e., prompts containing extracted entities), retain only English queries, and randomly sample 250 queries that triggered a web search and 250 that did not, yielding a total of 500 queries. We focus on entity-centered queries because they better reflect realistic user behavior, where information needs are typically anchored around identifiable entities such as people, organizations, locations, or products. Such queries provide a more grounded basis for evaluation, as they are associated with verifiable facts and reduce ambiguity in assessing response correctness. Moreover, entity-centric prompts are particularly well-suited for analyzing retrieval and tool-use behavior: queries involving well-known or static entities can often be resolved from internal knowledge, whereas those involving long-tail, ambiguous, or rapidly evolving entities are more likely to require external retrieval (e.g., web search). This distinction allows us to more effectively evaluate a system’s ability to decide when to rely on parametric knowledge versus when to invoke external tools.

Preprint. Under review.

“what happened to chaosium around the kickstarter for 7th edition?” “Does Sam’s club accept EBT online for grocery delivery same Day or next day because I wanted to purchase the Sam’s club plus membership” “should i get sugar free metamucil or the NOW psyllium husk” “how does walmart 401k match work” “Trump admin cutting $20M in DC security funding after federal law enforcement ordered to increase presence By Landon Mion, 3 hrs ago Fox Fox News Follow The Trump administration plans to cut millions in security funding for Washington, D.C., despite the president also directing federal law enforcement to increase its presence in the city because of its ẗotally out of controlc̈rime. In a grant notice posted last week, the Federal Emergency Management Agency (FEMA) said that D.C.’s urban security fund would receive $25.2 million, a 44% year-over-year reduction. The Department of Homeland Security, which oversees FEMA, said on Friday it slashed funds to multiple cities to be consistent with the c̈urrent threat landscape.C̈hicago, New York City, Los Angeles, San Francisco and Jersey City also had their security funds cut, but the decrease in D.C. was the largest for any urban area that received funding from the program last fiscal year” “What are the top 5 soulslike games? Don’t include titles by fromsoftware”

A.2.2

BFCL Task Dataset

We additionally evaluate models on the BFCL V4 Web Search dataset from the Berkeley Function Calling Leaderboard (BFCL) Patil et al. (2025). This dataset contains 100 multi-hop questions spanning diverse real-world topics, each annotated with single-hop sub-questions and corresponding ground-truth answers. Rather than evaluating full reasoning chains, we treat these sub-questions as standalone queries, resulting in 314 atomic questions. We focus on sub-questions to isolate and more precisely evaluate a model’s retrieval and tooluse capabilities. Full multi-hop questions entangle multiple factors, such as reasoning quality, intermediate decomposition, and retrieval, making it difficult to attribute errors to specific components. By decomposing them into atomic sub-questions, we reduce this confounding effect and enable a more controlled assessment of whether the model can correctly decide when to invoke web search and retrieve relevant information for a single, well-defined information need. “What is the most expensive tea in the world?” “Which country produces DaHong Pao?” “Who is the richest billionaire in China?” “Who is the performer of 2024 Super Bowl halftime show?” “What is the birthplace of Usher?” “Which is the NFL team based in Dallas?”

A.3

Scoring

A.3.1

Entity and InvivoQuery task

Factuality is measured with an automated two-stage pipeline built on top of the OpenAI Responses API: 1. Claim extraction. An extraction model (default: GPT-4o) is prompted with a structured JSON schema to decompose the model’s response into atomic, checkable claims (factual, numerical, historical, definition, other). 2. Claim verification. A verification model (default: GPT-4o) is given the extracted claims and uses its built-in web search tool to assess each claim against live web sources. It returns a boolean is correct flag and one-to-three source URLs per claim. 16

Preprint. Under review.

The factuality score for a response is correct claims ∈ [0, 1]. total claims Both the extraction and verification calls use exponential-backoff retry (up to five attempts) to handle transient API errors. s=

Claim Extraction and Verification Prompts. To evaluate factuality, we adopt a two-stage pipeline consisting of claim extraction followed by claim verification. (1) Claim Extraction. Extraction Prompt: You are a claim extraction engine. Extract all distinct, checkable claims from the RESPONSE. • A claim is an assertion that could be true or false • Split compound sentences into atomic claims • Do NOT add new claims • Prefer recall over precision Return only valid JSON matching the schema. Schema: _EXTRACTION_SCHEMA = { "type": "object", "properties": { "entity": {"type": "string"}, "question": {"type": "string"}, "claims": { "type": "array", "items": { "type": "object", "properties": { "id": {"type": "string"}, "claim": {"type": "string"}, "type": { "type": "string", "enum": [ "factual", "numerical", "historical", "definition", "other"], }, "span": {"type": "string"}, }, "required": ["id", "claim", "type", "span"], "additionalProperties": False, }, }, }, "required": ["entity", "question", "claims"], "additionalProperties": False, }

(2) Claim Verification. Verification Prompt: You are a claim verification engine. Verify each claim using the web search tool when needed, prioritizing reliable sources.

Preprint. Under review.

• Preserve the original id and claim text exactly • Set is correct=true only if the claim is clearly correct • If uncertain, set is correct=false and explain briefly • Include 1–3 plain-text source URLs in the reason Return only valid JSON matching the schema. Schema: _VERIFICATION_SCHEMA = { "type": "object", "properties": { "entity": {"type": "string"}, "question": {"type": "string"}, "verifications": { "type": "array", "items": { "type": "object", "properties": { "id": {"type": "string"}, "claim": {"type": "string"}, "is_correct": {"type": "boolean"}, "reason": {"type": "string"}, }, "required": ["id", "claim", "is_correct", "reason"], "additionalProperties": False, }, }, }, "required": ["entity", "question", "verifications"], "additionalProperties": False, }

A.3.2

BFCL Scorer

An additional BFCL LLM-as-Judge scorer is provided for BFCL benchmark questions. We’re using this prompt to guide the model to make the judgment: <bos><start of turn>user You are an expert evaluator for question answering systems. You will receive: • A question • A ground truth answer • A model’s answer Your task is to determine whether the model’s answer correctly contains or conveys the ground truth answer as the answer to the question. Evaluation criteria: Score 1 only if the model answer explicitly, unambiguously, and correctly provides the ground truth answer (or a semantically equivalent answer) in response to the question. Score 0 if the model answer is incorrect, contradictory, ambiguous, speculative, hedged, or gives multiple possible answers without clearly identifying the correct one. Accept semantically equivalent answers, paraphrases, standard aliases, and unambiguous abbreviations when they clearly refer to the same answer. Ignore differences in capitalization, punctuation, articles, and minor formatting. If the question asks for a specific entity, number, date, or fact, the model answer must match that fact accurately enough to be considered correct in context. If the model answer includes extra information, that is acceptable only if the final answer remains clearly correct and not contradicted. Return ONLY valid JSON in exactly this format: ”score”: 0 or 1

Preprint. Under review.

0 = incorrect 1 = correct <end of turn> <start of turn>model

A.4

Agentic Tool-Use Framework

Our agentic framework follows a standard tool-augmented language modeling paradigm, where the model first determines whether external information is required and subsequently generates an answer conditioned on retrieved evidence Yao et al. (2023). The explicit tool-selection stage is supported by prior works that frame tool use as a decision problem involving when and how to invoke external APIs Schick et al. (2023a); Patil et al. (2024). Representing this decision as a structured JSON object aligns with established function-calling and API-invocation approaches, enabling reliable parsing and separating reasoning from execution. A.4.1

Stage 1: Tool decision.

In this section, we present the exact prompt used for the different experimental setups. The self-awareness setup. Given an entity, the model is prompted with a structured JSON schema and asked to decide whether to invoke the web search tool, for example, for the entity CDA01, the self tool selection prompt is: <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. You have access to the following tools: • web search: Search the web for current information about entities, facts, or topics. Use this when you need up-to-date or factual information you don’t have. Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” Decide if you need to use any tools. Respond with a JSON object: { ”needs tool”: truefalse, ”tool name”: ”tool name” or null, ”tool input”: ”the input you need to give to the tool”, ”reasoning”: ”why you need this tool or why you don’t need tools” } Rules: • Only use tools when you genuinely need external information • If you already know the answer, set needs tool to false • Only select ONE tool at a time <end of turn> <start of turn>model

We adopt a simple and general tool description for our web search tool to avoid over constraining the model’s behavior while still providing sufficient guidance for effective use. Rather than specifying detailed rules or heuristics, the description emphasizes the tool’s core capability: retrieving current, factual information, along with a high-level condition for use when knowledge is missing or uncertain. This design is broadly consistent with prior work such as ReAct Yao et al. (2023) and Toolformer Schick et al. (2023b), which suggest that relatively lightweight tool descriptions, when combined with appropriate prompting or training, can enable effective tool selection. Then we will get the response from the model as follows: <bos><start of turn>model { "needs tool": true, "tool name": "web search", "tool input": "CDA01", "reasoning": "I do not have pre-existing knowledge about ’CDA01’.

Preprint. Under review.

is likely a specific code, identifier, or acronym that requires a web search to understand its meaning and provide a relevant response." } <end of turn>

We can parse the model’s response and then call the tool based on the model’s decision. Then, if the model is not calling the tool based on the response, we have the prompt, which is the same in the no-tool setup, to ask the task queries. If the model is calling the tool based on the response, we use the tool input as the search query and fetch the search results from the web search MCP server to the prompt: <|im start|>system You are a helpful assistant. <|im end|>

<|im start|>user Based on the following search results, please answer the question.

Search Results:

1. Current Transformer — Submetering — Submeter — CDA01-K12 Current Transformer, Solid Core, 100A, 100mA, 0.72” Opening, 48” Leads, +/-0.3% Accuracy, Black, For Submetering. Source: https://leviton.com/products/cda01-k12

2. Leviton CDA01-K12 Current Transformer, 100:0.1A, Solid ... The Leviton CDA01-K12 solid core current transformer (CT) is cost effective and less susceptible to damage during installation. Source: https://www.powermeterstore.com/product/....

3. VerifEye™, Sub-Metering Current Transformer, 100A, Solid ... Designed for accurately capturing measurements of power consumption, CTs are easy to specify and install. Source: https://www.graybar.com/...

4. CDA01-K12.pdf Leviton solid core CTs are cost effective and less susceptible to damage during installation. Source: https://www.bulbspro.com/media/pdf/CDA01-K12.pdf

5. Leviton® CDA01-R12 Solid Core Sub-Metering Current ... 100:0.1 current ratio, 100 A primary, 0.1 A secondary, 0.3% accuracy. Source: https://www.steinerelectric.com/...

Question: In a paragraph, could you tell me what you know about CDA01? <|im end|>

<|im start|>assistant

The no-tool setup. We skip the tool decision stage for the no-search setup. The prompt is asking the task query directly: 20

Preprint. Under review.

<|im start|>system You are a helpful assistant. <|im end|>

<|im start|>user In a paragraph, could you tell me what you know about Damodaran? <|im end|>

<|im start|>assistant <|im end|>

The A LWAYS T OOL setup. In the A LWAYS T OOL setup, we force the model to call the tool by setting the needs tool as True and let the model generate the tool input by itself: <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. You have access to the following tools: • web search: Search the web for current information about entities, facts, or topics. Use this when you need up-to-date or factual information you don’t have. Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” Use this tool. Respond with a JSON object: { ”needs tool”: true, ”tool input”: ”the input you need to give to the tool” }

Then we will get the model response as: <bos><start of turn>model { "needs tool": "tool input": }

true, "CDA01",

We perform an exact-match check: if the model follows the instruction to set needs tool to True, we leave it as True; otherwise, we set it to True and use the original query as the tool input. Perceived need prompts To test the model’s perception of need, we perform the prompts to ask the model to decide whether it needs help from an external tool without giving the exact information about the tool. <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” You can call for help from a tool. Respond with ONLY a JSON object in this exact schema: { ”needs tool”: true, } or { ”needs tool”: false, } <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions.

Preprint. Under review.

Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” Do you need help to answer the question? Answer: <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” Do you know the answer to the question? Answer:

Cost-Aware Tool Description Variants To study how the framing of tool cost affects search behaviour, we vary the natural-language description of the web search tool injected into the tool-selection prompt. All variants share the same base description. We consider two variants of cost-aware tool descriptions, differing in whether the model is explicitly given the remaining tool-call budget. (1) Explicit Budget (No Implicit Calculation Required). <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. You have access to the following tools: • web search: Search the web for current information about entities, facts, or topics. Use this when you need up-to-date or factual information you don’t have. • Each tool call costs $X. You have a total budget of $10000. You have 500 questions in total, have already answered nfin , and have made ncall tool calls so far. You have y tool calls remaining. Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” Decide whether to use a tool. Respond with a JSON object: { ”needs tool”: truefalse, ”tool name”: ”tool name” or null, ”tool input”: ”input to the tool”, ”reasoning”: ”justification” } Rules: • Use tools only when external information is necessary • If the answer is known, set needs tool to false • Select at most one tool <end of turn> <start of turn>model

(2) Implicit Budget (Requires Internal Calculation). <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. You have access to the following tools: • web search: Search the web for current information about entities, facts, or topics. Use this when you need up-to-date or factual information you don’t have. • Each tool call costs $X. You have a total budget of $10000. You have 500 questions in total, have already answered nfin , and have made ncall tool calls so far. Given the user’s question: “In a paragraph, could you tell me what you know about CDA01?” Decide whether to use a tool. Respond with a JSON object: { ”needs tool”: truefalse, ”tool name”: ”tool name” or null, ”tool input”: ”input to the tool”, ”reasoning”: ”justification” }

Preprint. Under review.

Rules: • Use tools only when external information is necessary • If the answer is known, set needs tool to false • Select at most one tool <end of turn> <start of turn>model

Budget-Aware Setting. Budget-aware variants provide the model with: the total number of questions N, the number already answered nfin , and the cumulative number of tool calls ncall . These values are updated after each sample and injected dynamically at inference time (i.e., not pre-computed). We vary the per-call cost X from 0 to 10,000 to study whether models adapt their tool-use decisions under different budget constraints. k In the explicit-budget variant, the remaining number of tool calls j 10000− X ·ncall . is given by y = X Stage 2: Response generation. If a search is triggered, results (title, snippet, URL) retrieved from the web-search backend are injected into the final prompt: <bos><start of turn>user You are an intelligent agent that decides when to use tools to answer questions. Based on the following search results, please answer the question. ⟨results⟩ Question: {query} <end of turn> <start of turn>model

If no search is triggered, the query is passed directly. A system message appropriate for the model family is prepended in both cases. A.5

Web-Search Backend

Google Search via SerpAPI. Web search is provided via a FastMCP server 5 that exposes a single web search(query, count) tool. One provider is supported: Google Search (via SerpApi). The server returns up to k =5 results per query, each comprising a title, snippet, and URL. The MCP client communicates with the server over stdio using the Model Context Protocol, enabling the agent loop to call the tool asynchronously without blocking the main generation thread. Perplexity Search. Web search is additionally provided via Perplexity’s official MCP server6 , which exposes a perplexity search tool that returns raw search results without LLM synthesis. As with the SerpAPI provider, the server returns up to k =5 results per query, each comprising a title, snippet, and URL. The results are returned in a structured markdown format and parsed into a list of result dictionaries by the MCP client. A.6

The Controller Framework

To identify which transformer layer best encodes information predictive of a model’s need for web search, we implemented a systematic layer search procedure. Given a set of model responses and their corresponding factuality scores, each prompt is encoded using every intermediate hidden state of the target transformer, extracting the last-token representation at each layer, including the input embedding layer (layer 0) and all L transformer blocks, yielding L + 1 embedding matrices of shape ( N × H ). These embeddings are computed in batches using AutoModel with output hidden states=True via HuggingFace Transformers, and cached to disk as a compressed .npz file to avoid re-computation across runs. 5 https://github.com/prefecthq/fastmcp 6 https://docs.perplexity.ai/docs/getting-started/integrations/mcp-server

Preprint. Under review.

For each layer independently, we trained a multilayer perceptron (MLP) classifier using the MLPClassifier implementation from scikit-learn. The input features consist of fixed-length embedding vectors extracted from transformer layers. Prior to training, all features were standardized using StandardScaler to ensure zero mean and unit variance within each training fold. The MLP was trained with a maximum of 100 iterations, and early stopping was enabled, where training terminates if the validation performance does not improve for five consecutive iterations. The optimization follows the default settings of scikit-learn, which employs the Adam optimizer. A fixed random seed of 42 was used to ensure reproducibility. Hyperparameters were selected via grid search over a set of candidate architectures and learning rates. Specifically, we considered hidden layer configurations of {∅, (128), (256), (128, 64), (1024, 64)}, where ∅ denotes a model without hidden layers, and initial learning rates of {10−3 , 10−4 }. Model selection was performed using 5-fold stratified cross-validation on the training data, preserving the class distribution across folds. For evaluation, we employed 5-fold stratified cross-validation over the entire dataset and reported out-of-fold predictions. Performance was measured using accuracy, balanced accuracy, as well as the confusion matrix and classification report. All reported results are based on these cross-validated predictions. After all layers are evaluated, the layer achieving the highest out-of-fold accuracy, aggregated as the maximum across all classifiers, is selected as the optimal representation layer, and a full summary including per-layer scores and the best classifier configuration is saved to disk.

Human Alignment with the LLM-as-Judge Evaluation

We evaluated the reliability of the LLM-as-judge verdicts by having three human annotators independently validate each claim-level judgment on a shared sample of 100 responses. For each claim, annotators indicated whether they agreed (1) or disagreed (0) with the LLM’s verdict. Annotators 1 and 2 endorsed the LLM’s judgments on 99.7% and 99.9% of claims, respectively, exhibiting near-identical patterns and agreeing with each other on 99.8% of shared claims. Annotator3 was more critical, disagreeing with 7.9% of LLM verdicts, and showed near-chance inter-annotator agreement with both Annotator1 and Annotator2, reflecting the near-zero variance in the lenient annotators’ labels. Across all three annotators, 73.1% of claim judgments received unanimous endorsement. The most common point of divergence occurred when Annotator3 flagged a verdict as incorrect while Annotators 1 and 2 endorsed it (18.7% of claims), indicating that disagreement with the LLM was driven almost entirely by Annotator3. Overall, the high endorsement rates suggest the LLM-as-judge verdicts are largely consistent with human judgment, with Annotator3 providing the most discriminative validation signal.

Example of True Negative Utility

In this section, we show some examples where the factuality is decreasing in the entity task across different models: Entity: Sky Blue <|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI. Knowledge cutoff: 2024-06 Current date: 2026-03-09 Reasoning:

medium

# Valid channels: analysis, commentary, final. included for every message.<|end|>

Channel must be

Preprint. Under review.

know about Sky Blue?<|end|> <|start|>assistant<|message|>

The response from GPT-OSS-120B without search, which gets a factuality score of 1: ”Sky blue is a light, calming shade of blue that resembles the color of a clear daytime sky, typically ranging from a pale cyan to a soft, pastel hue. In color theory, it sits between turquoise and baby blue on the spectrum and is often associated with tranquility, openness, and optimism, making it a popular choice in design, fashion, and interior décor to evoke a sense of spaciousness and serenity. Historically, sky blue has been used in art to depict atmospheric perspective, helping to convey depth and distance, while in cultural contexts it can symbolize freedom, peace, and the infinite possibilities of the horizon. The shade also appears in various branding and corporate identities to convey reliability and calmness, and it is a frequent favorite in wedding palettes, baby nurseries, and wellness spaces for its soothing visual impact.”

The prompt with search: <|im start|>system You are a helpful assistant. <|im end|>

<|im start|>user Based on the following search results, please answer the question.

Search Results:

1. Coventry City F.C. The club plays in the EFL Championship and is nicknamed the Sky Blues after its sky blue colours. Source: https://en.wikipedia.org/wiki/Coventry City F.C.

2. Club History The Sky Blues played their first game at the Ricoh Arena in 2005, winning 3-0. Source: https://www.ccfc.co.uk/...

3. Sky Blue FC Profile Sky Blue FC was founded in New Jersey in 2007, with its inaugural season in 2009. Source: https://www.planetsport.com/...

4. Founded in 1883 as Singers F.C. Coventry City F.C., now known as "The Sky Blues", was founded in 1883. Source: https://www.facebook.com/...

Question: In a paragraph, could you tell me what you know about Sky Blue?<|im end|>

<|im start|>assistant <|im end|>

The response that gets the factuality score of 0.58.

Preprint. Under review.

20 0.2

0.4

0.6

Facuality

0.8

1.0

0.0

Qwen-3-30B-IT (No Tool) 80

Count

1.0

0.2

0.4

0.6

Facuality

0.8

60 40

1.0

0.0

120

0.2

0.4

0.6

Facuality

0.8

140

Count

100 80 40

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

160

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

0.4

0.6

Facuality

0.8

1.0

mean=0.799 median=0.875

140 100 80 60 20

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

Llama3.2-3B-IT (No Tool)

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

Llama3.2-3B-IT (With Tool)

140

mean=0.591 median=0.667

mean=0.792 median=0.846

120 100

0.2

Gemma3-27B-IT (With Tool)

50 30

0.0

1.0

120

Facuality

0.8

120

mean=0.808 median=0.879

160

100

0.6

1.0

Mistral3.1-24B-IT (With Tool)

mean=0.701 median=0.857

140

0.4

Mistral3.1-24B-IT (No Tool)

160

0.2

mean=0.599 median=0.750

100

20 0.0

0.0

Gemma3-27B-IT (No Tool)

Count

60 40

Count

Facuality

0.8

mean=0.647 median=0.818

100

0.6

Count

100

0.4

Qwen-3-30B-IT (With Tool)

mean=0.685 median=0.789

120

0.2

Count

0.0

Count

100

mean=0.643 median=0.800

120

100

Count

Qwen3-30B-A3B (With Tool)

mean=0.703 median=0.800

120

Qwen3-30B-A3B (No Tool)

140

mean=0.783 median=0.857

100

Count

GPT-OSS-120B (With Tool)

120

mean=0.612 median=0.750

Count

GPT-OSS-120B (No Tool)

80 60 40 20

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

Figure 9: Entity Task: factuality distribution across different models. s

”The term S̈ky Bluep̈rimarily refers to Coventry City Football Club, an English professional football team based in Coventry, which plays in the EFL Championship. The club, originally founded in 1883 as Singers F.C., adopted the nickname T̈he Sky Bluesd̈ue to their distinctive sky blue kits. They joined the Football League in 1898 and have played at the Ricoh Arena (formerly known as the Coventry Building Society Arena) since 2005, marking their first game there with a 3-0 victory over QPR. The nickname S̈ky Blueı̈s also associated with Sky Blue FC, a women’s soccer team based in New Jersey, USA, which was founded in 2007 and began play in the Women’s Professional Soccer league in 2009. While both teams share the S̈ky Bluen̈ame, they are distinct entities from different countries and leagues.”

Additional Entity Task Results

In this section, we show the additional results for the entity task. In Figure 9, we show the factuality score distribution across all the models and entities. Visualizing the distribution is important because aggregate metrics alone (e.g., mean or accuracy) can obscure underlying differences in model behavior. The distribution provides a more fine-grained view of how factuality scores are spread, revealing patterns such as skewness, variance, and the presence of extreme cases. In particular, this figure allows us to examine how factuality shifts when tool use is enabled versus disabled. Rather than only observing average improvements, the distribution highlights whether gains are consistent across samples or driven by a subset of cases. D.1

Normative Lens

We operationalize True Need and True Utility by comparing model performance under the N O T OOL and A LWAYS T OOL settings. True Need corresponds to instances where performance without tool use falls into the Low or Mid categories, indicating that external tool support is likely necessary. We further define True Positive Utility as cases where tool use leads to performance improvements (e.g., Low → Mid/High), while True Negative Utility captures cases where performance degrades with tool use. Instances where performance remains unchanged are categorized as neutral utility. As shown in Figure 10, a consistent pattern emerges across all models: tool use is most beneficial when it is truly needed, can be harmful when unnecessary, and is often redundant otherwise. 26

Preprint. Under review.

Negative

Mid

Neutral

Positive

Negative

(d) Gemma-3-27B-IT

Low Mid

Neutral

Positive

Negative

Low Mid

Neutral

True Need

2 33 13 5 159 105 High 1 69 113

Performance without tool

True Need

Positive

6 44 18 5 128 80 High 0 84 135

Performance without tool

9 79 24 1 144 81 High 1 63 98 Low

Neutral

(b) Qwen3-30B-A3B Performance with tool Low Mid High

True Need

(a) GPT-OSS-120B Performance with tool Low Mid High

Mid

Positive

Negative

2 59 19 2 181 113 High 4 60 60

Performance without tool

Positive

Low

Performance with tool Low Mid High

True Need

Neutral

2 26 9 4 190 94 High 2 76 97

Performance without tool

Mid

True Need

Performance without tool

7 67 20 2 162 90 High 1 51 100 Low

Performance with tool Low Mid High

Negative

(e) Mistral-3.1-24B-IT

Low Mid

Neutral

True Need

Performance with tool Low Mid High

Positive

Negative

(f) Llama-3.2-3B-IT

Figure 10: Entity task: No-Tool vs. Force-Tool performance. Rows group entities by the model’s factuality score without a tool (reflecting parametric knowledge), while columns group scores when tool use is forced. Each cell reports the count and the column percentage. Off-diagonal cells indicate performance shifts due to tool use: cells above the diagonal show cases where the tool has positive utility, while cells below the diagonal indicate cases where the tool has negative utility. The dashed bracket marks the region of True Need, where Low or Mid No-Tool scores suggest insufficient parametric knowledge and thus a likely need for an external tool.

Descriptive Lens Gemma3-27B-IT

Descriptive Lens Qwen-3-30B-IT

Descriptive Lens GPT-OSS-120B

Descriptive Lens Qwen3-30B-A3B

Descriptive Lens Mistral3.1-24B-IT

Descriptive Lens Llama3.2-3B-IT

True Positive Utility Perceived Need Perceived Utility

Figure 11: [Entity Task] Venn diagrams of three sets: True Positive Utility, Perceived Need, and Perceived Utility. The Oracle shows the ideal nested structure (Perceived Utility ⊆ Perceived Need ⊆ True Positive Utility); each Descriptive Lens panel shows the empirical overlap for one model. Deviations from nesting quantify the misalignment between model perception and ground-truth benefit, motivating Latent Estimators.

This observation highlights the importance of accurately predicting when to invoke external tools, as indiscriminate usage may introduce noise or errors rather than improving factuality. D.2

Descriptive Lens

As shown in Figure 11, there is a consistent misalignment between the perceived need and utility and the true positive utility across all models. This discrepancy indicates that models often fail to accurately identify when tool use is genuinely beneficial. As a result, none of the models achieve optimal tool-calling performance, since effective tool use critically depends on correctly aligning perceived need with actual utility. 27

Preprint. Under review.

Call

No Need

325 15 130 30

Tool Calling

328

Need

No Call

Call

Perceived

No Need

157 15

Tool Calling

Need

No Call

Call

Perceived

No Need

257 205

Tool Calling

Need

No Call

Call

Perceived

No Need

315 138

Tool Calling

152

Need

No Call

Call

347

Perceived

No Need

279 63

Tool Calling

Perceived

Need

No Call

Call

152

Tool Calling

No Need

No Call

Perceived

Need

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(a) Perceived-need prompt v2

328

No Need

Call

17 155

Need

126 214 81

Tool Calling

459

Perceived

No Need

No Call

Tool Calling

59 394

Tool Calling

154

Need

No Call

No Need

Call

346

Need

Perceived

Need

No Call

No Need

Perceived

No Need

Call

73 269

Perceived

No Call

152

Tool Calling

Call

Tool Calling

Need

No Call

Perceived

No Need

Tool Calling

Perceived

Need

(g) GPT-OSS- (h) Qwen3-30B- (i) Qwen3-30B- (j) Gemma-3- (k) Mistral-3.1- (l) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(b) Perceived-need prompt v3 Figure 12: Entity Task: perceived need is only partially aligned with tool use. The x-axis shows perceived utility (number of entities predicted to need or not need external information), and the y-axis shows actual tool-use decisions. Percentages indicate how often the model follows its own prediction (call vs. not call). Results are shown for two prompt variants (v1 and v2). Some responses are excluded due to parsing failures (i.e., missing explicit yes/no decisions), so the total count is less than 500. Figure 12 shows two key observations. First, the model’s self-perceived need for tool use is highly sensitive to the prompting format, where even small variations can lead to noticeably different outcomes. Second, across these variations, perceived need and utility (i.e., actual tool-calling decisions) are consistently related but not perfectly aligned. Overall, the main takeaway remains robust across different prompt formulations: while prompting influences the model’s perception of need, it does not resolve the fundamental misalignment between perceived need and actual utility. D.2.1

Affordability

In this section, we present detailed results for both normative and descriptive affordability. To evaluate whether models prioritize the most beneficial entities under constrained tool usage, we employ the normalized discounted cumulative gain (NDCG) rank correlation metric. Specifically, NDCG measures the extent to which the model’s ranking of entities aligns with the optimal ranking based on maximum utility gain, thereby capturing how effectively the model selects high-utility entities when tool calls are limited. NDCG Rank Correlation. To evaluate the quality of the model’s budget-aware tool-call selection, we measure how well the instances chosen under perceived affordability align with the ideal selection under true affordability. For a given cost level c, let Sb be the set of instances for which the model invokes the tool, and let K = ⌊ B/c⌋ be the budget-permitted call limit (with K = n when c = 0). Following the same capping logic as the affordability evaluation, we construct SbK by retaining only the first K instances in Sb (in the order they appear), discarding any calls that exceed the budget. Relevance labels. The ground-truth relevance of invoking the tool on instance xi is the marginal factuality gain ∆⋆ ( xi ) = s WT ( xi ) − s NT ( xi ). Since NDCG requires non-negative relevance values, we replace ∆⋆ ( xi ) with its ordinal rank ri among all n instances (average rank for ties, ascending), yielding the relevance vector r = (r1 , . . . , rn ). b K ( xi ) = 1 { xi ∈ NDCG@K. We define the budget-capped perceived affordability indicator as A b b SK } ∈ {0, 1}. Instances are ranked by AK descending (called instances first), and NDCG@K is 28

Preprint. Under review.

Maximum Tool Calls 500

Actual Tool Calls

400 300 200 100 $0

,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

0,0

Tool Cost ($)

Figure 13: [Entity Task] Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget. computed as: NDCG@K =

bK DCG@K r, A IDCG@K (r)

r π (i )

i =1

log2 (i + 1)

DCG@K = ∑

(1)

b K , and IDCG@K is the DCG of the ideal ranking (instances where π is the permutation induced by A sorted by ri descending). A score of 1.0 means SbK = SK⋆ , i.e., the model selected exactly the K instances with the highest marginal gain ∆⋆ . Curve. Each cost level c yields one point K/n × 100%, NDCG@K on the NDCG curve, where the x-axis K/n × 100% is the budget-determined coverage — identical to the x-axis of the affordability plot — enabling direct comparison between the two. This formulation decouples how often the model invokes the tool (coverage, x-axis) from how well it prioritises the right instances within that budget (NDCG@K, y-axis). Figure 13 shows the model’s tool-calling frequency under different budgets with explicit budget-aware prompting. Figure 14 reports the corresponding NDCG-based rank correlation. We then evaluate implicit budget-aware prompting in Figures 15 and 16. We observe that implicit prompting fails to effectively constrain tool use. D.3

Controller Framework

Here, we present the confusion matrices of LNE, LUEx , and LUEx,d as additional results. For the LNE in Figure 17. The results show that when the predictor decides to call the tool, the model indeed requires external assistance in the majority of cases (¿70%) across all models. For LUEx and LUEx,d (Figures 18, 19, and 20), the results show that when the predictor decides to call the tool, it yields positive utility in the majority of cases (¿50%) across all models. Conversely, when the predictor decides not to call the tool, the utility is negative or neutral in most cases (¿60%). Tool-call decisions are guided by the latent need estimator’s predicted probabilities under a fixed budget constraint. In particular, we follow the predictor’s scores to rank instances by their likelihood of requiring tool use, and enable tool calling for the top-k instances within a given budget. This strategy yields improved performance under the same budget compared to alternative allocation schemes. Figure 21 illustrates the effectiveness of this approach across different budget levels.

InvivoQuery Task

In this section, we show the additional results for the entity task. 29

Preprint. Under review.

Gemma3-27B GPT-OSS-120B Qwen3-30B

1.0

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.8

NDCG@k

0.6 0.4 0.2 0.0 0%

20%

40%

60%

80%

% Queries with Tool Calls

100%

Figure 14: [Entity Task] The NDCG rank correlation under different budgets across different models. The correlation is low, which reflects that the models are not choosing the best utility gain tool calling. Cost prompt v1.

Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT

Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description

Maximum Tool Calls 500

Utility Gain over No-Tool

0.25

Actual Tool Calls

400

0.20

300

0.15

200

0.10

100

0.05 60%

80%

% Queries with Tool Calls

100%

(a) Utility gain with hard stop after exceeding the budget.

40%

20%

$1 ,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

0 0%

$1 0,0

0.00

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 15: Cost-aware tool use on the Entity Task. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget. In Figure 22, we show the factuality score distribution across all the models and entities for the InvivoQuery Task. Visualizing the distribution is important because aggregate metrics alone (e.g., mean or accuracy) can obscure underlying differences in model behavior. The distribution provides a more fine-grained view of how factuality scores are spread, revealing patterns such as skewness, variance, and the presence of extreme cases. In particular, this figure allows us to examine how factuality shifts when tool use is enabled versus disabled. Rather than only observing average improvements, the distribution highlights whether gains are consistent across samples or driven by a subset of cases. E.1

Normative Lens

As shown in Figure 23, a consistent pattern emerges across all models: tool use is most beneficial when it is truly needed, can be harmful when unnecessary, and is often redundant otherwise. This observation highlights the importance of accurately predicting when to invoke external tools, as indiscriminate usage may introduce noise or errors rather than improving factuality. 30

Preprint. Under review.

Gemma3-27B GPT-OSS-120B Qwen3-30B

1.0

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.8

NDCG@k

0.6 0.4 0.2 0.0 0%

20%

40%

60%

80%

100%

% Queries with Tool Calls

Figure 16: [Entity Task] The NDCG rank correlation under different budgets across different models. The correlation is low, which reflects that the models are not choosing the best utility gain tool calling. Cost prompt v2. Predicted

141 59 (39%) (44%)

No Need

352 29 (79%) (55%) 95 24 (21%) (45%)

Actual

225 75 (61%) (56%)

Need Need

No Need

Need

128 24 (29%) (44%)

Need

Actual

Need

318 30 (71%) (56%)

Actual

Predicted

No Need

Need

71 105 (21%) (67%)

Need

No Need

Predicted

No Need

271 52 (79%) (33%)

Actual

Need

67 97 (19%) (64%)

Need

No Need

Predicted

No Need

282 54 (81%) (36%)

Actual

No Need

62 82 (17%) (59%)

Need

No Need

Need

299 57 (83%) (41%)

Actual

Predicted

No Need

Predicted

Need

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 17: [Entity Task.] Confusion matrix for the LNE estimator

E.2

Descriptive Lens

As shown in Figure 24, there is a consistent misalignment between the perceived need and utility and the true positive utility across all models. This discrepancy indicates that models often fail to accurately identify when tool use is genuinely beneficial. As a result, none of the models achieve optimal tool-calling performance, since effective tool use critically depends on correctly aligning perceived need with actual utility. Figure 25 shows two key observations. First, the model’s self-perceived need for tool use is highly sensitive to the prompting format, where even small variations can lead to noticeably different outcomes. Second, across these variations, perceived need and utility (i.e., actual tool-calling decisions) are consistently related but not perfectly aligned. E.3

Controller Framework

Here, we present the confusion matrices of LNE, LUEx , and LUEx,d as additional results for the InvivoQuery Task. For the LNE in Figure 31 and Figure 32. The results show that when the predictor decides to call the tool, the model indeed requires external assistance in the majority of cases (¿75%) across all models. For LUEx and LUEx,d (Figures 33, 34, and 35), the results show that when the predictor decides to call the tool, it yields positive utility in the majority of cases (¿50%) across all models. Tool-call decisions are guided by the latent need estimator’s predicted probabilities under a fixed budget constraint. In particular, we follow the predictor’s scores to rank instances by their likelihood of requiring tool use, and enable tool calling for the top-k instances within a given budget. This strategy yields improved performance under the same budget compared to alternative allocation schemes. Figure 36 illustrates the effectiveness of this approach across different budget levels. 31

Accuracy

Preprint. Under review.

1.00 0.80 0.60 0.40 0.20 0.00

LUEx, d

Perceived Utility

LUEx

-3B 27B 20B 30B B-IT 24B ma3- SS-1 Qwen3-wen3-30istral3.1-Llama3.2 Gem GPT-O Q M Figure 18: The LUEs can predict the True Utility more accurately across most models, especially for small and weaker models. We show the confusion matrixs of the two predictors in Figure 19 and 20. The same pattern is observed also in the InvivoQuery Task in Figure 33 and the BFCL Task in FigurePredicted 48. Predicted Predicted Predicted Predicted Predicted Actual

128 121 (47%) (53%)

Help

No Help

145 106 (53%) (47%)

No Help

Actual

78 219 (57%) (60%)

Help

No Help

60 143 (43%) (40%)

No Help

Actual

90 195 (46%) (64%)

Help

No Help

104 111 (54%) (36%)

No Help

Actual

90 162 (39%) (60%)

Help

No Help

139 109 (61%) (40%)

No Help

Actual

98 202 (47%) (69%)

Help

No Help

Actual

Help

110 90 (53%) (31%)

No Help

52 103 (50%) (26%) 51 294 (50%) (74%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 19: [Entity Task.] Confusion matrix for the LUEx estimator

Results for BFCL task

In this section, we show the additional results for the entity task. In Figure 37, we show the factuality score distribution across all the models and entities for the BFCL Task. Visualizing the distribution is important because aggregate metrics alone (e.g., mean or accuracy) can obscure underlying differences in model behavior. The distribution provides a more fine-grained view of how factuality scores are spread, revealing patterns such as skewness, variance, and the presence of extreme cases. In particular, this figure allows us to examine how factuality shifts when tool use is enabled versus disabled. Rather than only observing average improvements, the distribution highlights whether gains are consistent across samples or driven by a subset of cases. F.1

Normative Lens

As shown in Figure 38, a consistent pattern emerges across all models: tool use is most beneficial when it is truly needed, can be harmful when unnecessary, and is often redundant otherwise. This observation highlights the importance of accurately predicting when to invoke external tools, as indiscriminate usage may introduce noise or errors rather than improving factuality. F.2

Descriptive Lens

As shown in Figure 39, there is a consistent misalignment between the perceived need and utility and the true positive utility across all models. This discrepancy indicates that models often fail to accurately identify when tool use is genuinely beneficial. As a result, none of the models achieve optimal tool-calling performance, since effective tool use critically depends on correctly aligning perceived need with actual utility. Figure 40 shows two key observations. First, the model’s self-perceived need for tool use is highly sensitive to the prompting format, where even small variations can lead to noticeably different outcomes. Second, across these variations, perceived need and utility (i.e., actual tool-calling decisions) are consistently related but not perfectly aligned. F.3

Controller Framework

Here, we present the confusion matrices of LNE, LUEx , and LUEx,d as additional results for the BFCL Task. For the LNE in Figure 46 and Figure 47. The results show that when the predictor 32

Preprint. Under review.

Help Help

145 106 (53%) (47%)

Actual

78 219 (57%) (60%)

Predicted

No Help

128 121 (47%) (53%)

No Help

92 193 (49%) (61%)

60 143 (43%) (40%)

Help Help

Help

94 121 (51%) (39%)

Predicted

No Help

Actual

95 157 (38%) (63%)

Help

Actual

Help

156 92 (62%) (37%)

Predicted

No Help

Help

Actual

Help

Predicted

No Help

64 236 (42%) (68%)

Actual

90 110 (58%) (32%)

Help

No Help

Help

Actual

No Help

Predicted

No Help

Predicted

Help

No Help

52 103 (50%) (26%) 51 294 (50%) (74%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 20: [Entity Task.] Confusion matrix for the LUEx,d estimator

Gemma3-27B GPT-OSS-120B Qwen3-30B

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LNE

Ideal curve Cost Description LUEx

0.20

0.15

0.10

0.05 0.00

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.25

0.20

Utility Gain over No-Tool

0.25

Gemma3-27B GPT-OSS-120B Qwen3-30B

0.05

20%

40%

60%

80%

100%

% Queries with Tool Calls

0.00

20%

40%

60%

80%

% Queries with Tool Calls

(a) LNE

100%

(b) LUEx Gemma3-27B GPT-OSS-120B Qwen3-30B

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LUEx, d

Utility Gain over No-Tool

0.25 0.20 0.15 0.10 0.05 0.00

20%

40%

60%

80%

% Queries with Tool Calls

100%

Figure 21: [Entity Task] Tool-call decisions are guided by the latent need estimator’s predicted probabilities under a fixed budget constraint.

decides to call the tool, the model indeed requires external assistance in the majority of cases (¿75%) across all models. For LUEx and LUEx,d (Figures 48, 49, and 50), the results show that when the predictor decides to call the tool, it yields positive utility in the majority of cases (¿60%) across all models. Conversely, when the predictor decides not to call the tool, the utility is negative or neutral in most cases (¿50%). Tool-call decisions are guided by the latent need estimator’s predicted probabilities under a fixed budget constraint. In particular, we follow the predictor’s scores to rank instances by their likelihood of requiring tool use, and enable tool calling for the top-k instances within a given budget. This strategy yields improved performance under the same budget compared to alternative allocation schemes. Figure 51 illustrates the effectiveness of this approach across different budget levels. 33

Preprint. Under review.

GPT-OSS-120B (With Tool) 120

40 20 0.0

0.2

0.4

0.6

Facuality

0.8

Qwen-3-30B-IT (No Tool)

20 0.0

0.2

0.4

0.6

0.8

Facuality

1.0

80 60 20

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

mean=0.609 median=0.667

140

mean=0.547 median=0.600

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

0.0

Mistral3.1-24B-IT (No Tool) 100

0.8

Facuality

1.0

60 40

0.4

0.6

Facuality

0.8

1.0

0.4

0.6

Facuality

0.8

1.0

0.2

0.4

0.6

0.8

Facuality

1.0

0.2

0.4

0.6

Facuality

0.8

1.0

Llama3.2-3B-IT (With Tool) mean=0.541 median=0.600

80 60 40

20 0.0

0.0

100

mean=0.517 median=0.551

20 0.2

0.2

20 0.0

Llama3.2-3B-IT (No Tool)

Count

0.6

mean=0.597 median=0.667

120

0.4

1.0

Mistral3.1-24B-IT (With Tool)

mean=0.559 median=0.644

100

0.2

0.8

Count

0.6

Facuality

100

40 20

0.4

mean=0.625 median=0.714

120

0.2

Gemma3-27B-IT (With Tool)

40 20

0.0

Gemma3-27B-IT (No Tool)

Count

100

Qwen-3-30B-IT (With Tool)

120

60 40

mean=0.643 median=0.727

140

100

Count

120

mean=0.545 median=0.571

1.0

Count

100

Qwen3-30B-A3B (With Tool)

160

mean=0.629 median=0.727

120

100

Count

Qwen3-30B-A3B (No Tool)

mean=0.465 median=0.500

140

Count

mean=0.525 median=0.556

Count

GPT-OSS-120B (No Tool) 100

20 0.0

0.2

0.4

0.6

Facuality

0.8

1.0

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

Figure 22: [InvivoQuery Task] factuality distribution across different models.

Neutral

Positive

44 51 18 Mid 30 162 70 High 11 60 54

Performance without tool

31 35 21 Mid 23 210 72 High 14 35 59 Low

Negative

(d) Gemma-3-27B-IT

Low

Neutral

Positive

Mid

Neutral

True Need

Performance without tool

True Need

Positive

(b) Qwen3-30B-A3B Performance with tool Low Mid High

True Need

(a) GPT-OSS-120B Performance with tool Low Mid High

Neutral

29 44 16 22 214 60 High 17 39 59 Low

Positive

Negative

46 39 16 28 219 51 High 17 37 47

Negative

(e) Mistral-3.1-24B-IT

Low Mid

Neutral

True Need

Negative

Performance with tool Low Mid High

Performance without tool

Positive

20 28 19 Mid 24 178 74 High 15 67 75 Low

True Need

Neutral

Performance with tool Low Mid High Performance without tool

Performance without tool

53 40 14 Mid 62 173 34 High 40 39 45 Low

True Need

Performance with tool Low Mid High

Positive

Negative

(f) Llama-3.2-3B-IT

Figure 23: [InvivoQuery Task] N O T OOL vs. A LWAYS T OOL performance. Rows group entities by the model’s factuality score without a tool (reflecting parametric knowledge), while columns group scores when tool use is forced. Each cell reports the count and the column percentage. Off-diagonal cells indicate performance shifts due to tool use: cells above the diagonal show cases where the tool has positive utility, while cells below the diagonal indicate cases where the tool has negative utility. The dashed bracket marks the region of True Need, where Low or Mid No-Tool scores suggest insufficient parametric knowledge and thus a likely need for an external tool.

Results for the Entity Task with Perplexity Web Search

Table 4 shows the general performance of GPT-OSS-120B with the Perplexity Search 7 , Brave Search 8 , and Tavily Search 9 as the backbend search engine. 7 https://docs.perplexity.ai/docs/getting-started/integrations/mcp-server 8 https://github.com/brave/brave-search-mcp-server 9 https://docs.tavily.com/documentation/mcp

Preprint. Under review.

Descriptive Lens Gemma3-27B-IT

Descriptive Lens GPT-OSS-120B

Descriptive Lens Qwen-3-30B-IT

Descriptive Lens Qwen3-30B-A3B

Descriptive Lens Mistral3.1-24B-IT

True Positive Utility Perceived Need Perceived Utility

Descriptive Lens Llama3.2-3B-IT

Figure 24: [InvivoQuery Task] Venn diagrams of three sets: True Positive Utility, Perceived Need, and Perceived Utility. The Oracle shows the ideal nested structure (Perceived Utility ⊆ Perceived Need ⊆ True Positive Utility); each Descriptive Lens panel shows the empirical overlap for one model. Deviations from nesting quantify the misalignment between model perception and ground-truth benefit, motivating Latent Estimators.

No Need

Call

159 35

Need

119 198 44 139

Tool Calling

Perceived

No Need

Call

Need

Tool Calling

Call

213 15

Tool Calling

Perceived

No Need

No Call

Call

115 52

Need

No Call

Perceived

No Need

268 65

Tool Calling

Call

125 184

Need

No Call

Perceived

No Need

144 47

Tool Calling

No Call

49 260

Need

No Call

Call

157 34

Tool Calling

Perceived

No Need

No Call

Perceived

Need

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

139 28

265 21

No Need

Call

206

292 25 160 23

Tool Calling

Perceived

Need

No Call

168

Call

155 178

Tool Calling

No Call

303

No Need

No Call

294 15

Need

Call

No Need

Tool Calling

Call

185

Perceived

Need

No Call

Perceived

No Need

Call

190

Perceived

Need

Tool Calling

No Need

(a) Perceived-need prompt v1

No Call

Need

Tool Calling

No Need

Tool Calling

Need Call

Perceived

No Call

Perceived

(g) GPT-OSS- (h) Qwen3-30B- (i) Qwen3-30B- (j) Gemma-3- (k) Mistral-3.1- (l) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(b) Perceived-need prompt v2 Call

75 258

Call

17 197

308

303

24 143

No Call

278

No Need

Call

183

Need

38 279 16 167

Tool Calling

Perceived

No Need

No Call

190

Tool Calling

Need

Tool Calling

No Need

Tool Calling

Need

Tool Calling

No Need

No Call

Perceived

Need

Call

Perceived

No Need

No Call

Perceived

Need

Call

Perceived

No Need

No Call

Perceived

Need

(m) GPT-OSS- (n) Qwen3-30B- (o) Qwen3-30B- (p) Gemma-3- (q) Mistral-3.1- (r) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(c) Perceived-need prompt v3 Figure 25: [InvivoQuery Task] perceived need is only partially aligned with tool use. The x-axis shows perceived utility (number of entities predicted to need or not need external information), and the y-axis shows actual tool-use decisions. Percentages indicate how often the model follows its own prediction (call vs. not call). Results are shown for three prompt variants (v2 and v3). Some responses are excluded due to parsing failures (i.e., missing explicit yes/no decisions), so the total count is less than 500.

Preprint. Under review.

Perceived

True

No Need Need

Need

(e) Mistral-3.1-24B-IT

True Utility

148 231

(h) Qwen3-30B-A3B

Tool Calling

Call

No Call

Call

No Call

Tool Calling

(f) Llama-3.2-3B-IT

Other Positive

True Utility

Other Positive

250 130

No Need

136 263

(i) Qwen3-30B-A3B-IT

Perceived

Need

Other Positive

No Call

Tool Calling

(d) Gemma-3-27B-IT

No Need

161 83

Call

142 270

No Need

232 28

No Call

207 165

(j) Gemma-3-27B-IT

Tool Calling

Call

No Call

Call

No Call

162 199

(k) Mistral-3.1-24B-IT

True Utility

True

No Need Need

Need

Tool Calling

Call

(g) GPT-OSS-120B

Perceived

No Need

314 71

(b) Qwen3-30B-A3B

True

No Need Need

Need

True Utility

Perceived

195 148

Other Positive

(a) GPT-OSS-120B

No Need

True Utility

Tool Calling

Other Positive

True

170 206

Need No Need Need

No Need

No Need Need

True

No Need Need

Need

Utility

Perceived

Other Positive

Need

Perceived

246 148

(l) Llama-3.2-3B-IT

Figure 26: The perceived need and utility are not aligned with the true need and utility. InvivoQuery Task; Left: perceived need matrices. Right: true vs. perceived utility across models. Gemma3-27B Mistral3.1-24B-IT GPT-OSS-120B Llama3.2-3B Qwen3-30B Cost Description Maximum Tool Calls Qwen-3-30B-IT No Cost Description

Utility Gain over No-Tool

500

Actual Tool Calls

400

0.15

300

0.10

200

0.05

100

0.00

(a) Utility gain with hard stop after exceeding the budget.

% Queries with Tool Calls

100%

80%

60%

0,0

40%

20%

,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

0 0%

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 27: Cost-aware tool use on the Invivo Task with implicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget. Figure 52 shows that, even when using Perplexity search as the backend, not all tool calls yield positive utility. Only when the model genuinely requires external assistance do most tool calls result in a benefit. Figure 53 shows that after changing the web search backend, the model’s self-perception of need and utility is still partly aligned. 36

Preprint. Under review.

Gemma3-27B GPT-OSS-120B Qwen3-30B

1.0

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.8

NDCG@k

0.6 0.4 0.2 0.0 0%

20%

40%

60%

80%

% Queries with Tool Calls

100%

Figure 28: [InvivoQuery Task] The NDCG rank correlation under different budgets across different models. The correlation is low, which reflects that the models are not choosing the best utility gain tool calling. Cost prompt v1.

Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description

Maximum Tool Calls 500

Utility Gain over No-Tool

Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT

Actual Tool Calls

400

0.15

300

0.10

200

0.05

100

0.00 80%

% Queries with Tool Calls

100%

(a) Utility gain with hard stop after exceeding the budget.

60%

40%

$1 0,0

20%

$1 ,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

0 0%

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 29: Cost-aware tool use on the Invivo Task with explicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget. Model GPT-OSS-120B (Perplexity) GPT-OSS-120B (Brave) GPT-OSS-120B (Tavily) GPT-OSS-120B (Google via SerpAPI)

N O T OOL

A LWAYS T OOL

S ELF - DECISION

O PTIMAL

Score

Calls

Score

Calls

Score

Calls

Score

Calls

0.61 0.61 0.61 0.61

0 0 0 0

0.78 0.78 0.75 0.78

500 500 500 500

0.72 0.73 0.73 0.73

149 150 150 152

0.82 0.82 0.81 0.83

307 305 269 300

Table 4: Performance on the entity task under different tool-use strategies. N O T OOL: no access to tools. A LWAYS T OOL: tool is always invoked. S ELF - DECISION: model autonomously decides. O PTIMAL: oracle policy selecting the best decision per instance.

However, in Figure 54, we still observed that the perceived need and utility are not aligned with the true positive utility, which leads to non-optimal results with the Perplexity web search backend. 37

Preprint. Under review.

1.0

Gemma3-27B GPT-OSS-120B Qwen3-30B

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.8

NDCG@k

0.6 0.4 0.2 0.0 0%

20%

40%

60%

80%

100%

% Queries with Tool Calls

Accuracy

Figure 30: [InvivoQuery Task] The NDCG rank correlation under different budgets across different models. The correlation is low, which reflects that the models are not choosing the best utility gain tool calling. Cost prompt v2.

1.00 0.80 0.60 0.40 0.20 0.00

Perceived Need

LNE

B B B B B IT a3-T2-7OSS-1Q20wen3-3e0n3-30Btr-al3.1-2la4ma3.2-3 m m L Ge GP Qw Mis Figure 31: InvivoQuery task: The LNE can predict the True Need more accurately across most models, especially for small and weaker models.

Preprint. Under review.

Predicted

No Need

Need

115 9 (24%) (50%)

Need

380 22 (82%) (61%) 84 14 (18%) (39%)

Actual

Need

No Need

363 9 (76%) (50%)

Actual

100 3 (21%) (33%)

Need

No Need

Need

96 10 (21%) (31%)

Predicted

No Need

387 6 (79%) (67%)

Actual

372 22 (79%) (69%)

Need

No Need

Predicted

No Need

Need

118 34 (29%) (37%)

Need

Actual

Need

289 57 (71%) (63%)

Actual

Predicted

No Need

Need

96 26 (22%) (45%)

Actual

346 32 (78%) (55%)

No Need

Need

No Need

Predicted

No Need

Predicted

Need

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Accuracy

Figure 32: [InvivoQuery Task.] Confusion matrix for the LNE estimator 1.00 0.80 0.60 0.40 0.20 0.00

Perceived Utility

LUEx, d

LUEx

-3B 27B 20B 30B B-IT 24B ma3-PT-OSS-1 Qwen3-wen3-30istral3.1-Llama3.2 m e G Q G M Figure 33: The LUE can predict the True Utility more accurately across most models, especially for small and weaker models. In the Figure 55, we then show the breakdown of the misalignment through the normative lens and the descriptive lens.

Preprint. Under review.

Predicted

169 55 (43%) (51%)

Help Help

No Help

Actual

124 129 (52%) (50%)

Help

223 53 (57%) (49%)

No Help

Help

94 162 (38%) (65%)

Predicted

No Help

116 131 (48%) (50%)

Help

Actual

Help

Actual

147 57 (38%) (49%)

Predicted

No Help

Actual

Help

156 88 (62%) (35%)

No Help

Predicted

No Help

236 60 (62%) (51%)

No Help

Help

Actual

116 65 (30%) (57%)

Help

No Help

Help

Actual

No Help

Predicted

No Help

Predicted

Help

269 50 (70%) (43%)

No Help

157 112 (61%) (46%) 100 131 (39%) (54%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 34: [InvivoQuery Task.] Confusion matrix for the LUEx estimator

177 47 (43%) (51%)

Help

230 46 (57%) (49%)

Help

Actual

90 163 (46%) (54%)

Predicted

No Help

121 135 (47%) (56%)

106 141 (54%) (46%)

Help Help

Help

139 105 (53%) (44%)

Predicted

No Help

Actual

149 55 (41%) (40%)

Help

Actual

Help

214 82 (59%) (60%)

Predicted

No Help

Help

Actual

Help

Predicted

No Help

102 79 (28%) (59%)

Actual

263 56 (72%) (41%)

Help

No Help

Help

Actual

No Help

Predicted

No Help

Predicted

Help

No Help

132 137 (61%) (49%) 86 145 (39%) (51%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 35: [InvivoQuery Task.] Confusion matrix for the LUEx,d estimator

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LNE

0.15

Gemma3-27B GPT-OSS-120B Qwen3-30B

Utility Gain over No-Tool

Gemma3-27B GPT-OSS-120B Qwen3-30B

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LUEx

0.15

0.10

0.05

0.00

20%

40%

60%

80%

100%

% Queries with Tool Calls

20%

(a) LNE

60%

80%

100%

(b) LUEx Gemma3-27B GPT-OSS-120B Qwen3-30B

Utility Gain over No-Tool

40%

% Queries with Tool Calls

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LUEx, d

0.15 0.10 0.05 0.00 0%

20%

40%

60%

80%

% Queries with Tool Calls

100%

Figure 36: [InvivoQuery Task] Tool-call decisions are guided by the latent need estimator’s predicted probabilities under a fixed budget constraint.

Preprint. Under review.

GPT-OSS-120B (No Tool)

GPT-OSS-120B (With Tool)

Qwen-3-30B-IT (No Tool)

mean=0.752 median=1.000

200

100 75 50

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

0.8

Facuality

1.0

150 100 50

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

0.0

Mistral3.1-24B-IT (No Tool) 175

0.4

0.6

Facuality

0.8

1.0

mean=0.707 median=1.000

200

150

100

0.2

Mistral3.1-24B-IT (With Tool)

mean=0.325 median=0.000

200

Count

0.6

150

100

150

125 100

100

mean=0.452 median=0.000

25 0.0

0.2

0.4

0.6

Facuality

0.8

25 0

1.0

0.0

Llama3.2-3B-IT (No Tool)

0.2

0.4

0.6

0.8

Facuality

1.0

0.0

0.2

0.4

200

0.8

1.0

0.0

0.2

0.4

0.6

Facuality

0.8

1.0

mean=0.643 median=1.000

175

200

0.6

Facuality

Llama3.2-3B-IT (With Tool)

250

150

Count

0.4

mean=0.707 median=1.000

200

125

100

0.0

0.2

0.4

0.6

Facuality

0.8

125 100 75 50

mean=0.185 median=0.000

50 0

0.2

Gemma3-27B-IT (With Tool)

150

0.0

Gemma3-27B-IT (No Tool)

175

100 50

25 0

100

mean=0.742 median=1.000

200

150

Count

125

Count

150

Qwen-3-30B-IT (With Tool)

mean=0.309 median=0.000

200

Count

175

Count

mean=0.354 median=0.000

200

25 0

1.0

0.0

0.2

0.4

0.6

0.8

Facuality

1.0

Figure 37: [BFCL Task] factuality distribution across different models.

Positive

Negative

Neutral

77 101

Correct

23 113

Neutral

Positive

Performance without tool

Incorrect

Positive

Negative

(b) Qwen3-30B-A3B Performance with tool Incorrect Correct

True Need

Performance without tool

(a) GPT-OSS-120B Performance with tool Incorrect Correct

Negative

(d) Gemma-3-27B-IT

Incorrect

86 135

Correct

Neutral

85 Positive

Negative

80 Positive

Correct

True Need

Performance without tool

Correct

75 144

Negative

(e) Mistral-3.1-24B-IT

Correct

Neutral

102 158 9

True Need

Neutral

110

Incorrect

Performance without tool

108 144

True Need

Correct

Incorrect

Performance with tool Incorrect Correct

True Need

70 127

Performance without tool

Incorrect

Performance with tool Incorrect Correct

True Need

Performance without tool

Performance with tool Incorrect Correct

45 Positive

Negative

(f) Llama-3.2-3B-IT

Figure 38: [BFCL task]: No-Tool vs. Force-Tool performance. Rows group entities by the model’s factuality score without a tool (reflecting parametric knowledge), while columns group scores when tool use is forced. Each cell reports the count and the column percentage. Off-diagonal cells indicate performance shifts due to tool use: cells above the diagonal show cases where the tool has positive utility, while cells below the diagonal indicate cases where the tool has negative utility. The dashed bracket marks the region of True Need, where Low or Mid No-Tool scores suggest insufficient parametric knowledge and thus a likely need for an external tool.

Preprint. Under review.

Descriptive Lens Gemma3-27B-IT

Descriptive Lens GPT-OSS-120B

Descriptive Lens Qwen-3-30B-IT

Descriptive Lens Qwen3-30B-A3B

Descriptive Lens Mistral3.1-24B-IT

True Positive Utility Perceived Need Perceived Utility

Descriptive Lens Llama3.2-3B-IT

Figure 39: [BFCL Task] Venn diagrams of three sets: True Positive Utility, Perceived Need, and Perceived Utility. The Oracle shows the ideal nested structure (Perceived Utility ⊆ Perceived Need ⊆ True Positive Utility); each Descriptive Lens panel shows the empirical overlap for one model. Deviations from nesting quantify the misalignment between model perception and ground-truth benefit, motivating Latent Estimators.

No Call

Call

No Need

51 247 0

Tool Calling

94 160

Need

No Call

Call

222 90

Perceived

No Need

Call

302

Need

Tool Calling

Perceived

No Need

No Call

122 121

Tool Calling

Need

Tool Calling

No Need

Call

Perceived

Need

No Call

Perceived

No Need

Call

29 101

Need

Tool Calling

Call No Call

No Need

111 73

Tool Calling

Perceived

No Call

Perceived

Need

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(a) Perceived-need prompt v1

Need

No Need

184

243

303

Call

312

Call

254

Call

298

130

No Call

Tool Calling

No Need

Tool Calling

Need

Tool Calling

No Need

Tool Calling

Need

Tool Calling

No Need

Tool Calling

Need Call

Perceived

No Need

No Call

Perceived

Need Call

Perceived

No Need

No Call

Perceived

Need Call

Perceived

No Call

Perceived

(g) GPT-OSS- (h) Qwen3-30B- (i) Qwen3-30B- (j) Gemma-3- (k) Mistral-3.1- (l) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(b) Perceived-need prompt v2

184

243

303

Call

312

Call

254

Call

298

130

No Call

Tool Calling

No Need

Tool Calling

Need

Tool Calling

No Need

Tool Calling

Need

Tool Calling

No Need

Tool Calling

Need

Call

Perceived

No Need

No Call

Perceived

Need

Call

Perceived

No Need

No Call

Perceived

Need

Call

Perceived

No Need

No Call

Perceived

Need

(m) GPT-OSS- (n) Qwen3-30B- (o) Qwen3-30B- (p) Gemma-3- (q) Mistral-3.1- (r) Llama-3.2120B A3B A3B-IT 27B-IT 24B-IT 3B-IT

(c) Perceived-need prompt v3 Figure 40: [BFCL Task]: perceived need is only partially aligned with tool use. The x-axis shows perceived utility (number of entities predicted to need or not need external information), and the y-axis shows actual tool-use decisions. Percentages indicate how often the model follows its own prediction (call vs. not call). Results are shown for three prompt variants. Some responses are excluded due to parsing failures (i.e., missing explicit yes/no decisions), so the total count is less than 500. 42

Preprint. Under review.

Perceived

Need

True

(e) Mistral-3.1-24B-IT

Need

True Utility

No Call

144

101

211

159 11 Tool Calling

(f) Llama-3.2-3B-IT

Tool Calling

Call

No Need

37 223

117 53

No Call

(i) Qwen3-30B-A3B-IT

Perceived

Other Positive

True Utility

Other Positive

(d) Gemma-3-27B-IT

No Need

82 139

126 18

Call

True Utility

No Need

No Call

(h) Qwen3-30B-A3B

Other Positive

149 29

Tool Calling

Call

(j) Gemma-3-27B-IT

Tool Calling

Call

No Call

Call

No Call

128

152

126 53

(k) Mistral-3.1-24B-IT

True Utility

219

71 116

True Utility

113 14

Other Positive

Need No Need Need

No Need

Tool Calling

No Call

(g) GPT-OSS-120B

Perceived

Need

No Need Need

(b) Qwen3-30B-A3B

True

No Need Need

Perceived

Call

True Utility

(a) GPT-OSS-120B

No Need

111 141 28

Tool Calling

Other Positive

17 100

True

123 74

Need No Need Need

No Need

No Need Need

True

No Need Need

Need

Utility

Perceived

Other Positive

Need

Perceived

146 10

(l) Llama-3.2-3B-IT

Figure 41: [BFCL Task] The perceived need and utility are not aligned with the true need and utility. Left: perceived need matrices. Right: true vs. perceived utility across models.

Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT

Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description

Maximum Tool Calls 300 250 200 150 100 50 0

0.2 0.1 40%

60%

80%

% Queries with Tool Calls

100%

20%

0,0

0.0

(a) Utility gain with hard stop after exceeding the budget.

0.3

0.4

,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

Actual Tool Calls

Utility Gain over No-Tool

0.5

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 42: Cost-aware tool use on the BFCL Task with implicit budget notification.. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget.

Preprint. Under review.

Gemma3-27B GPT-OSS-120B Qwen3-30B

1.0

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.8

NDCG@k

0.6 0.4 0.2 0.0 0%

20%

40%

60%

80%

100%

% Queries with Tool Calls

Figure 43: [BFCL Task] The NDCG rank correlation under different budgets across different models. The correlation is low, which reflects that the models are not choosing the best utility gain tool calling. Cost prompt v1.

Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT

Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description

Maximum Tool Calls 300 250 200 150 100 50 0

0.2 0.1 40%

60%

80%

% Queries with Tool Calls

100%

20%

0,0

0.0

(a) Utility gain with hard stop after exceeding the budget.

0.3

0.4

,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

Actual Tool Calls

Utility Gain over No-Tool

0.5

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 44: Cost-aware tool use on the BFCL Task with explicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget.

Preprint. Under review.

Gemma3-27B GPT-OSS-120B Qwen3-30B

1.0

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.8

NDCG@k

0.6 0.4 0.2 0.0 0%

20%

40%

60%

80%

100%

% Queries with Tool Calls

Accuracy

Figure 45: [BFCL Task] The NDCG rank correlation under different budgets across different models. The correlation is low, which reflects that the models are not choosing the best utility gain tool calling. Cost prompt v2.

1.00 0.80 0.60 0.40 0.20 0.00

Perceived Need

LNE

27B 20B 30B B-IT 24B -3B ma3P-T-OSS-1Qwen3w- en3-3i0stral3.1L-lama3.2 m e G Q G M Figure 46: BFCL task: The LNE can predict the True Need more accurately across most models, especially for small and weaker models. Predicted

62 31 (26%) (42%)

No Need

242 18 (91%) (38%) 25 (9%)

Actual

178 43 (74%) (58%)

Need Need

No Need

Need

84 52 (43%) (43%)

Need

Actual

Need

110 68 (57%) (57%)

Actual

Predicted

No Need

Need

37 58 (16%) (66%)

Need

No Need

Predicted

No Need

189 30 (84%) (34%)

Actual

Need

27 35 (11%) (53%)

Need

No Need

Predicted

No Need

221 31 (89%) (47%)

Actual

No Need

45 72 (22%) (67%)

Need

No Need

Need

162 35 (78%) (33%)

Actual

Predicted

No Need

Predicted

Need

29 (62%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Accuracy

Figure 47: [BFCL Task.] Confusion matrix for the LNE estimator 1.00 0.80 0.60 0.40 0.20 0.00

Perceived Utility

LUEx, d

LUEx

-3B 27B 20B 30B B-IT 24B ma3- SS-1 Qwen3-wen3-30istral3.1-Llama3.2 Gem GPT-O Q M Figure 48: The LUE can predict the True Utility more accurately across most models, especially for small and weaker models..

Preprint. Under review.

Predicted

No Help

Help Help

116 63 (68%) (44%)

Actual

79 22 (32%) (34%)

Help

171 42 (68%) (66%)

Help

54 81 (32%) (56%)

No Help

39 105 (26%) (65%)

Predicted

No Help

Actual

Help

113 57 (74%) (35%)

Actual

40 104 (25%) (67%)

Help

No Help

118 52 (75%) (33%)

Predicted

No Help

Help Help

Help

Predicted

No Help

Actual

59 68 (34%) (49%)

Actual

115 72 (66%) (51%)

Help

No Help

Help

Actual

No Help

Predicted

No Help

Predicted

Help

No Help

95 61 (64%) (37%) 54 104 (36%) (63%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 49: [BFCL Task.] Confusion matrix for the LUEx estimator

Predicted

No Help

Help Help

131 48 (72%) (36%)

Actual

71 30 (29%) (45%)

Help

176 37 (71%) (55%)

Help

51 84 (28%) (64%)

No Help

43 101 (25%) (72%)

Predicted

No Help

Actual

Help

131 39 (75%) (28%)

Actual

45 99 (28%) (64%)

Help

No Help

115 55 (72%) (36%)

Predicted

No Help

Help Help

Help

Predicted

No Help

Actual

63 64 (32%) (54%)

Actual

132 55 (68%) (46%)

Help

No Help

Help

Actual

No Help

Predicted

No Help

Predicted

Help

No Help

99 57 (66%) (35%) 50 108 (34%) (65%)

(a) GPT-OSS- (b) Qwen3-30B- (c) Qwen3-30B- (d) Gemma-3- (e) Mistral-3.1- (f) Llama-3.2120B A3B A3B-Instruct 27B-IT 24B-IT 3B-IT

Figure 50: [BFCL Task.] Confusion matrix for the LUEx,d estimator

Gemma3-27B GPT-OSS-120B Qwen3-30B

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LNE

0.4

Ideal curve Cost Description LUEx

0.4

0.3

0.2

0.1 0.0

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

0.5

Utility Gain over No-Tool

0.5

Gemma3-27B GPT-OSS-120B Qwen3-30B

0.1

20%

40%

60%

80%

% Queries with Tool Calls

0.0

100%

20%

40%

60%

80%

% Queries with Tool Calls

(a) LNE

100%

(b) LUEx Gemma3-27B GPT-OSS-120B Qwen3-30B

Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B

Ideal curve Cost Description LUEx, d

Utility Gain over No-Tool

0.5 0.4 0.3 0.2 0.1 0.0

20%

40%

60%

80%

% Queries with Tool Calls

100%

Figure 51: [BFCL Task] Tool-call decisions are guided by the latent need estimator’s predicted probabilities under a fixed budget constraint.

Preprint. Under review.

Performance without tool

13 58 23 Mid 3 159 92 High 2 55 95 Low

Neutral

True Need

Performance with tool Low Mid High

Positive

Negative

Figure 52: [Entity Task, Perplexity Search] True need and positive utility are correlated, but not perfectly aligned. Rows group by the model’s (GPT-OSS-120B) factuality scores under N O T OOL (parametric knowledge), while columns show scores under A LWAYS T OOL. Scores are bucketed into low (0–0.1), mid (0.1–0.9), and high (0.9–1). Cells above the diagonal indicate positive utility, while those below indicate negative utility. The bracket highlights the true need region, where low N O T OOL scores reflect insufficient parametric knowledge.

140

Call

144

Call

144

82 274

356

No Call

356

(a) Perceived Need Prompt V1

Tool Calling

No Need

Tool Calling

Need

Tool Calling

No Need

No Call

Perceived

Need

Call

Perceived

No Need

No Call

Perceived

Need

(b) Perceived Need Prompt V2

Figure 53: [Entity Task; Perplexity Search] Perceived need is only partially aligned with tool call (perceived utility). Model: GPT-OSS-120B. The x-axis shows the model’s perceived need, and the y-axis shows perceived utility / tool-call decisions.

64 1

30 78

True Positive Utility Perceived Need Perceived Utility

52 62

3 Figure 54: [Entity Task; Perplexity Search] Perceived signals only partially align with true utility. Venn diagrams of True Positive Utility, Perceived Need, and Perceived Utility for GPT-OSS-120B on the entity task illustrate this misalignment. Ideally, Perceived Utility ⊆ Perceived Need ⊆ True Positive Utility. However, this nesting is violated, indicating misalignment with true utility and helping to explain the suboptimal performance of S ELF - DECISION.

Perceived

Tool Calling

No Need

Call

No Call

207 141

True Utility

Other Positive

True

No Need Need

Need

15 137

(a) True Need VS Perceived Need

65 262

(b) True Utility VS Perceived Utility

Figure 55: [Entity Task; Perplexity Search] The perceived need and utility are not aligned with the true need and utility. Entity Task; Top: perceived need matrices. Bottom: true vs. perceived utility across models.

Preprint. Under review.

No Cost Description

Maximum Tool Calls 500

Utility Gain over No-Tool

GPT-OSS-120B Cost Description 0.20

Actual Tool Calls

400

0.15

300 200

0.10

100

0.05 80%

100%

(a) Utility gain with hard stop after exceeding the budget.

60%

40%

% Queries with Tool Calls

20%

0,0

,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

0 $1

0.00

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 56: [Entity Task; Perplexity Search] Cost-aware tool use with explicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget.

No Cost Description

Maximum Tool Calls 500

Utility Gain over No-Tool

GPT-OSS-120B Cost Description 0.20

Actual Tool Calls

400

0.15

300 200

0.10

100

0.05 80%

100%

(a) Utility gain with hard stop after exceeding the budget.

60%

40%

% Queries with Tool Calls

20%

0,0

,00 $5 0 0 $2 0 50 $1 00 $5 0 $2 5 $1 0

0 $1

0.00

Tool Cost ($)

(b) Tool-calling behavior without hard stop.

Figure 57: [Entity Task; Perplexity Search] Cost-aware tool use with implicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost increases, despite being provided with cost and remaining budget.

Record · ID 151770 · SHA-256 0a3eae6894fc4056
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.