arXiv CSopen access

Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents

Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents Shiyang Chen

arXiv:2606.16364v1 [cs.AI] 15 Jun 2026

Beijing Institute of Technology [email protected]

Abstract LLM agents mis-call tools, and the natural guess is that the model failed to see the right tool in a crowded harness. We show the opposite, through a lens concurrent activationspace work explicitly sets aside—the model’s attention to labeled tool-definition segments. On real BFCL failures, by per-candidate attention argmax the model attends most to the correct tool 80% of the time (vs. 21% chance), and the gold is the literally under-attended segment on only 10%: it looks at the right tool and still picks wrong. This directly refutes the intuitive “crowded-harness / lost-in-the-middle / didn’t-seeit” explanation: the failure is at the decision readout, not the harness, and we pin it there three ways. (1) Input vs. readout: repairing the prompt (reordering or duplicating the gold tool) recovers ≤ 23% of failures, while readout-side interventions recover 59–91%. (2) Representation-invariance: two gold-pointed interventions in different representations— an additive attention-logit bias and a residual-stream steering vector—recover largely the same failures (per-task Jaccard 0.865 pooled, 0.79–0.91 per model), so the bottleneck is localized to the readout independent of which representation is poked. The per-model recovery rates differ on every model (coinciding at 0.909 only pooled), so we claim no superiority—the per-task overlap is the evidence. (3) A training-free, gold-free selector: the same per-segment attention closes most of the gold-free–vs–oracle gap on BFCL (+11.9 pts pooled function-name selection vs. the +17.9-pt oracle headroom) and adds an absolute +14.9 pts on SealTools (no Seal oracle ceiling run); every model positive (exact McNemar p ≤ 8 × 10−4 each, p < 10−7 on 3 of 5, pooled 6.5 × 10−20 ). Scopes differ: the causal attention-bias dose-response is bidirectional and monotonic on the 10 maskhonoring models (3–32B; Phi-3.5 is the lone flat exception, a method artifact), the full 0.5–32B span carrying only the correlational diagnostic; the deployable selector is evaluated on 5 single-turn models and does not yet transfer to a multi-turn loop. We position honestly: concurrent work establishes the readout bottleneck in activation space (Wu et al. 2026; Wang et al. 2026); our contribution is the attention-segment account those methods set aside—the perceptual “attended-not-picked” refutation, the input-vs-readout contrast, and the attentionlogit lever—and we delimit the fix to function selection, not arguments (§8).

Copyright © 2026, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

1

Introduction

LLM agents are wrapped in a harness: a system prompt enumerating the tools or functions the model may invoke, with signatures and natural-language descriptions. Harness quality materially affects whether the agent succeeds, yet it is evaluated by running an end-to-end agent benchmark (Yan et al. 2024; Patil et al. 2025; Yao et al. 2024; Liu et al. 2024b; Qin et al. 2024) and reading off a success rate—output-level evaluations that tell you that a call failed, never how the model’s internal attention over the harness related to its choice. Our title asks the provocative version—is the model even looking at your tools?—but the honest answer is more interesting: on real BFCL failures the model usually does attend most to the correct tool yet still mis-selects it (§8). The operative question is whether the model’s selection follows its attention—and whether that attention is a readable, steerable handle on the selection. We show it is: a per-instance, per-call control signal (not a static ranking of harness designs—attention does not universally rank them). We study the model’s internal attention to harness content as a directly observable, training-free, and—crucially— causally actionable handle. We define Harness Attention Allocation (HAA): for a prompt whose harness is decomposed into labeled segments (each tool definition is one segment), HAA is the raw per-segment attention mass that the model routes to each segment from the tokens where the tool decision is being made. The headline quantity is not this raw mass but the attention margin attn margin = HAA(gold) − HAA(distractors),

(1)

which differences out the attention-sink and gold-position common-mode rather than correcting either per segment (sink mass is logged only as a separate control). What this paper claims, and what it concedes. Concurrent activation-space work already establishes that the correct tool is linearly readable inside the model and that toolselection failures sit at the output readout, not in forming the representation (Wu et al. 2026; Wang et al. 2026). We do not re-claim that thesis. We give it an attention-segment account—the lens those methods explicitly set aside (Wu et al. (2026) caution against reading raw attention)—and the converging concessions below read as evidence the readout bottleneck is real, not as others having done the interesting

part. The three results below—the looking-vs-picking refutation (§8), the input-vs-readout contrast with a representationinvariant readout lever (§6), and a gold-free selector (§8), detailed in the contributions—are none of them provided by the residual-stream line. Prediction itself is benchmarkdependent—a hidden-state probe out-predicts the margin on synthetic data (§5)—so we do not pitch attn margin as a better detector; the refutation, the input-vs-readout contrast, and the attention-logit lever are the contribution, and “attention is not explanation” (Jain and Wallace 2019; Wiegreffe and Pinter 2019) does not apply because we manipulate it. Contributions. 1. The looking-vs-picking gap. On 198 real BFCL failures, per-candidate attention argmax localizes the gold tool 80% of the time (vs. 21% chance; the gold is underattended only 10%)—the model attends to the right tool and mis-selects it (Section 8). Wu et al. (2026) disclaim raw attention; the segment-grounded margin recovers the perceptual side of the readout bottleneck that the activation line sets aside. 2. Readout, not input—and representation-invariant. On the same failures, input-side repairs (reorder, duplicate the gold tool in the prompt) recover ≤ 23%, while readoutside interventions recover 59–91% (Section 6, Table 1). Two gold-pointed levers in different representations— an additive attention-logit bias (ours; untouched by the residual-stream line) and a residual steering vector— recover largely the same failures (per-task Jaccard 0.865 pooled, 0.79–0.91 per model), localizing the bottleneck to the readout independent of representation. Their permodel recovery rates differ on every model (coinciding at 0.909 only pooled; Table 1), so we read the per-task overlap—not the pooled rate—as the equivalence, and claim no superiority. The attention-bias dose-response is bidirectional and monotonic on the 10 mask-honoring models (3–32B; Phi-3.5 is the lone flat exception, a method artifact, and the two smallest Qwen models carry the correlational diagnostic only; Section 7). 3. A training-free, gold-free selector (single-turn). The same per-segment attention, as a confidence-gated selector, lifts function-name selection on two real benchmarks: BFCL +11.9 pts pooled (+11.2 over five single-turn models incl. Qwen3-8B and tool-finetuned xLAM-2-8B) and Seal-Tools +14.9 pts pooled (an absolute gold-free gain; no Seal oracle ceiling run), every model positive (exact McNemar p ≤ 8 × 10−4 each; p < 10−7 on 3 of 5; pooled 6.5 × 10−20 ), closing most of the gold-free–vs– oracle gap on BFCL and held out on disjoint splits (+12.1 pts, App. A), at ∼1.2× decoding cost (Section 8). Its edge over a zero-corpus residual-cosine selector (+11.9 vs. +5.1 pts) is that it needs no per-tool corpus, not that attention is the more accurate signal. 4. Honest scope. The contribution is which function is named, not its arguments. Under the official AST metric the constrained-rescoring protocol loses to free generation, so we do not pitch a deployment win: as a scope-delimiter we report that the name fix is recoverable inside free generation as a soft advisory (pooled AST within noise,

+1.8 pts n.s.; +6.0 on the weakest model, vs. −18.3 for hard name-forcing; Section 8), and that the selector does not transfer to a multi-turn loop (Section 9). Prediction is benchmark-dependent—a trained probe wins on the synthetic benchmark, the margin on real BFCL, neither localizes or acts (Section 5)—and the name-token circularity is small (0.893 → 0.876 real-BFCL AUROC excluding emitted name tokens, Section 8).

2

Related Work

Positional utilization, sinks, attribution, and causal heads. Liu et al. (2024a) show “lost in the middle” at output level and Xiao et al. (2024) document attention sinks; HAA corrects for both via a content-token margin and a separate sinkmass control, then tests the lost-in-the-middle effect inside attention on tool-definition segments (Section 4). Hsieh et al. (2024) calibrate per-passage attention bias for long-context RAG (unit: passage; goal: QA accuracy), while ContextCite (Cohen-Wang et al. 2024) and AttnLRP (Achtibat et al. 2024) attribute a given output to context; HAA’s unit is instead a labeled tool-definition segment, scored per-instance for percall tool selection and reused as a deployed selector (we do not claim it ranks harness designs). Induction heads (Olsson et al. 2022), function vectors (Todd et al. 2024), retrieval heads (Wu et al. 2025), and ICL head analyses (Yin and Steinhardt 2025) explain ICL/needle mechanisms but give no per-instance tool-selection signal over enumerated tool definitions. Attention as intervention (InstABoost, SpotLight, Attention Buckets). Guardieiro et al. (2025) (v3) steers behavior by adding a constant bias to attention toward an instruction span. Venkateswaran and Contractor (2026) (SpotLight), the closest prior “measure-then-boost” intervention, adds a deficit-gated additive bias log(ψtarget /ψcurrent ) when a span’s share is deficient, but is boost-only (no bidirectional doseresponse), has no candidate selection (preset user-marked span), and runs no tool/agent tasks. Earlier, Attention Buckets (Chen et al. 2024) and its successor MoICE (Lin et al. 2024) tied a position→attention-trough effect on tool use to function-call failure and intervened in attention space (ensembling parallel RoPE-base forwards), reaching GPT-4-level tool use on ToolBench, but use a global positional waveform with no way to name a tool’s segment, no diagnostic margin/AUROC, no signed dose-response, and no gold-free selector. Concurrent Agent-Radar (Zhang, Tian, and Zhang 2026) steers each agent’s attention toward relevant conversation history in multi-agent systems—evidence that attention steering is a live axis—but targets dialogue context, not tool-definition segments, and selects no tools. We adopt the additive-bias mechanism on labeled tool-definition segments; our deltas across all of these are the per-tool-segment instantiation, a gold-vs-distractor margin, a bidirectional signed knob, real-failure recovery, and a gold-free selector. Internal-signal diagnostics/fixes for tool use (concurrent 2025–2026). A growing line reads—and in several cases steers—tool choice from hidden states: Wu et al. (2026) show tool calling is linearly readable and steerable in activation space (an α-sweep flips the chosen tool, with a

gold-free cosine selector); Wang et al. (2026) (ASA) is a backbone-training-free representation-engineering steering method; alongside SAE pre-execution monitors (Tatsat and Shater 2026), hallucination classifiers (Healy et al. 2026), and tool-need probes (Sun et al. 2026)—and the broader knowing– doing gap in tool use (Cheng et al. 2026). So hidden states are both an accurate readout—which we confirm (§5)—and causally steerable; this independent activation-space evidence makes the readout-bottleneck claim more credible, not less. We concede that thesis and contribute the attention account it leaves open (Table 13): the perceptual “attended-not-picked” refutation that the residual line never measures (§8), segmentgrounded localization, an intervention on attention logits rather than the residual stream, a bidirectional dose-response, and a gold-free per-candidate selector. Closest on the attention side, Noel (2026) detect tool-use failures training-free from a global attention-topology (graph-Laplacian spectrum) signal, but it is detection-only—no per-tool-segment grounding, no causal lever or signed dose-response, and it selects no tool. PASTA (Zhang et al. 2024) reweights attention on user-marked spans—a different mechanism for a different purpose. Closest measurement: MindGuard. Wang et al. (2025) (MindGuard) is the nearest prior work on our measurement primitive: for agent security it compares a sink-filtered attention “energy” from the decision token to an invoked vs. uninvoked tool’s metadata—essentially an attn margin readout. We do not claim the readout as novel; MindGuard is converging evidence it is real. But MindGuard is correlational and detective only (poisoning provenance), never intervenes and never chooses a tool, whereas HAA establishes the same attention as a causal, bidirectional knob (§6), a fix on real failures (§8), and a gold-free selector (§10). The “Tool Attention” gating of Sadani and Kumar (2026) adopts MindGuard’s attention-energy framing only as motivation: it gates tool schemas before the forward pass with an embeddingsimilarity proxy in a simulation-based evaluation—no transformer attention is measured at inference, no causal doseresponse, no gold-free selector. What we add over the closest concurrent work. The readout-bottleneck thesis is not ours (Wu et al. 2026; Wang et al. 2026); we concede it and contribute its attention account—the lens Wu et al. (2026) explicitly disclaim. Table 13 states exactly what HAA adds over both.

3

Method

Controlled tool-selection benchmark. A task presents a system prompt enumerating K tool definitions (1 gold +K−1 distractors) and a user query that unambiguously requires the gold tool; the model emits tool name(arg=value,...) and ground truth is the gold name. The library has 24 distinct tools (each with a signature, terse/verbose descriptions, and a query template); tasks are deterministic given a seed. We vary verbosity, gold position ∈ {first,middle,last}, and K (K=6 main), giving 6 conditions/task. For the generality/baseline/selector studies we add a confusable variant whose distractors are members of the gold’s 3-tool

near-synonym family (e.g. get current weather vs. get weather forecast), inducing real failure variance on instruct models that saturate the distinct-distractor benchmark. Extracting HAA (eager HF attention). We run HuggingFace transformers 5.10.2 in eager-attention mode (with output attentions on; vLLM (Kwon et al. 2023) cannot expose attention weights), greedily generate, record success, then run one forward over prompt+generation with attentions on; tool definitions are matched to token indices by offset-mapping substring search. With Q the query positions (generated answer tokens; we also log a last-prompt-token variant), a layer’s per-segment mass is P (ℓ) 1 mℓ (K) = H|Q| h,q∈Q,k∈K Ah,q,k and the layer-averaged attn margin is the gold-minus-mean-distractor difference (Eq. 1). Causal intervention: additive attention bias. We pass a custom 4D additive attention mask adding a constant bias δ to the attention logits of every query position toward a chosen segment’s key columns, at every layer and head— the constant-bias intervention of Guardieiro et al. (2025). It requires a model whose attention path honors a passed 4D additive mask: Llama, Qwen, Yi, and Mistral do, Phi-3 does not (Section 7). We separately log sink mass (attention to token 0) as a control and report an attention-rollout flow score (Abnar and Zuidema 2020) and the cross-layer gold-segment concentration slope. Constrained decision. For causal/baseline/selector runs, for each candidate c we teacher-force the call prefix name( and take its length-normalized log-prob; predicted tool is argmax and P (gold) is the softmax over candidate scores. We sweep target ∈ {boost gold, boost distractor} and δ ∈ {−12, −8, −4, 0, +4, +8, +12}.

4

Correlational results

On 150 tasks × 6 conditions (3600 rows) over MetaLlama-3.1-8B-Instruct (Dubey et al. 2024), Qwen2.5-7B/3BInstruct, and Qwen2.5-Coder-7B-Instruct (Qwen Team 2025), the gen-query attn margin predicts per-task tool-selection success with pooled AUROC 0.751 ([0.686, 0.817]; permodel 0.679–1.000, two cells near-ceiling and uninterpretable; 2000-resample bootstrap CIs; Appendix B, Table 10). These are generated-answer-token positions, so this is post-hoc attribution; the matched pre-decision last-token margin is much weaker (0.603 pooled). Failures separate from successes by attending relatively less to the gold definition— a distributional statement that does not imply the gold is the literally under-attended segment on any given failure (§8). We treat this as supporting evidence, not the headline: a trained hidden-state probe predicts the same outcome at least as well (§5), so the value of attn margin is its causal actionability, not its raw AUROC. Lost in the middle, inside the harness. attn margin by gold position reproduces the Liu et al. (2024a) U-shape on tool-definition segments (first 0.0449, middle 0.0271, last 0.0405; pooled n=1200 each). Task success barely moves

(a) Causal dose-response: P(gold)

0.8 0.6 0.4

boost gold boost distractor

0.2

5

Trained-probe baseline: who predicts better is benchmark-dependent

Before building on attn margin, we confront the strongest competing explanation. The concurrent tool-interpretability literature reads tool choice from hidden states with a trained probe (Wu et al. 2026; Wang et al. 2026; Healy et al. 2026; Tatsat and Shater 2026; Sun et al. 2026): is attention just a worse version of that signal? On the confusable benchmark we train a logistic probe (L2 -regularized, stratified 5-fold CV) on the final-layer residual stream at the decision token and compare it head-to-head with attn margin (Appendix B, Table 5). Who wins is benchmark-dependent. On the synthetic benchmark the probe wins (AUROC 0.97/0.98 on the two models with failure variance, vs. attn margin(gen) 0.922/0.573; the edge is real—shuffled-label control 0.49/0.51, a 32-dim PCA probe still 0.90/0.95). But on real BFCL live multiple (300 tasks, same CV) the training-free attn margin(gen) post-hoc separates failures competitively-to-better—0.898 vs. probe 0.736 on 1.5B, 0.824 vs. 0.857 on 7B (near-tie). This is not a clean flip (the margin uses generated-answer positions, the probe a predecision hidden state), but the training-free margin stays robust on diverse real schemas with no labels while the probe’s controlled-benchmark edge shrinks—a tool-selectionspecific observation, not a general law (synthetic-trained probes transfer robustly elsewhere, McKenzie et al. 2025). So which signal wins is benchmark-, data-, and query-positiondependent. The decisive difference is actionability, not accuracy. A trained residual probe says a call will fail but not—without an extra per-tool direction vector—which segment to select; HAA localizes it directly (80% vs. 21% chance on 198 real failures, §8) and deploys as a gold-free selector. Activation methods can also be made causal (Wu et al. 2026; Wang et al. 2026 steer the residual stream; the equivalence is the point, §6)—HAA’s distinct value is direct segment localization plus that selector.

6

Causal dose-response

We intervene on the measured quantity. With the additive bias of Section 3 we run 4 models × 80 tasks × 3 positions × 7 deltas × 2 targets = 13,440 trials (Figure 1): pooled, boosting the gold segment drives P (gold) 0.18 → 0.90 and success 0.19 → 0.97 over δ ∈ [−12, +12], while boosting a distractor collapses both (P (gold) → 0.02). Per-model Spearman of P (gold) vs. δ is +0.857 to +0.964 (boost gold) and −0.893 to −1.000 (boost distractor) on the four base models (Figure 1); the “attention is not explanation” objection (Jain and Wallace 2019) does not apply because we manipulate attention.

0.0

10

5

0

attention bias

1.0

tool-selection success

1.0

P(gold)

here (all positions near ceiling), motivating the confusable benchmark in Sections 7–10. Pooled gold-segment attention has a positive cross-layer concentration slope (+9.3 × 10−4 over 28 layers): the gold-vs-distractor signal concentrates in deeper layers.

5

10

(b) Causal dose-response: success

0.8 0.6 0.4

boost gold boost distractor

0.2 0.0

10

5

0

attention bias

5

10

Figure 1: Pooled causal dose-response (4 models, n=960/cell). Boosting the gold segment (solid) drives P (gold) and constrained-selection success up monotonically; boosting a distractor (dashed) collapses them. The signed, bidirectional shape— not a one-sided “more salience helps” curve—converts the Section 4 correlation into a causal handle.

This is not seed noise: across three seeds the boost-gold Spearman is {+0.964, +0.929, +0.964} (Qwen2.5-3B) and {+0.964, +0.964, +0.964} (Llama-3.1-8B). The causal effect is localized to deep layers. Restricting the gold-boost to one quarter of layers at a time (Qwen2.53B/Llama-3.1-8B, 1,920 trials) concentrates the effect in the last quarter: boosting only the deepest quarter (δ=+8) lifts pooled P (gold) 0.666 → 0.914 and success 0.817 → 0.992, exceeding the all-layer boost (0.773/0.842), while boosting the first two quarters hurts (P (gold) ↓ 0.32–0.37). That toolselection is decided in late layers is not new—activationspace work localizes it there (Wu et al. 2026; Tatsat and Shater 2026) (and the layer-wise structure of attention-asrelevance is concurrently mapped for re-ranking, Chen et al. 2026), consistent with late retrieval/function-vector heads (Wu et al. 2025; Todd et al. 2024) and the slope of Section 4. Our distinct point is narrower and interventional: the attention-bias knob is sharpest, and only effective, in the last quarter, which a layer-restricted additive mask shows directly rather than by activation patching. Input vs. readout, and representation-invariance. A readout bottleneck repaired by an attention lever is only apparently paradoxical. The gold is already the most-attended segment (argmax over candidates) on 80% of failures (§8), so the late-layer readout is not failing to see the gold—it underweights the existing, small gold-minus-distractor margin (Eq. 1). The additive bias amplifies that margin past the latelayer conversion threshold—why the effect is sharp only in the deepest quarter (above) and why prompt re-presentation, leaving the margin unchanged, barely helps. The repair levers split cleanly into two camps on the same 900 real BFCL live multiple tasks (198 failures, gold matched; Table 1). Input-side edits to the prompt—reorder the gold tool first (18.2% recovery), duplicate it (16.7%)—barely help, since the harness already showed the model the gold tool (§8). Readout-side interventions recover far more—the attention-logit boost 90.9%, an ASA-style residual-stream steering vector 90.9%, the output-logit lever 59.1%—so “any salience bump recovers failures” is refuted: only read-

Table 1: Repair levers on the same 900 BFCL live multiple tasks (pooled over 3 models; 198 baseline failures, 702 successes). Rec. = failures recovered; Dmg. = successes broken. Pooled, the attention-bias knob matches an ASA-style residual steering vector (0.909 each; both are gold-pointed oracle levers, and the pertask recoveries overlap with Jaccard 0.865); per model the rates differ (attn {0.893, 0.964, 0.847} vs. residual {0.946, 0.904, 0.881}). Both dominate prompt edits and the output-logit lever; the desc-only control (gold segment minus the literal name( tokens) still recovers 44%—not a pure copy tautology—but at high damage. Intervention

Acc.

Rec.

Dmg.

Baseline (δ=0) Attn boost gold seg (δ=+8, ours) Residual steering toward gold (oracle) Output logit bias on gold name Attn boost desc-only (exclude name) Prompt reorder gold-first Prompt duplicate gold

0.780 0.959 0.966 0.910 0.574 0.780 0.784

— 0.909 0.909 0.591 0.439 0.182 0.167

— 0.027 0.019 0.000 0.387 0.051 0.041

out interventions work, which is what a readout bottleneck predicts. The two strongest readout levers are moreover representation-invariant in which failures they recover: the attention-logit bias and the residual steering vector overlap with per-task Jaccard 0.865 pooled (0.79–0.91 per model; offline in analyze lever invariance.py). We lead with this overlap, not the recovery rate, because the permodel rates in fact differ on every model and average to the same 0.909 only pooled (per-model attn/residual in Table 1); both arms are gold-pointed interventions on the same segment, so part of the overlap is mechanical, and the Jaccard— not the rate—carries the equivalence. So the bottleneck is localized to the readout regardless of representation, and we make no superiority claim; the equivalence is the evidence, while the attention-logit site is one the residual-stream line (Wu et al. 2026; Wang et al. 2026) does not touch. HAA’s remaining distinct value is the gold-free selector (§8) and segment localization (§8), neither of which an oracle steering vector provides. The boosted gold segment includes the literal name( tokens, so the gold-boost is partly a copy effect; but boosting only the description tokens (excluding the name) still recovers 43.9% (at high 38.7% damage), so it is not a pure tautology—converging with Zheng et al. (2026), who find the skill body (not name) carries the decisive cross-encoder attention—while the zero-damage output-logit lever (59.1%) is a weaker tautology-free alternative.

7

Generality and scaling

We replicate the causal intervention and the confusable correlational probe on 7 additional models spanning 6 families and 0.5–32B (per-model Spearman ρ of P (gold) vs. δ and confusable AUROC in Appendix C, Table 12). The causal dose-response is carried by the 10 mask-honoring models (3–32B; the two smallest Qwen models show “—” for ρ and contribute the correlational diagnostic only), and

it holds at scale (e.g. Qwen-32B +0.643/−0.929; Yi-1.5-9B +0.929/−0.893; confusable AUROC at Qwen-14B 0.926); Phi-3.5 is the lone flat exception, as it does not consume a passed 4D additive mask. Not a 2024-era artifact. The dose-response is equally clean on two 2025-era models (3,360 trials each): Qwen3-8B (Yang et al. 2025) (thinking off; boost-gold success 0.171 → 1.000, ρ=+0.821/−1.000) and the tool-finetuned LlamaxLAM-2-8B (Prabhakar et al. 2025) (success 0.158 → 1.000, ρ=+0.964/−0.893)—tool-specific finetuning does not close the attention-causal pathway. Robustness to harness size. The headline benchmark uses K=6 tools; real harnesses are larger. Re-running the goldboost at K=12/20 (Qwen2.5-3B, Llama-3.1-8B) leaves the dose-response intact (ρgold =+0.857 to +0.929; δ=+8 still drives success to 0.97/0.68–0.79), and larger harnesses restore the failure variance K=6 lacks (Qwen-3B baseline 0.55 → 0.37)—so the knob matters more as the harness grows, exactly the deployment regime. Phi-3 scope limit. Phi-3.5-mini-instruct (Abdin et al. 2024) has a literally constant response (P (gold)=0.164 across all δ and both targets) and no usable confusable-probe rows under the offset-mapping path that works on every other family: it does not consume a passed 4D additive mask—a methodological scope limit. Training-free output-confidence/lexical baselines. Against the training-free signals that are attn margin’s real deployment-cost peers (pgold , first-token max-prob/negentropy, lexical overlap; confusable benchmark, 960 pooled; Appendix B, Table 11), pooled attn margin=0.691 beats pgold =0.656, the first-token signals (≈ 0.62), and lexical (0.558); the gap is large on 0.5B (0.921 vs. 0.833, nonoverlapping CIs) though pgold edges it on 1.5B (overlapping CIs). Unlike pgold , attn margin is a segment-grounded localizer (§8) and causally actionable (§6)—the reason to prefer it.

8

Real-benchmark anchor: the BFCL live-multiple split

We run two protocols on the same 300 BFCL live multiple tasks: a diagnostic run (free-generate, read the gen-query attn margin) and a recovery run (constrained candidate scoring of the function name). On the diagnostic run the gen-query attn margin post-hoc separates function-name-selection failure with pooled AUROC 0.893 ([0.833, 0.943]; per-model 0.898/0.824/0.970). “Success” throughout this section is constrained function-name selection on BFCL-derived prompts, not the official BFCL AST/argument evaluation; §8 and §8 re-test under argumentlevel (AST) checking and show the fix is function-selection only. Quantifying the name-token circularity of the post-hoc margin. The gen-query positions include the tokens of the just-emitted name, so copy/induction heads could make the diagnostic partly circular. Re-running the diagnostic with de-circularized variants (300 tasks × 3 models): pooled

Table 2: Real-benchmark fixes (constrained function-name selection, not official AST): oracle boost (δ=+8, known gold) vs. gold-free confidence-gated S2 (∆ vs. base, points; exact McNemar p ≤ 8 × 10−4 per model, < 10−7 on 3 of 5 models—all but Qwen2.5-1.5B (p=3.7 × 10−3 ) and Qwen3-8B (p=8×10−4 ); pooled 6.5×10−20 ). The gold-free selector closes most of the gold-free–vs–oracle gap on BFCL (0.899 vs. 0.959) and replicates on two 2025-era models and a second real benchmark (294 tasks/model). Model

Base

Oracle δ+8 Gold-free S2 (∆)

BFCL live multiple (300 tasks/model) Qwen2.5-1.5B 0.813 0.977 0.880 (+6.7) Qwen2.5-7B 0.723 0.940 0.900 (+17.7) Llama-3.1-8B 0.803 0.960 0.917 (+11.3) Pooled (3) 0.780 0.959 0.899 (+11.9) Qwen3-8B 0.837 — 0.913 (+7.7) xLAM-2-8B 0.720 — 0.847 (+12.7) Pooled (5) 0.779 — 0.891 (+11.2) Seal-Tools (single-call, K=5) Qwen2.5-1.5B 0.796 — Qwen2.5-7B 0.714 — Llama-3.1-8B 0.765 — Pooled 0.759 —

0.915 (+11.9) 0.874 (+16.0) 0.932 (+16.7) 0.907 (+14.9)

AUROC moves 0.893 → 0.876 excluding the emitted name tokens from the query set, 0.895 excluding the literal name tokens from each segment’s keys, and 0.877 [0.841, 0.910] with both—only ≈ 4% of the above-chance separation is name-token copying. The per-model shift is nonmonotone—excluding name tokens raises the Qwen2.5-7B AUROC (0.824 → 0.905)—so no single model is circularitydominated. The same run gives the matched pre-decision last-prompt-token margin on real BFCL: pooled AUROC 0.787 [0.731, 0.840]—weaker than post-hoc but far from the near-chance synthetic value (§4): on real schemas the margin is usable before the call is emitted, which the selector below exploits. At the sweet-spot δ=+8 the oracle boost on the knowngold segment recovers 90.9% of constrained function-nameselection failures while breaking only 2.7% of successes, lifting pooled success 0.780 → 0.959 (net +17.9 points; Table 2). This is an oracle fix—boosting the segment we already know is gold—so it upper-bounds what a gold-free deployment could recover; δ=+8 is the joint sweet spot on all three models (δ=+4 under-boosts, δ=+12 over-boosts), matching Section 6 and dev-selected in 99% of held-out 50/50 splits for the two Qwen models (App. A). A gold-free selector that works on real BFCL. The oracle boost knows the gold; can a gold-free version recover that headroom on real data? The same confidence-gated S2 selector as Section 10 (gold-free: last-prompt-token per-candidate HAA over the baseline top-3, argmax, deferred when the output top-1-vs-top-2 logprob margin is below the per-model median; no gold, no tuning) improves all three models, lifting pooled accuracy 0.780 → 0.899 (+11.9 pts in-sample, paired task-level bootstrap [+9.3, +14.4], exact McNemar 128 fixes

vs. 21 breaks, p=6.5 × 10−20 ; Table 2; held-out τ gives an indistinguishable +12.1 pts, positive in 100% of splits, App. A), positive for every fixed τ ∈ [0.5, 3.0]. It replicates beyond 2024-era models: on the 2025-generation Qwen38B (thinking disabled) it adds +7.7 pts (0.837 → 0.913, p=8 × 10−4 ) and on the tool-finetuned Llama-xLAM-2-8B +12.7 pts (0.720 → 0.847, p=3 × 10−8 ); five-model pooled +11.2 pts. The gold-free 0.899 closes most of the gap to the oracle’s 0.959 without knowing the gold—a 6.0-pt residual gap, ∼ 32 of the +17.9-pt oracle headroom the oracle only upper-bounds. The gold-free edge is over a zerocorpus residual baseline: a residual-space counterpart with no learned per-tool direction (argmax over candidates of the cosine between the decision-token residual and each segment’s mean residual) lifts the same tasks only +5.1 pts at ∼4× the damage. We do not read this as “attention is the more accurate signal”—a trained per-tool residual direction (the very thing the probe of §5 shows residuals support) might match or beat it. The honest claim is narrower: as oracle levers attention and residual steering tie (Table 1), but per-candidate attention is a strong gold-free, zero-corpus selector—no pertool corpus needed, the property that matters when one is unavailable. (It still does not transfer multi-turn, §9.) Second real benchmark: Seal-Tools. To rule out a BFCLspecific artifact we re-ran the identical end-to-end pipeline on the single-call subset of Seal-Tools (Wu et al. 2024) (294 tasks across the in/out-of-domain splits, K=5 realstyle API schemas per task, same per-model median gate). The gold-free gated selector improves all three models again (per model +11.9/+16.0/+16.7 pts, Table 2), pooled 0.759 → 0.907 (+14.9 pts, [+12.4, +17.3], McNemar p=1.1 × 10−32 )—with larger gains than on BFCL. Attention localizes the gold even when the output misselects (validating attribution). On the 198 real baseline failures pooled across the three models (offline from the percandidate last-prompt-token HAA logs; no GPU), the argmax over per-candidate HAA points to the gold tool on 0.80 of failures (per-model 0.71/0.86/0.80), far above the 1/K chance rate 0.21 (Table 15; chance-corrected tool-selection and the role of shortlist size K are studied orthogonally by Repantis et al. 2026)—the mechanism the gold-free selector exploits. This is not the gold being under-attended: in ∼90% of failures the gold out-attends the wrongly-picked tool (pairwise; it is the literally under-attended segment on only 10%, and the most-attended segment overall—argmax over candidates—on 80%; Table 15), yet decoding does not follow it. HAA is a segment-grounded localizer, not a detector of neglect: the model is looking at the gold tool, it just mis-selects. End-to-end AST evaluation: the constrained protocol, not the selector, fails the real metric. The selector’s gains above are on constrained name selection; the official BFCL metric also scores arguments (AST). We evaluated the whole deployable pipeline at the AST level (Appendix B, Table 8) and report both findings. Positive: name repair propagates— the gated pipeline beats its own forced-name baseline arm on every model (pooled +5.2 pts on BFCL, +8.7 on Seal-Tools).

Negative: the constrained-rescoring protocol it rides on loses to free generation outright (−19.1 pts pooled), and the deficit is selector-independent—even forcing the gold name scores below free-gen (0.24/0.44/0.55 vs. 0.28/0.67/0.77): teacherforcing name( pulls the model out of its native call format and costs more argument accuracy than perfect name selection buys back. The AST deficit is a teacher-forcing artifact, not a property of the name signal. Delivering the same gold-free S2 recommendation as a soft advisory (a one-line prompt hint) and free-generating the whole call—so the native format is never overridden (Table 9, 3 models, 900 tasks)—isolates the protocol from the signal: the advisory scores pooled AST +1.8 pts above free generation, within noise (McNemar p=0.06, n.s.; significant only on the weakest model, +6.0 on Qwen2.51.5B), versus −18.3 for hard name-forcing on the same rows. We read this as a scope-delimiter, not a deployment win: once the constrained-protocol confound is removed the name fix is at worst neutral and at best modestly positive at the real metric, with net-zero argument-level effect (§8); we do not claim AST improvement as a contribution. Free generation and arguments: the fix is functionselection, not the whole call. The recovery numbers above use constrained name scoring; the matched free-generation name rate is high (pooled 0.929) but overstates real call accuracy, which also depends on arguments. Re-running the δ-sweep under free generation (3 models × 300 tasks; ast ok = gold name and all ground-truth arguments; Table 7): the constrained sweet-spot δ=+8 over-boosts and collapses name accuracy (0.929 → 0.20), but a calibrated δ=+2 recovers function-name selection (recovery 0.67/damage 0.04), so the selection fix does transfer; argument-level recovery is net-zero (best 0.13/0.13). Boosting the tool-definition segment fixes which function is named, but arguments come from the query and reasoning—a scope result, not a failure. Compute cost: the gold-free selector adds ∼1.2×. On real BFCL prompts (median over 40 tasks; Appendix B, Table 6), one ∼15 ms HAA forward plus three candidate scorings is dwarfed by the 226–283 ms baseline generation: ∼1.2× over baseline decoding. The real constraint is not latency but the eager-attention requirement (no vLLM/FlashAttention; §3).

9

Multi-turn ecological anchor: τ -bench

To test whether HAA survives a real multi-step context, we run a scripted, teacher-forced (gold-advanced) probe over the τ -bench (Yao et al. 2024) airline domain (14 real tools, a stateful DB; 120 users × 3–4 steps with a known gold tool; not a full agent loop). The load-bearing multiturn result is the causal knob: the causal additive-bias knob transfers cleanly (boost-gold pooled P (gold) 0.06 → 0.85, Spearman +1.00; boost-distractor collapses success). The diagnostic transfers too, but only as an existence proof: on the two models with genuine multi-step failure variance the genquery attn margin separates per-step failure with AUROC 0.985/0.998—structurally trivial, since this probe has only 7 unique gold tools across 4 depths, so within a fixed decision type the gold is constant and attention-to-gold separates

near-perfectly; we do not lean on these uncalibrated numbers. The diagnostic and the causal control both transfer to a real multi-step context (≥ 3B saturate; the pre-decision last-token margin is weak, 0.48/0.73, so multi-step HAA is post-hoc). The gold-free selector, however, does not transfer— gating on the near-chance multi-step last-token margin hurts (−27 pts)—a sharp, honest open problem. Per-model numbers: Appendix B, Table 4.

10

Reduction to practice: a gold-free selector

Section 8 showed the gold-free confidence-gated selector working on two real benchmarks; here we dissect why the gate is needed, on the confusable benchmark. A panel-vote (boost each top-3 candidate, pick the highest post-boost self-prob) hurts by 32 points (the symmetric boost lets a wrong-butconfident distractor win), so we choose with the diagnostic, not the intervention: for each candidate c in the baseline top-3 we take its last-prompt-token attention and pick arg maxc (one extra forward, no gold). Raw S2 helps on 0.5/1.5/8B (+10.6/+2.5/+19.9) but regresses on 3B (−4.7), so we gate: override the baseline top-1 with the S2 pick only when the output is uncertain (top-1-vs-top-2 logprob margin below the per-model median; no outcome tuning). All four models then improve by +8.5 to +12.6 points (paired bootstrap CIs exclude zero; exact McNemar p ≤ 3.5 × 10−9 ), positive for every fixed τ ∈ [0.25, 3.0]—a broad plateau, not a knife-edge (Table 14)—and held-out dev/test selection of τ matches the in-sample gain on both the confusable benchmark (+11.2 pp) and real BFCL (+12.1 pp, positive in every split; App. A). Per-candidate attention argmax is not a new primitive (ICR Chen, Jiménez Gutiérrez, and Su 2025 de-biases it to re-rank documents; MindGuard Wang et al. 2025 reads cross-tool attention); our contribution is narrower—the tool-selection instantiation, the link to the causal knob, and gating on the output-logprob margin. Limitations. The fix is scoped to function selection, not arguments: the tool-definition segment carries no argument content (§8). The selector does not transfer to multi-turn (pre-decision signal near-chance, §9), and interventions that predict failures can still hurt in deployment (Vasudev et al. 2026)—our confidence gate bounds but does not eliminate the damage (13/9% residual), so per-call abstention stays open. The 4D-mask intervention needs a model that honors a passed mask (Phi-3 out of scope) and white-box per-layer attention (closed frontier models out of reach); causality of raw attention is contested (Jain and Wallace 2019; Wiegreffe and Pinter 2019), hence our headline is interventional. The correlational runs are single-seed (selector/causal are multiseed with bootstrap CIs) and the τ -bench AUROCs resample near-duplicate rows; a full agent loop is future work. Conclusion. The model usually looks at the right tool and still mis-picks it: on real BFCL failures the gold is the mostattended segment 80% of the time (vs. 21% chance), so the failure is at the readout, not the harness, and two levers in different representations recover the same failures (Jaccard ≈ 0.87). We concede that readout-bottleneck thesis to concurrent activation-space work (Wu et al. 2026; Wang et al.

2026) and contribute its attention-segment account—the perceptual “attended-not-picked” evidence, the attention-logit lever, and a training-free gold-free selector (+11.9/+14.9 pp). Looking is not picking; the gap is at the readout, and attention is where you can see it.

References Abdin, M.; Aneja, J.; Awadalla, H.; Awadallah, A.; et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219. Abnar, S.; and Zuidema, W. 2020. Quantifying Attention Flow in Transformers. In Proceedings of the Association for Computational Linguistics (ACL), 4190–4197. Achtibat, R.; Hatefi, S. M. V.; Dreyer, M.; Jain, A.; Wiegand, T.; Lapuschkin, S.; and Samek, W. 2024. AttnLRP: AttentionAware Layer-Wise Relevance Propagation for Transformers. In International Conference on Machine Learning (ICML). Chen, H.; Zhuang, S.; Yao, Z.; Zuccon, G.; and Leelanupab, T. 2026. Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking. arXiv preprint arXiv:2602.22591. SIGIR 2026. Chen, S.; Jiménez Gutiérrez, B.; and Su, Y. 2025. Attention in Large Language Models Yields Efficient Zero-Shot ReRankers. In International Conference on Learning Representations (ICLR). ArXiv:2410.02642; In-Context Re-ranking (ICR). Chen, Y.; Lv, A.; Lin, T.-E.; Chen, C.; Wu, Y.; Huang, F.; Li, Y.; and Yan, R. 2024. Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). ArXiv:2312.04455; “Attention Buckets”. Cheng, Y.; Fan, C.; JafariRaviz, M.; Rezaei, K.; and Feizi, S. 2026. Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use. arXiv preprint arXiv:2605.14038. Cohen-Wang, B.; Shah, H.; Georgiev, K.; and Madry, A. 2024. ContextCite: Attributing Model Generation to Context. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2409.00729. Dubey, A.; Jauhri, A.; Pandey, A.; et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Guardieiro, V.; Khare, A.; Stein, A.; and Wong, E. 2025. Instruction Following by Principled Boosting Attention of Large Language Models. ArXiv:2506.13734; v3 (2026-0326) uses an additive (not multiplicative) attention bias; cited for the additive attention-steering mechanism (InstABoost), arXiv:2506.13734. Healy, K.; Srinivasan, B.; Madathil, V.; and Wu, J. 2026. Internal Representations as Indicators of Hallucinations in Agent Tool Selection. ArXiv:2601.05214, arXiv:2601.05214. Hsieh, C.-Y.; Chuang, Y.-S.; Li, C.-L.; Wang, Z.; Le, L. T.; Kumar, A.; Glass, J.; Ratner, A.; Lee, C.-Y.; Krishna, R.; and Pfister, T. 2024. Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. In Findings of the Association for Computational Linguistics (ACL Findings).

Jain, S.; and Wallace, B. C. 2019. Attention is not Explanation. In Proceedings of NAACL-HLT, 3543–3556. Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. Lin, H.; Lv, A.; Chen, Y.; Zhu, C.; Song, Y.; Zhu, H.; and Yan, R. 2024. Mixture of In-Context Experts Enhance LLMs’ Long Context Awareness. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2406.19598; MoICE, successor to Attention Buckets. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2024a. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics (TACL), 12: 157–173. Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. 2024b. AgentBench: Evaluating LLMs as Agents. In International Conference on Learning Representations (ICLR). McKenzie, A.; Pawar, U.; Blandfort, P.; Bankes, W.; Krueger, D.; Lubana, E. S.; and Krasheninnikov, D. 2025. Detecting High-Stakes Interactions with Activation Probes. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2506.10805; synthetic-trained activation probes transfer robustly to real OOD data (mean AUROC ¿0.91). Noel, V. 2026. Spectral Guardrails for Agents in the Wild: Detecting Tool-Use Hallucinations via Attention Topology. arXiv preprint arXiv:2602.08082. Olsson, C.; Elhage, N.; Nanda, N.; Joseph, N.; DasSarma, N.; Henighan, T.; Mann, B.; Askell, A.; Bai, Y.; Chen, A.; et al. 2022. In-context Learning and Induction Heads. Transformer Circuits Thread. Patil, S. G.; Mao, H.; Yan, F.; Ji, C. C.-J.; Suresh, V.; Stoica, I.; and Gonzalez, J. E. 2025. The Berkeley Function-Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of PMLR. Prabhakar, A.; Liu, Z.; Zhu, M.; Zhang, J.; Awalgaonkar, T.; Wang, S.; Liu, Z.; Chen, H.; Hoang, T.; Niebles, J. C.; Heinecke, S.; Yao, W.; Wang, H.; Savarese, S.; and Xiong, C. 2025. APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay. ArXiv:2504.03601; xLAM-2-fc-r model family, arXiv:2504.03601. Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. In International Conference on Learning Representations (ICLR). Qwen Team. 2025. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. Repantis, V.; Gawde, A.; Singh, H.; and Blackwell II, J. 2026. How Many Tools Should an LLM Agent See? A ChanceCorrected Answer. arXiv preprint arXiv:2605.24660.

Sadani, A.; and Kumar, D. 2026. Tool Attention Is All You Need: Dynamic Tool Gating and Lazy Schema Loading for Eliminating the MCP/Tools Tax in Scalable Agentic Workflows. ArXiv:2604.21816; embedding-space ISO proxy for expected attention, simulation-based evaluation, arXiv:2604.21816. Sun, C.-E.; Liu, L.; Yan, G.; Wang, Z.; and Weng, T.-W. 2026. LLM Agents Already Know When to Call Tools—Even Without Reasoning. ArXiv:2605.09252, arXiv:2605.09252. Tatsat, H.; and Shater, A. 2026. Beyond the Black Box: Interpretability of Agentic AI Tool Use. ArXiv:2605.06890, arXiv:2605.06890. Todd, E.; Li, M. L.; Sharma, A. S.; Mueller, A.; Wallace, B. C.; and Bau, D. 2024. Function Vectors in Large Language Models. In International Conference on Learning Representations (ICLR). Vasudev, R.; Russak, M.; Bikel, D.; and Alshikh, W. 2026. Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention. ArXiv:2602.03338, arXiv:2602.03338. Venkateswaran, P.; and Contractor, D. 2026. Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 3752–3770. ArXiv:2505.12025. Wang, Y.; Zhou, R.; Fu, R.; Cao, S.; Zeng, H.; Lu, J.; Fan, S.; Zhao, J.; and Pan, L. 2026. ASA: TrainingFree Representation Engineering for Tool-Calling Agents. ArXiv:2602.04935, arXiv:2602.04935. Wang, Z.; Du, H.; Shi, G.; Zhang, J.; Cheng, H.; Yao, Y.; Guo, K.; and Li, X.-Y. 2025. MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning. ArXiv:2508.20412, arXiv:2508.20412. Wiegreffe, S.; and Pinter, Y. 2019. Attention is not not Explanation. In Proceedings of EMNLP-IJCNLP, 11–20. Wu, M.; Zhu, T.; Han, H.; Tan, C.; Zhang, X.; and Chen, W. 2024. Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark. In CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC), 372–384. ArXiv:2405.08355. Wu, W.; Wang, Y.; Xiao, G.; Peng, H.; and Fu, Y. 2025. Retrieval Head Mechanistically Explains Long-Context Factuality. In International Conference on Learning Representations (ICLR). Wu, Z.; Wang, Z.; Cho, S.; Yang, Y.; Koshiyama, A.; Bulathwela, S.; and Perez-Ortiz, M. 2026. Tool Calling is Linearly Readable and Steerable in Language Models. ArXiv:2605.07990, arXiv:2605.07990. Xiao, G.; Tian, Y.; Chen, B.; Han, S.; and Lewis, M. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations (ICLR). Yan, F.; Mao, H.; Ji, C. C.-J.; Zhang, T.; Patil, S. G.; Stoica, I.; and Gonzalez, J. E. 2024. Berkeley Function-Calling Leaderboard (BFCL). https://gorilla.cs.berkeley.edu/leaderboard. html.

Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; et al. 2025. Qwen3 Technical Report. ArXiv:2505.09388, arXiv:2505.09388. Yao, S.; Shinn, N.; Razavi, P.; and Narasimhan, K. 2024. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In arXiv preprint arXiv:2406.12045. Yin, K.; and Steinhardt, J. 2025. Which Attention Heads Matter for In-Context Learning? In International Conference on Machine Learning (ICML). Zhang, H.; Tian, Y.; and Zhang, T. 2026. Enhancing Multi-Agent Communication through Attention Steering with Context Relevance. ArXiv:2605.30136; Agent-Radar, arXiv:2605.30136. Zhang, Q.; Singh, C.; Liu, L.; Liu, X.; Yu, B.; Gao, J.; and Zhao, T. 2024. Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs (PASTA). In International Conference on Learning Representations (ICLR). ArXiv:2311.02262. Zheng, Y.; Zhang, Z.; Ma, C.; Yu, Y.; Zhu, J.; Wu, Y.; Xu, T.; Dong, B.; Zhu, H.; Huang, R.; and Yu, G. 2026. SkillRouter: Skill Routing for LLM Agents at Scale. arXiv preprint arXiv:2603.22455.

A

Held-out selection of τ and δ

Both data-chosen quantities on BFCL—the gate threshold τ (per-model median output-logprob margin) for the gold-free selector, and the oracle boost strength δ—are validated out-of-sample by repeated random subsampling: 200 random 50/50 task splits, each choosing the quantity on the dev half and scoring on the disjoint test half (code/analyze heldout split.py). Table 3 reports the held-out test effects. The selector’s held-out pooled gain (+12.1 pp) is statistically indistinguishable from the insample +11.9 pp and positive in 100% of splits, so the headline is not an artifact of in-sample thresholding. For the oracle boost, dev selects δ=+8 for the two Qwen models in 99% of splits (Llama prefers δ=+12, but δ=+8 still recovers 97% on its test half), confirming δ=+8 as a data-driven rather than hand-tuned operating point. Table 3: Held-out dev/test split (BFCL live multiple, 200 random 50/50 resamples). Selector: τ = dev-half permodel median margin, gold-free gated S2 scored on test. Oracle: δ ∗ chosen on dev over {4, 8, 12} by net gain, recovery scored on test. Brackets are 2.5–97.5 percentile bands over splits.

Model

Selector held-out ∆ (gold-free, pp)

Qwen2.5-1.5B +7.0 [+3.3, +11.3] Qwen2.5-7B +17.7 [+12.7, +22.7] Llama-3.1-8B +11.4 [+7.3, +15.3] Pooled

+12.1 [+9.5, +15.1]

Oracle held-out recovery (δ ∗ on dev) 89.3% 95.6% 97.2% —

B

Supporting tables

The trained-probe comparison (Section 5), the correlational AUROC table (Section 4), the training-free outputconfidence/lexical baseline table (Section 7), the per-decision compute table (§8), and the per-model τ -bench numbers (§9) are below. Table 4: τ -bench airline multi-step: AUROC of attn margin (gen-query, post-hoc) predicting per-step tool selection. Only the two smaller models have real failure variance; ≥3B saturate (succ. ≥ 0.95) so their AUROC is trivial/noise (the 7B figure is ∼23 failures of noise). Model

succ attn margin AUROC

Qwen2.5-0.5B Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B Llama-3.1-8B

0.55 0.49 1.00 0.95 1.00

0.985 0.998 1.000 0.387∗ —

pgold

status

0.653 0.970 0.998 0.236 —

inform. inform. ceiling noise 0 fails

Table 5: Trained activation probe vs. training-free attn margin (AUROC, confusable benchmark, n=480/model, 5-fold CV). “gen”/“last” are the generated-answer and decision-token query sets for attn margin. SHUF = shuffledlabel leakage control; PCA -32 = 32-dim probe (d≪n). Columns: attn margin (gen/last query sets), probe(hL ), probe+attn, and SHUF/PCA controls. †Qwen-3B/Llama-8B are near-ceiling (succ 0.98/1.00, ∼10/2 failures), so their AUROCs are noise (SHUF ̸= 0.5) and not interpretable—we read only the two models with real failure variance. Model

succ attn margin g/l

probe

+attn SHUF/PCA

Qwen2.5-0.5B 0.71 0.922 / 0.697 Qwen2.5-1.5B 0.49 0.573 / 0.416

0.970 0.979

0.980 0.49 / 0.90 0.977 0.51 / 0.95

Qwen2.5-3B† 0.98 0.801 / 0.896 0.83 ±.34 0.84 0.40 / 0.91 Llama-3.1-8B† 1.00 1.000 / 0.554 1.000 1.000 0.47 / 1.00

Table 6: Per-decision compute on real BFCL prompts (median over 40 tasks, 1×H100; median seqlen 228 tokens). “S2 sel” is the gold-free deployable path (one HAA forward + 3 candidate scorings); “oracle” is one biased forward. The gold-free selector adds only ∼1.2× over baseline decoding. Model

gen (ms)

Qwen2.5-1.5B 225.8 Qwen2.5-7B 282.6

C

HAA S2 sel oracle mem (ms) (ms) (ms) (GB) 14.3 15.1

55.7 58.4

41.3 42.4

S2 ovh

4.74 1.25× 18.6 1.21×

Scaling table

Generality and scaling (Section 7): causal Spearman ρ (boost gold; boost distractor) and confusable diagnostic AUROC across 11 models / 6 families / 0.5–32B (including the 2025generation Qwen3-8B and the tool-finetuned Llama-xLAM2-8B).

Table 7: Free-generation δ-sweep on BFCL live multiple (pooled, 3 models / 900 tasks; §8). name ok = called function name = gold; ast ok adds argument matching (real-metric proxy). “rec” = baseline failures recovered, “dmg” = baseline successes broken. The constrained sweet-spot δ=+8 over-boosts free generation; a calibrated δ=+2 recovers names (rec 0.67/dmg 0.04) but gives net-zero argument recovery (rec 0.13/dmg 0.13). Metric

base

δ=+1 δ=+2 δ=+4 δ=+8

acc 0.929 name ok rec — dmg —

0.95 0.56 0.02

0.94 0.67 0.04

0.61 0.48 0.38

0.20 0.25 0.80

acc 0.630 rec — dmg —

0.61 0.09 0.08

0.59 0.13 0.13

0.25 0.07 0.65

0.07 0.00 0.89

ast ok

D

Reproducibility checklist (AAAI)

• Code will be released upon publication. • Data. The controlled benchmark is fully deterministic from a seed; the codebase regenerates it on every worker so no data is shipped. The confusable variant uses the same generator with a 3-tool nearsynonym family taxonomy embedded in the code. BFCL live multiple is public. Seal-Tools (Wu et al. 2024) test splits (test in domain + test out domain, single-call subset, K=5 candidates) are public on the authors’ GitHub. • Seeds. Trigger v1/v2 are multi-seed (3 on Qwen2.5-0.5B, 2 on Qwen2.5-1.5B). Correlational/causal/BFCL/generality runs are single-seed; we report bootstrap CIs over tasks/conditions. • Hardware. 1×NVIDIA H100 per job; per-job wall-clock ∼ 15–120 minutes depending on model size and δ sweep. Per-decision inference cost (gold-free selector ∼1.2× baseline decoding; peak memory 4.74/18.6 GB at 1.5/7B) is in Table 6 (§8). • Statistical reporting. Reported AUROCs/accuracies carry 2000-resample bootstrap 95% CIs; the paired selector-comparison CIs use 10,000 resamples (with exact two-sided McNemar). • Models. Exact HF identifiers: NousResearch/ Meta-Llama-3.1-8B-Instruct (a public reupload used for gated-access reasons; weights identical to the official meta-llama repo), Qwen/Qwen2.5{0.5B,1.5B,3B,7B,14B,32B}-Instruct, Qwen/Qwen2.5-Coder-7B-Instruct, 01-ai/Yi-1.5-9B-Chat, HuggingFaceH4/ zephyr-7b-beta (Mistral), microsoft/ Phi-3.5-mini-instruct, Qwen/Qwen3-8B (enable thinking=False), Salesforce/ Llama-xLAM-2-8b-fc-r. • Software. transformers 5.10.2, attn implementation="eager", dtype bf16, output attentions=True; greedy generation. • Hyperparameters. Additive bias δ ∈

Table 8: End-to-end AST evaluation of the gated S2 pipeline (§8). “free” = plain free-generation baseline (name parse / name + argument AST); “forced” arms teacher-force the arm’s chosen name( and free-generate arguments. “gold ceiling” forces the known-gold name—the selector-independent upper bound of the constrained protocol. The gated pipeline beats its own forced-base arm everywhere (+5.2 pts pooled BFCL, +8.7 Seal-Tools) but the constrained protocol’s ceiling sits below free-gen AST on every BFCL model—the protocol, not the selector, fails the real metric. †xLAM-2-8B free-generates in its native JSON tool-call format, which our text parser undercounts; its free-gen columns are not comparable and it is excluded from pooled rows. Benchmark

free name free AST forced-base AST gated AST gold ceiling

Model

Qwen2.5-1.5B Qwen2.5-7B Llama-3.1-8B BFCL live multiple Qwen3-8B xLAM-2-8B† Pooled (3 main) Qwen2.5-1.5B Qwen2.5-7B Llama-3.1-8B Pooled

Seal-Tools

0.893 0.943 0.940 0.913 0.347† 0.926

0.277 0.667 0.773 0.717 0.300† 0.572

0.187 0.353 0.447 0.417 0.427 0.329

0.213 0.410 0.520 0.460 0.507 0.381

0.237 0.443 0.547 0.507 0.570 0.409

0.891 0.929 0.942 0.921

0.194 0.701 0.844 0.579

0.204 0.364 0.476 0.348

0.252 0.469 0.585 0.435

0.282 0.537 0.619 0.480

Table 9: Advisory-mode AST (§8): the gold-free S2 recommendation delivered as a soft system-prompt hint with free generation, vs. free generation and vs. hard name-forcing, on BFCL live multiple (300 tasks/model). Forcing the name inverts the AST effect (−18.3 pts pooled); the soft advisory lifts it back to within noise of free generation (+1.8 pts, n.s. pooled; +6.0 on the weakest model). “gated” intervenes only below the per-model median name margin (more conservative; soft advisory rarely damages, so the gate is optional). Model

free AST force-S2 (hard) advise-all (soft)

gated

0.210 (−6.7) 0.427 (−24.0) 0.530 (−24.3)

0.337 (+6.0) 0.653 (−1.3) 0.780 (+0.7)

0.310 0.650 0.777

0.389 (−18.3)

0.590 (+1.8)

0.579 (+0.7)

Qwen2.5-1.5B Qwen2.5-7B Llama-3.1-8B

0.277 0.667 0.773

Pooled

0.572

{−12, −8, −4, 0, +4, +8, +12} on a 4D additive attention mask, applied to every layer and head; benchmark K=6 tools, Ntasks =150 (correlational) / 80 (causal); selector TOPK=3 over baseline-ranked candidates; rollout uses  = 12 A + 21 I row-normalized; bootstrap nboot =2000. • Per-model rows (pooled): correlational 3600; causal 13,440 (4 models) +12,376 (5 generality models) +6,720 (2 2025-era models); BFCL correlational 900, BFCL recovery 900, BFCL non-attention baselines 900, BFCL gold-free selector 900 (each 3 models ×300); end-to-end AST pipeline 1500 (5 models ×300); Seal-Tools pipeline 882 (3 × 294); de-circularization 900; baseline comparison 960; trigger v1 1800; selector/gated-selector 3240 (4 models: 1080+720+720+720; S2 and the gate are re-computed on the same row set). The confidence gate adds no GPU compute (it reuses the logged constraineddecoding margin). Paired statistics for the selector comparisons (task-level paired bootstrap, 10,000 resamples; exact two-sided McNemar) are recomputed offline from the per-task logs.

Table 10: AUROC of the gen-query attn margin predicting per-task tool-selection success (n=900/model, 3600 pooled; bootstrap 95% CIs). These are generated-answer-token query positions and are therefore post-hoc attribution; the matched pre-decision last-token margin (right column) is much weaker. †Near-ceiling cells (Llama-8B, 2 failures; Coder-7B, 3 failures) are not interpretable and not bolded; with so few minority-class events the NaN-dropping bootstrap silently discards resamples and yields optimistically tight CIs. Model

Success attn margin AUROC [95% CI] Rollout / Last-tok †

Llama-3.1-8B Qwen2.5-3B Qwen2.5-7B Qwen2.5-Coder-7B†

0.998 0.959 0.954 0.997

0.974 [0.940, 1.000] 0.799 [0.721, 0.868] 0.679 [0.566, 0.784] 1.000 [1.000, 1.000]

0.624 / 0.604 0.596 / 0.718 0.501 / 0.530 0.663 / 0.985

Pooled

0.977

0.751 [0.686, 0.817]

0.446 / 0.603

Table 11: Baseline comparison on the confusable benchmark, n=480/model, 960 pooled. Signal (type) attn margin (INT) pgold (OUT) pmargin12 (OUT) f tmaxprob (OUT) f tnegent (OUT) lexmargin (LEX) Success rate

Qwen2.5-0.5B

Qwen2.5-1.5B

Pooled

0.921 [0.891, 0.948] 0.833 [0.795, 0.871] 0.560 [0.507, 0.614] 0.728 [0.671, 0.780] 0.728 [0.671, 0.779] 0.646 [0.588, 0.703]

0.597 [0.543, 0.650] 0.621 [0.568, 0.673] 0.565 [0.513, 0.615] 0.457 [0.406, 0.509] 0.456 [0.405, 0.509] 0.486 [0.434, 0.533]

0.691 [0.653, 0.729] 0.656 [0.617, 0.695] 0.530 [0.493, 0.568] 0.619 [0.583, 0.655] 0.620 [0.585, 0.656] 0.558 [0.519, 0.594]

0.713

0.573

0.643

Table 12: Causal dose-response Spearman ρ and confusable AUROC vs. model size/family. The 11-model causal subset carries a ρ value; two extra rows (Qwen2.5-0.5B/1.5B) report only the confusable AUROC and show “—” for ρ, so the table has 13 rows. Model (size)

ρgold

ρdist

Confus. AUROC

Qwen2.5-0.5B Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B Qwen2.5-Coder-7B Llama-3.1-8B Yi-1.5-9B Zephyr-7B (Mistral) Qwen2.5-14B Qwen2.5-32B Qwen3-8B xLAM-2-8B (Llama) Phi-3.5-mini‡

— — +0.964 +0.857 +0.964 +0.964 +0.929 +0.893 +0.607 +0.643 +0.821 +0.964 flat‡

— — −0.964 −0.929 −0.893 −1.000 −0.893 −0.464 −0.821 −0.929 −1.000 −0.893 flat‡

0.922 0.553 0.793 0.628 — — 0.794 0.738 0.926 — — — N/A

Table 13: What HAA adds over the two closest concurrent works (both in our bibliography). All three locate the bottleneck at the readout; they differ in representation and in what each shows. “part.” = partial (ASA argues prompts are brittle vs. steering, but runs no input-fix recovery contrast). toollin. ASA HAA (2026) (2026) (ours) Representation space Reads raw attention Segment-grounded localizer “Attended-not-picked” (80%) Input- vs. readout-fix contrast Repr.-invariance (J≈.87) Bidirectional dose-response Gold-free selector Real BFCL evaluation

resid. — — — — — — ✓ ✓

resid. — — — part. — — — —

attn. ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Table 14: Gold-free selector on the confusable benchmark, four models (multi-seed; paired bootstrap CIs and McNemar in text). Raw S2 regresses on 3B; confidence-gated S2 improves all four by +8.5 to +12.6 (§10).

‡

Phi-3.5 “flat” is a methodological artifact—it does not honor a passed 4D additive mask (§7)—not a scientific negative about attention causality.

Model

n

Baseline

raw S2 (∆)

0.5B 1.5B 3B 8B

1080 720 720 720

0.479 0.804 0.769 0.772

0.584 (+10.6) 0.569 (+9.0) 0.829 (+2.5) 0.889 (+8.5) 0.722 (−4.7) 0.879 (+11.0) 0.971 (+19.9) 0.899 (+12.6)

gated S2 (∆)

Table 15: Segment-attribution validation on the 198 real BFCL live multiple baseline failures (per-candidate last-prompt-token HAA; zero-GPU recompute from the selector logs). “argmax = gold” = how often per-candidate at1 tention localizes the correct tool; “chance” = K ; “under” = how often HAA(gold) < HAA(picked) (§8). Model

fails argmax = gold chance under

Qwen2.5-1.5B Qwen2.5-7B Llama-3.1-8B

56 83 59

0.71 0.86 0.80

0.24 0.20 0.20

0.16 0.07 0.08

Pooled

198

0.80

0.21

0.10