AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

Published as a conference paper at COLM 2026

AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs Yiderigun Borjigin1 1 Saarland University

Alexander Hermann2

Christian Cyron2,3

2 Hamburg University of Technology

Roland Aydin1,4

3 Helmholtz-Zentrum Hereon

4 German Research Centre for Artificial Intelligence (DFKI)

arXiv:2608.14320v1 [cs.AI] 14 Aug 2026

Abstract The anchoring effect is a cognitive bias in which an initial reference value shifts a later judgment toward itself. This effect is well established in human judgment and decision-making, and recent work suggests that large language models (LLMs) exhibit similar behavior. However, existing work on anchoring in LLMs typically evaluates only a narrow set of anchor pathways and rarely distinguishes irrelevant from plausible anchors. We introduce A NCHOR B ENCH, a benchmark for the anchoring effect in LLMs that evaluates multiple anchor pathways under an explicit anchor relevance axis. Across fourteen models, including ten open-weight models and four frontier API models, and a large set of controlled prompts, we find that (1) anchoring is strongly pathway-dependent, (2) plausible anchors usually induce larger shifts than irrelevant ones when introduced through stronger pathways, (3) anchor influence generally weakens as the anchor moves farther from the evidence-supported answer, most clearly on External and RAG, and (4) high task accuracy on the anchor-free control condition (Acc10 : answers within 10 points of gold) does not guarantee robustness: even frontier API models above 95% control accuracy remain susceptible to plausible anchors.

1

Introduction

Large language models (LLMs) are moving beyond text generation into decision-support tasks that require quantitative or evidence-based judgment, including medical question answering (Singhal et al., 2025) and time-series forecasting (Jin et al., 2024; Gruver et al., 2023). In these settings, average accuracy is not enough. A model may give a reasonable answer but still be influenced by a value that appears in the surrounding context, such as an earlier guess in the conversation, a value in a provided example, a retrieved document, or a tool output. This kind of error is especially concerning in high-stakes areas such as medicine. We study this failure mode through the lens of anchoring: the tendency for judgments to shift toward a reference value, as introduced by Tversky & Kahneman (1974). Recent work shows that LLMs exhibit broader human-like cognitive biases across reasoning and decision-making (Jones & Steinhardt, 2022; Hagendorff et al., 2023; Echterhoff et al., 2024; Itzhak et al., 2024), and more directly that anchoring appears in LLMs (Nguyen, 2024; Lou & Sun, 2025; Huang et al., 2025; Takenami et al., 2025). However, the existing literature still leaves two important gaps. First, most prior studies test only one or two anchor pathways, usually prompt-embedded anchors. They rarely compare multiple realistic anchor pathways side by side. Second, they rarely distinguish clearly between unjustified shifts toward irrelevant anchors and potentially reasonable shifts toward plausible anchors. We address these gaps with A NCHOR B ENCH,1 a controlled diagnostic for the anchoring effect in LLMs. As shown in Figure 1, the benchmark covers five anchor pathways that correspond to standard ways context reaches a deployed model: External (the prompt), History (conversation), In-Context Learning (ICL, demonstrations), Retrieval-Augmented Generation (RAG, retrieved documents), and Tool (tool outputs). These pathways also have 1 Code and data: https://github.com/Ydrg9989/AnchorBench.

1

Published as a conference paper at COLM 2026

Anchor Pathway

External

Anchor Relevance

Self-generated

× Selective accessibility Value construction

Anchoring is pathway-dependent. Plausible anchors induce stronger shifts. Influence weakens at larger anchor offsets.

Accuracy does not imply robustness.

Figure 1: A NCHOR B ENCH is a unified benchmark for studying anchoring in LLMs, (a) inspired by four human anchoring paradigms, (b) A NCHOR B ENCH varies anchor pathway and anchor relevance, (c) evaluates them in a shared numeric judgment setup with standardized inference and metrics, (d) compares anchoring behavior across fourteen LLMs. loose parallels in the human anchoring literature (Tversky & Kahneman, 1974; Epley & Gilovich, 2001; Mussweiler & Strack, 1999; Chapman & Johnson, 1999), which we use as informal motivation rather than strict one-to-one equivalences. It also introduces an explicit anchor relevance axis with control, irrelevant, and plausible conditions. Because every item contains structured numeric evidence and a deterministic gold answer, the benchmark measures not only whether outputs shift, but also whether the shift is justified. Across the experiments, we find that (1) anchoring is strongly pathway-dependent, (2) plausible anchors usually produce larger shifts than irrelevant anchors when the pathway is strong, (3) anchor influence weakens as the anchor moves farther from the underlying evidence, and (4) accuracy and robustness come apart: even frontier API models with very high control-condition accuracy (Acc10 : answers within 10 points of gold) remain measurably susceptible to anchoring, though at smaller magnitudes than open-weight models. Our contributions are as follows: 1. A multi-pathway diagnostic benchmark for LLMs. We design five realistic anchor delivery pathways that mirror how context reaches a deployed model, loosely informed by classic human anchoring theory, and evaluate them under a shared framework. 2. A relevance-aware evaluation design. We separate control, irrelevant, and plausible anchors. This lets us distinguish unambiguous bias (any shift toward an irrelevant anchor, which carries no task-relevant information) from sensitivity to plausible anchors, where a bounded shift can be consistent with rational evidence integration but a sufficiently large shift cannot. 3. A broad empirical comparison across model families and access regimes. We evaluate fourteen models, including ten open-weight models and four frontier API models, across five suites and 9,000 condition-controlled prompts per model. 2

Published as a conference paper at COLM 2026

2

Related work

Human anchoring theory. Anchoring is a classic finding in judgment and decision-making: estimates are often drawn toward an initial value, even when that value is arbitrary or only weakly informative (Tversky & Kahneman, 1974). No single mechanism explains it. For externally provided anchors, selective-accessibility accounts argue that people test anchorconsistent hypotheses and retrieve anchor-consistent knowledge (Strack & Mussweiler, 1997; Mussweiler & Strack, 1999); for self-generated anchors, anchoring-and-adjustment accounts emphasize insufficient adjustment from an internal starting point (Epley & Gilovich, 2001); and value-construction accounts tie anchoring to how values are activated and assembled during judgment (Chapman & Johnson, 1999). Anchor effectiveness also depends on plausibility and extremity (Wegener et al., 2001; Furnham & Boo, 2011). The human effect is sizable and relevance-dependent: Teovanović (2019) reports Cohen’s d from 0.14 to 1.00 across 24 items, and Li et al. (2021) finds stronger anchoring for related than random anchors—a reference scale we revisit in Appendix A.17.13. These distinctions motivate our design: some pathways correspond to classic external or self-generated anchoring, while others operationalize selective-accessibility- or value-construction-style influence through demonstrations, retrieval, and tools. Anchoring effect in LLMs. Anchoring also appears in LLMs. Frontier models shift their estimates toward previously mentioned values, and prompt-based mitigations such as chain-of-thought, reflection, or explicit instructions to ignore the anchor give only limited and inconsistent relief (Nguyen, 2024; Lou & Sun, 2025); the effect persists in multi-turn price negotiation, where reasoning models are less susceptible (Takenami et al., 2025). S YN A NCHORS (Huang et al., 2025) is the closest prior benchmark and is complementary: it varies the magnitude of in-prompt anchors and uses causal tracing, whereas A NCHOR B ENCH fixes the numeric value and varies its framing (irrelevant vs. plausible) and delivery pathway. Where they overlap, our findings agree. Most of this work studies one pathway at a time, which leaves open how External, History, ICL, RAG, and Tool anchoring compare within a unified setup. Cognitive bias in LLMs. Anchoring is one of several human cognitive biases documented in LLMs, which have been used as a lens on systematic model failure (Jones & Steinhardt, 2022; Hagendorff et al., 2023), studied as an emergent effect of instruction tuning (Itzhak et al., 2024), and observed in decision-making and in survey response, where models are unreliable stand-ins for human respondents (Echterhoff et al., 2024; Tjuatja et al., 2024); broader evaluations and surveys catalogue many such biases and their mitigations (Malberg et al., 2025; Sumita et al., 2024). The pattern is especially well documented in LLM-as-a-judge pipelines, where position and related judging biases distort model-based evaluation (Koo et al., 2024; Zheng et al., 2023; Wang et al., 2024; Ye et al., 2025; Wang et al., 2025) and evaluators show anchoring in multi-attribute scoring (Stureborg et al., 2024)—so the effect reaches evaluation pipelines, not only end-user tasks. Context influence beyond anchoring. Numerical anchoring is one instance of a broader sensitivity to non-informative context, alongside sycophancy and persuasion, where models shift toward a user’s view or a confidently framed claim (Sharma et al., 2024; Ranaldi & Pucci, 2025) and toward authoritative-sounding sources (Anagnostidis & Bulian, 2024; Zhou et al., 2023; Nguyen et al., 2025). We see anchoring as narrower: it isolates a single numeric value rather than opinions or arguments, which lets us measure the shift against a deterministic gold answer.

3

Benchmark setup and evaluation design

A NCHOR B ENCH is a controlled diagnostic rather than a sample of organic user queries: every item is synthetic and built so that the gold answer is fixed by the evidence alone, which lets us measure anchoring as a deviation from a known target. It combines two controlled design factors: the anchor pathway, which determines how an anchor reaches

3

Published as a conference paper at COLM 2026

Shared prompt (External suite)

Anchor sentence by condition

A SaaS company is pricing a premium add-on. The following customer research scores (0–100) indicate price tolerance.

Control: No added sentence. Irrelevant (a=23):

Evidence: – Online panel score: 62 – In-store intercept: 58 – Trade-off analysis: 65 – Retention predictor: 54 – Benchmark comparison: 61

Note: This is assessment case #23 in the current evaluation batch. Plausible (a=23): Note: A recent industry report suggested the WTP index might be around 23 .

[anchor sentence inserted here]

The anchor value a is identical in both conditions; only the framing differs. Irrelevant framings use metadata (case IDs, queue numbers), whereas plausible framings cite a prior estimate.

Given these signals, what is your best estimate for the WTP index on a 0–100 scale? Return only a single integer 0–100 on the last line. Gold: y∗ = 60 (mean of evidence).

(a) Shared prompt

(b) Anchor conditions

Figure 2: Example A NCHOR B ENCH item (External suite, pricing domain). All conditions share the same scenario, evidence, question, and answer instruction. Only the anchor sentence differs. Additional examples are in Appendix A.2. the model, and anchor relevance, which determines how the anchor is framed relative to the task. 3.1

Item construction

Each item is a numeric judgment task on a shared 0–100 scale across six domains (pricing, operations, logistics, resource consumption, market adoption, compliance). As illustrated in Figure 2a, the model receives a set of numeric evidence ratings and must return a single integer estimate. The gold answer is the rounded arithmetic mean of the visible ratings:   (1) y∗ = round 1k ∑kj=1 e j , where e1 , . . . , ek are the ratings shown to the model. This makes the task a deterministic aggregation problem: a model that correctly averages the given numbers matches gold exactly. Item difficulty is governed by two parameters. A latent center θ ∼ Unif[30, 70] sets the evidence range; ratings are sampled as e j ∼ N (θ, σ2 ), clipped to [0, 100]. Easy items present all five ratings with low noise (σ =8). Hard items hide two of the five ratings, add a conflicting value, and raise noise (σ =15), leaving fewer and less consistent signals for aggregation. The six core domains are business-oriented, but a pilot adding medical, legal, and consumer domains shows the same UAIpls >UAIirr > 0 pattern (Appendix A.17.5). Provenance of the items. Items come from author-designed templates: scenarios, evidence labels, questions, and anchor framings are drawn from pools generated with LLM assistance and verified by the authors, while the numeric evidence, anchor values, and gold answers are produced by seeded sampling, never by a model. Holding the numeric value and its position fixed while varying only the framing sentence is the matched control the relevance axis needs, and organic queries cannot supply it—at a cost in external validity we state rather than claim away. 3.2

Anchor conditions and relevance axis

Each item is presented under five matched conditions, illustrated in Figure 2b. The control condition presents the task without any anchor. The four anchored conditions cross two relevance levels—irrelevant and plausible—with a low and a high anchor direction each, yielding: control, irrelevant_low/high, and plausible_low/high. 4

Published as a conference paper at COLM 2026

The defining feature of the relevance axis is that an irrelevant and a plausible condition in the same direction use an identical numeric anchor value a; only the surrounding sentence differs. An irrelevant framing presents a as incidental metadata (e.g., “assessment case #a in the current batch”); a plausible framing presents it as a substantive prior estimate (e.g., “a recent industry report suggested around a”). Since the gold answer is determined entirely by the evidence (uninformative about a), any difference between irrelevant and plausible responses isolates the effect of framing, not the anchor value itself. As a concrete example, on one real External item Qwen-7B answers 43 (the evidence mean) in control, still 43 under the irrelevant framing of a=85, but 75 under the plausible framing— a +32 shift that isolates the framing alone. Per-pathway worked examples and further case studies are in Appendices A.2 and A.17.12. Anchor placement. Low and high anchors are placed symmetrically around the evidence center θ: alow = max(0, θ −δ) and ahigh = min(100, θ +δ), with offsets δ ∈ {15, 25, 40}. Varying the offset lets us test how anchor influence changes as the anchor moves farther from the evidence-supported answer (§4.3). The History suite is an exception: its anchor is the model’s own Stage 1 response rather than a designer-specified value (§3.3), so its offset varies by item. Full generation and validation details are in Appendix A.1. These five conditions are each delivered through a distinct anchor pathway, implemented as a benchmark suite (§3.3). 3.3

Benchmark suites

The five suites hold the numeric judgment task fixed and vary only the pathway by which an anchor reaches the model; they are controlled operationalizations rather than end-to-end deployment systems (worked prompts in Appendix A.2). External places an anchor sentence directly in the user prompt. The prompt format is identical across all models and conditions; only the framing sentence varies. History uses a two-stage conversation: Stage 1 shows partial evidence to elicit a response near the target anchor, and Stage 2 reveals the full evidence, so the anchor is the model’s own Stage-1 answer. The single-stage control creates a format asymmetry analyzed in Appendix A.10. ICL has two variants: ICL-metadata (the main benchmark) places anchors only in incidental demonstration metadata (case IDs, batch numbers)—a deliberately weak manipulation; ICL-dist instead uses demonstration answers matching the anchor distribution, a stronger and more realistic few-shot signal (Appendix A.11). RAG embeds the anchor in one document of a frozen three-document mini-corpus. The other two documents contain factual evidence consistent with the gold answer. Prompt format is identical across models. Tool returns the anchor in a tool-call response. Models supporting structured tool messages (Qwen, Llama) receive them natively; others (Gemma, OLMo, APIs) get an equivalent plaintext rendering—a known confound compared within-model in Appendix A.12. Cross-suite comparability. E XTERNAL and RAG have no format confounds and provide the strongest cross-model evidence. H ISTORY and T OOL have format-related confounds (described above); we interpret their magnitudes qualitatively. Each suite contains 360 items (6 domains × 60 items) in five conditions, giving 1,800 prompts per model per suite. All data generation is seed-controlled and deterministic (Appendix A.1). 3.4

Evaluation metrics

All suites share the same metrics, so metric semantics are comparable; recall that History and Tool magnitudes are format-confounded and should be read qualitatively (§3.3). We 5

Published as a conference paper at COLM 2026

measure three dimensions of behavior: task accuracy, anchor susceptibility, and relevance discrimination. Notation. Let yi∗ be the gold answer for item i, yctrl,i the control (no-anchor) response, and yanchor,i,r the response under relevance r ∈ {irr, pls}. Let ai,r be the anchor value, constructed in the prompt for External/ICL/RAG/Tool and equal to the model’s Stage 1 answer for History. (a) Task accuracy (control-only). We measure task accuracy only on control responses, because these reflect task performance without anchoring. We report both mean absolute error and a coarse accuracy-within-tolerance measure: MAEc = n1 ∑i |yctrl,i − yi∗ |,

  Acc10 = n1 ∑i 1 |yctrl,i − yi∗ | ≤ 10 ,

(2)

MAEc is average prediction error on the native 0–100 scale; Acc10 is the fraction of control responses within 10 points (10% of the range) of gold. These capture task competence but do not by themselves measure anchoring. (b) Anchor susceptibility. Unified Anchor Influence (UAI) measures what fraction of the anchor–control gap is closed by the anchored response: UAIi,r =

yanchor,i,r − yctrl,i . ai,r − yctrl,i

(3)

UAI = 0 indicates no shift and UAI = 1 indicates full movement to the anchor. Items where the control response already falls within ε = 3 points of the anchor (| ai,r − yctrl,i | < 3) are excluded, because the near-zero denominator makes the ratio unstable. Exclusion rates are low and conclusions are stable across ε ∈ {1, 3, 5} (Appendix A.6). Because UAI is undefined for excluded items, we also report the denominator-free TowardAnchor Rate (TAR):   TARi,r = 1 (yanchor,i,r − yctrl,i ) · ( ai,r − yctrl,i ) > 0 . (4) TAR > 0.50 indicates a systematic shift toward the anchor. TAR covers all items and is independent of the gap magnitude, so the two metrics together guard against exclusion-rule artifacts. (c) Relevance discrimination. The discrimination gap captures whether a model responds differently to plausible and irrelevant anchors: Disc∆ = UAIpls − UAIirr .

(5)

Disc∆ > 0 indicates greater sensitivity to plausible than irrelevant anchors, and the reverse if negative. It should be read alongside absolute UAI: a model with Disc∆ > 0 but high UAIirr is still strongly susceptible, so positive discrimination alone is not evidence of robustness. 3.5

Experimental setup

We evaluate fourteen instruction-tuned models: ten open-weight models from four families— Llama 3.1/3.2 (Grattafiori et al., 2024), Qwen2.5 (Qwen Team et al., 2025), Gemma 3 (Gemma Team, 2025), and OLMo 2 (OLMo Team et al., 2025), plus four frontier API models from OpenAI (Singh et al., 2025), Anthropic (Anthropic, 2025), Google (Comanici et al., 2025), and xAI (xAI, 2025) (full model panel: Appendix A.1). All models use greedy decoding (temperature 0); answers are extracted by deterministic hierarchical parsing (median parse rate 99.9%). A sampled-decoding check (Appendix A.15) and a prompt-based mitigation probe (Appendix A.16) confirm the main conclusions are stable.

6

Published as a conference paper at COLM 2026

Model

E XTERNAL Acc

Uirr Upls Acc

2.5-1.5B 2.5-3B 2.5-7B

34% 64% 72%

3.2-1B 3.2-3B 3.1-8B

H ISTORY Uirr

ICL Uirr

Uirr

Upls

0.02 0.20 32% 0.29 0.01 0.09 61% −0.08 0.01 0.27 79% −0.06

0.98 59% 0.01 0.00 22% 0.02 0.23 46% 0.49 78% −0.04 −0.04 74% −0.02 0.08 89% 0.37 88% −0.03 −0.01 64% −0.00 0.20 99%

0.26 0.08 0.00

0.82 0.18 0.17

47% 69% 83%

0.13 0.15 49% 0.14 0.29 72% 0.07 0.36 83%

0.14 55% 0.73 81% 0.23 77%

— 0.03 0.31

— 0.21 0.48

3-1B- IT 3-4B- IT

37% 55%

0.02 0.18 39% −0.17 0.20 0.43 56% 0.23

2-13B 2-32B

89% −0.01 0.16 86% −0.00 53% 0.05 0.40 54% −0.30

0.30 0.07 0.03

Upls Acc

T OOL

Uirr Upls Acc

0.48 0.15 0.13

Upls Acc

RAG

0.22 60% 0.10 0.09 — 0.14 72% 0.14 0.22 95% 0.15 85% −0.02 0.12 73%

0.13 46% −0.00 −0.03 37% −0.00 0.04 26% −0.05 −0.03 0.50 39% 0.00 −0.06 48% 0.05 0.01 63% 0.18 0.21 0.32 57% 0.03 0.02 87% −0.00 0.05 71% 0.75 70% −0.02 −0.01 58% 0.01 0.45 72%

0.09 0.01

0.07 0.02

5.4- MINI 99% 0.00 0.14 94% −0.02 0.04 99% −0.00 −0.01 99% 0.01 0.09 99% 0.00 H AIKU 4.5 100% 0.00 0.12 98% −0.02 −0.06 97% −0.00 −0.02 98% 0.00 0.06 98% −0.01 2.5-F LASH 96% 0.02 0.07 95% 0.03 −0.00 99% −0.00 0.00 84% 0.08 0.17 98% 0.03 3- MINI 99% −0.00 0.16 98% 0.10 0.10 99% −0.00 0.00 99% −0.01 0.02 99% −0.01

0.06 0.01 0.06 0.02

Table 1: Task accuracy (Acc, =Acc10 , control-only) and absolute anchor influence on irrelevant (Uirr ) and plausible (Upls ) anchors for fourteen models across five suites. Any positive Uirr is an unjustified shift toward a value that carries no task information; Upls is sensitivity to a plausibly framed value, and the plausible–irrelevant gap Disc∆ = Upls − Uirr is analyzed in §4.2. Anchoring varies strongly by pathway: External and RAG show the broadest positive effects; ICL is near zero; History and Tool vary by model. Bold marks the largest Upls per suite. Caveats: Llama-1B/Tool is undefined (parse rate 0.9%); History uses a single-stage control (Appendix A.10); Tool format varies by model (Appendix A.12). For space, instruction-tuned model names are abbreviated by omitting the “-Instruct” suffix; full model identifiers are listed in Table 3.

4

Empirical findings

4.1

Finding 1: Anchoring is pathway-dependent

Table 1 reports task accuracy (Acc10 , control-only) and absolute anchor influence on irrelevant (UAIirr ) and plausible (UAIpls ) anchors for all fourteen models across five suites. We report both UAI columns rather than only their difference, because any positive UAIirr is unjustified bias while UAIpls is sensitivity to a plausibly framed value. The central finding is that susceptibility depends strongly on the anchor pathway. E XTERNAL and RAG show the broadest positive discrimination. ICL (metadata) is uniformly near zero. H ISTORY and T OOL show substantial effects, but their magnitudes are inflated by format confounds: H ISTORY compares a two-stage anchored protocol against a single-stage control (matched-format analysis reduces Disc∆ by up to 70% for some models; Appendix A.10), and T OOL mixes structured and plaintext tool messages across model families (Appendix A.12). The Tool effect is also provenance-sensitive: a model-elicited call or a noisy JSON envelope barely moves the panel mean but splits individual models (Qwen-7B collapses, Gemma-4B strengthens; Appendix A.17.10), so we read Tool numbers as robust on average but not per model. No model is consistently best or worst across suites, and across the ten open-weight models the range of suite-mean Disc∆ is 0.40 (bootstrap 95% CI [0.20, 0.60]; Table 10). 4.2

Finding 2: Plausible anchors induce stronger shifts

Across the 69 model–suite cells with computable Disc∆ , 55 (80%) show positive discrimination—greater susceptibility to plausible than irrelevant anchors—rising to 48/55 (87%) once ICL, where both UAI values are near zero, is excluded. Wilcoxon signedrank tests confirm the pattern on E XTERNAL, RAG, and T OOL (pBH < 0.01) and H ISTORY (pBH ≈ 0.02), but not ICL (pBH = 0.81, n.s.; Table 9). 7

Published as a conference paper at COLM 2026

External

RAG

ICL Open-weight API

0.4

UAIpls

0.3 0.2 0.1 0.0

15

25 Anchor offset δ

40

15

25 Anchor offset δ

40

15

25 Anchor offset δ

40

Figure 3: Anchor influence attenuates with distance (suites with designer-controlled offsets only). UAIpls decreases monotonically with anchor offset δ on External and RAG; ICL (dotted) stays near zero at all distances. Lines show suite means across models; shaded bands are 95% CIs (±1.96 SE). API models show monotonic attenuation on External; on RAG, they peak at δ=25 before decreasing. History is omitted because its offsets are not designer-controlled (§3.3); Tool is omitted due to format confounds (Appendix A.12). The two relevance levels mean different things: a shift toward an irrelevant anchor is bias by construction, while a shift toward a plausible anchor is only a problem once it exceeds what a rational update could justify (§4.4). Either way, positive Disc∆ is not itself evidence of robustness; low overall susceptibility remains the desirable outcome. ICL: a weak floor, not immunity. The near-zero standard ICL result reflects a deliberately weak manipulation (anchors only in demonstration metadata); a distribution-matching ICL-dist variant raises mean UAIpls from ≈0.03 to 0.16 (Disc∆ > 0 for 8/10 open-weight models), comparable to RAG (Appendix A.11). Beyond the binary axis. Relevance is graded, so we check the two-level axis against a four-point credibility spectrum on E XTERNAL: panel-mean UAI is +0.09 for a placebo framing (the same number given as a document’s age), +0.05 for irrelevant, +0.27 for plausible, and +0.49 for authority. Credibility modulates the shift, but the axis is not clean at the bottom—the placebo floor is above zero and above the irrelevant condition, which no purely rational account predicts (Appendix A.17.2). Applying the same manipulation at matched mild / standard / strong intensity across pathways puts RAG (0.05→0.08→0.18) one step below E XTERNAL (0.25→0.32→0.40) rather than differing in kind, with H ISTORY non-monotone (Appendix A.17.4). 4.3

Finding 3: Anchor influence generally decreases with anchor offset

Figure 3 breaks down UAIpls by anchor offset (δ ∈ {15, 25, 40}) on the suites with designercontrolled offsets. On External and RAG, mean UAIpls decreases monotonically with larger offsets (open-weight External: 0.32 → 0.26 → 0.18; RAG: 0.23 → 0.15 → 0.06), indicating that models shift less proportionally toward more extreme anchors on average, consistent with the human anchoring literature (Chapman & Johnson, 2002). Individual models mostly follow this trend but exceptions exist (e.g., Gemma-1B on External shows the opposite pattern). API models show the same monotonic decrease on External at roughly 2× smaller magnitudes; on RAG, however, three of four API models peak at δ=25 before decreasing, suggesting a non-linear interaction between anchor distance and the RAG retrieval context for high-accuracy models. ICL effects remain near zero at all distances. 4.4

Finding 4: Accuracy does not imply robustness

Task accuracy and anchoring discrimination are only weakly correlated (r = −0.24, bootstrap 95% CI [−0.43, −0.00]; Table 10; Figure 8), and restricting to the format-confound-free suites (External + RAG) gives r = −0.17 [−0.49, 0.16], so the weak association is not an artifact of History/Tool format differences. Higher accuracy comes with slightly lower 8

Published as a conference paper at COLM 2026

UAIpls

Model 3-4B- IT 3.1-8B 2-13B 2.5-7B Rational ceiling (w=1)

UAIirr

k =1

k =2

k =3

k =1

k =2

k =3

0.21 0.68 0.48 0.53

0.20 0.36 0.34 0.34

0.25 0.50 0.29 0.36

0.14 0.25 0.19 0.22

−0.13 −0.03 0.11 0.01

0.13 0.20 0.06 0.02

0.50

0.33

0.25

0

0

0

Table 2: Anchoring under partial evidence on E XTERNAL. Only k of the five ratings are shown while gold remains the full-five mean, so the model is genuinely uncertain. An updater treating the plausible anchor as one extra rating of equal credibility would produce UAIpls = w/(k+w) (last row); for an irrelevant anchor the rational value is 0 at every k. Bold marks cells above the ceiling. Disc∆ , but the relationship explains little variance and does not prevent anchoring: on E X TERNAL OLMo-32B (53% accuracy) shows far higher discrimination (0.35) than OLMo-13B (89%; Disc∆ = 0.17). API models confirm the pattern: all four achieve ≥96% accuracy on E XTERNAL yet show positive Disc∆ (0.05–0.16). Scale does not remove this: an extended panel of five larger models is near-ceiling on accuracy yet still shows positive UAIpls on most suites (Appendix A.17.11). Anchoring degrades prediction quality. Plausible anchors increase MAE by +3.6 (External) and +3.5 (Tool), while irrelevant anchors have near-zero impact (Appendix A.9). Classifying each anchored response by whether it moves error up or down relative to the evidence-only gold answer, error-increasing shifts outnumber error-reducing ones (29% vs. 23% overall), a gap driven by plausible anchors (37% vs. 20%) and largest on E XTERNAL and T OOL (≥22 pp; Appendix A.13). How much anchoring is too much? A positive UAI is hard to interpret on its own, since some movement toward a plausible value can be rational. Treating the anchor as one extra piece of evidence among n=5 ratings gives a rational UAI ceiling of w/(n+w) = 0.167 (w=1); for irrelevant anchors w=0, so any positive UAIirr is bias. Inverting the relation gives the implied weight wimp = n · UAI/(1 − UAI): the credibility a model must be granting the lone anchor for its shift to be rational. Of the 55 non-ICL plausible cells, 16 have a UAIpls CI entirely above 0.167 and 5 imply wimp > 5—one anonymous number outweighing all five displayed ratings combined (Appendix A.17.1). We therefore read 0.167 as a practical-significance threshold, with the caveat that it is a reference and not a target: the desirable behavior on a plausible anchor is not zero movement but movement that stays inside the band. Finally, the answer-only protocol is an upper bound. Chain-ofthought attenuates UAIpls without removing it (External: Gemma-4B 0.37→0.23, Llama-8B 0.39→0.37; Appendix A.17.7), and an explicit averaging rule appended to the prompt collapses the panel mean from 0.33 to 0.06—but a prompt that instead invites the model to weigh sources by credibility brings it back to 0.26 (Appendix A.17.8), which is the regime the decision-support settings of §1 target. 4.5

Stress test: anchoring under partial evidence

The main task shows all five ratings, so the gold answer is fully determined and the model has little genuine uncertainty for an anchor to exploit. To test whether the effect depends on that, we re-render E XTERNAL items showing only k of the five ratings while still scoring against the full-five mean. The anchor now carries real information and the rational ceiling rises with the hidden evidence: an updater treating it as w ratings of equal credibility closes w/(k +w) of the gap, giving UAIpls ceilings of 0.50, 0.33, and 0.25 at k =1, 2, 3 (for w=1). The effect survives this reframing (Table 2). Three of the four models exceed the ceiling at some k: Llama-8B by +0.18 at k =1 and +0.25 at k =3, Qwen-7B at all three levels, and 9

Published as a conference paper at COLM 2026

OLMo-13B at k =2 and k =3. The direction of change is what rational updating predicts, since UAIpls falls as more evidence becomes visible, but the level sits above what that update justifies—bias layered on top of legitimate updating rather than instead of it. Gemma-4B is the useful counterexample, staying at or below the ceiling at every k, so the design does not mechanically force positive UAI. UAIirr , whose rational value is 0 however much evidence is hidden, stays far lower throughout. 4.6

Statistical robustness

Our conclusions survive a battery of robustness checks, tagged below as supports headline or edge-case caveat. Supports headline. About 80% of cells keep the sign of Disc∆ across ε ∈ {1, 3, 5}, so the effect is not manufactured by the exclusion rule (Appendix A.6); bootstrap CIs on every per-suite contrast exclude zero outside ICL (Table 10); harder items—fewer, noisier ratings—anchor more on E XTERNAL and RAG, as expected if the anchor fills an evidence gap (Appendix A.14); stochastic decoding preserves the sign in 5 of 6 cells (Appendix A.15); and a weighted-mean gold moves UAIpls by only +0.02 (Appendix A.17.6). Edge-case caveats. RAG influence survives rank changes and distractors (0.08→0.13) but collapses to 0.03 once documents carry relevance scores, so what matters for realism is the relevance signal, not the anchor’s position (Appendix A.17.9). The flat ICL result is a weak manipulation rather than immunity (Appendix A.11). And T OOL panel means stay within 0.02 under model-elicited calls and noisy payloads while individual models move in opposite directions (Appendix A.17.10). Suite-specific analyses are in Appendix A.4.

5

Conclusion

We introduce A NCHOR B ENCH, a controlled diagnostic for anchoring in LLMs across five delivery pathways. By separating irrelevant from plausible anchors, it tests not just whether models shift, but whether the shift is justified. Across fourteen models, anchoring is not uniform: it depends strongly on how the anchor reaches the model. The clearest effects appear in External and RAG, where plausible anchors induce larger shifts than irrelevant anchors, and influence generally decreases as the anchor offset grows. High task accuracy does not guarantee robustness—strong models are still moved by plausible anchors, and the effect persists once the evidence is genuinely incomplete. Average task performance is therefore not enough to certify reliable LLM judgment: robustness evaluation must test the realistic pathways through which numeric context reaches a model.

Limitations A NCHOR B ENCH studies anchoring in a controlled setting: synthetic deterministicaggregation tasks make comparison clean but do not cover the complexity of real judgment, and some pathways require cautious interpretation—History uses a different interaction structure from the control, Tool is affected by cross-family format differences, and small effects are sensitive to the UAI exclusion rule, so a few cross-suite comparisons are less precise. Prompt-based mitigations weaken but do not remove it (Appendix A.16). Our suites also stop short of a fully agentic loop in which an agent plans its own retrieval and tool calls—a natural next step for testing whether these pathway effects persist.

LLM usage disclosure LLMs assisted with code, LATEX, and draft review, and generated part of the synthetic items under an author-designed pipeline, with all such data verified by the authors. Design, metrics, analyses, and writing are the authors’ own; no LLM generated or altered results. The fourteen models in Table 3 are experimental subjects, not authoring tools.

10

Published as a conference paper at COLM 2026

Reproducibility statement Code and the benchmark data are at https://github.com/Ydrg9989/AnchorBench; the source of this paper accompanies the arXiv version. The benchmark is also on the Hugging Face Hub at https://huggingface.co/datasets/Yiderigun/AnchorBench, covering the five core suites and the partial-evidence variant behind Table 2 — 14,400 prompts, each carrying its gold answer, and its anchor value wherever the anchor is fixed in advance, so UAI is computable from the release alone. H ISTORY is the exception by construction: its anchor is the model’s own Stage-1 estimate, so it exists only once the two-stage protocol has been run. The raw generations (≈520 MB) are distributed separately; results/CHECKSUMS.sha256 in the repository lets a download be checked against the bytes these numbers were computed from.

Ethics statement This paper studies the anchoring effect as a reliability problem in LLMs. Anchoring can shift model outputs in decision-support and other consequential settings. Our benchmark uses synthetic scenarios and no personal, sensitive, or proprietary data, and the study does not involve human subjects. A possible concern is that this work may reveal settings in which models are vulnerable to anchoring. We believe the value of documenting this behavior outweighs that risk, because the goal is to improve evaluation and support mitigation. More broadly, our results show that strong average performance is not evidence of robust behavior.

Acknowledgments Most of this work was carried out while Yiderigun Borjigin was at Helmholtz-Zentrum Hereon. We thank Marius Tacke for his early review and feedback, and Kartik Bali for helpful discussions.

References Sotiris Anagnostidis and Jannis Bulian. How susceptible are llms to influence in prompts?, 2024. URL https://arxiv.org/abs/2408.11865. Anthropic. Introducing claude haiku 4.5, 2025. URL https://www.anthropic.com/news/ claude-haiku-4-5. Gretchen B. Chapman and Eric J. Johnson. Anchoring, activation, and the construction of values. Organizational Behavior and Human Decision Processes, 79(2):115–153, 1999. doi: 10.1006/obhd.1999.2841. Gretchen B. Chapman and Eric J. Johnson. Incorporating the irrelevant: Anchors in judgments of belief and value. In Heuristics and Biases: The Psychology of Intuitive Judgment, pp. 120–138. Cambridge University Press, 2002. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261. Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. Cognitive bias in decision-making with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12640–12653. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-emnlp.739. URL https://aclanthology.org/2024. findings-emnlp.739/. Nicholas Epley and Thomas Gilovich. Putting adjustment back in the anchoring and adjustment heuristic: Differential processing of self-generated and experimenter-provided anchors. Psychological Science, 12(5):391–396, 2001. doi: 10.1111/1467-9280.00372. 11

Published as a conference paper at COLM 2026

Adrian Furnham and Hua Chu Boo. A literature review of the anchoring effect. The Journal of Socio-Economics, 40(1):35–42, 2011. doi: 10.1016/j.socec.2010.10.008. Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786. Aaron Grattafiori et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/ 2407.21783. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nature Computational Science, 3:833–838, 2023. doi: 10.1038/s43588-023-00527-x. Yiming Huang, Biquan Bie, Zuqiu Na, Weilin Ruan, Songxin Lei, Yutao Yue, and Xinlei He. An empirical study of the anchoring effect in llms: Existence, mechanism, and potential mitigations. arXiv preprint arXiv:2505.15392, 2025. doi: 10.48550/arXiv.2505.15392. URL https://arxiv.org/abs/2505.15392. Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias. Transactions of the Association for Computational Linguistics, 12:771–785, 2024. doi: 10.1162/tacl_a_00673. URL https://aclanthology.org/2024.tacl-1.43/. Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-llm: Time series forecasting by reprogramming large language models, 2024. URL https://arxiv.org/ abs/2310.01728. Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 11785–11799, 2022. doi: 10.48550/arXiv.2202.12299. URL https://arxiv.org/abs/ 2202.12299. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 517–545. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-acl.29. URL https: //aclanthology.org/2024.findings-acl.29/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309. 06180. Lunzheng Li, Zacharias Maniadis, and Constantine Sedikides. Anchoring in economics: A meta-analysis of studies on willingness-to-pay and willingness-to-accept. Journal of Behavioral and Experimental Economics, 90:101629, 2021. ISSN 2214-8043. doi: 10.1016/j.socec.2020.101629. URL https://www.sciencedirect.com/science/article/ pii/S2214804320306728. Jiaxu Lou and Yifan Sun. Anchoring bias in large language models: An experimental study. Journal of Computational Social Science, 9(1):11, December 2025. ISSN 2432-2725. doi: 10.1007/s42001-025-00435-2. URL https://doi.org/10.1007/s42001-025-00435-2. Simon Malberg, Roman Poletukhin, Carolin M. Schuster, and Georg Groh. A comprehensive evaluation of cognitive biases in LLMs. In Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities, pp. 578–613. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.nlp4dh-1.50. URL https://aclanthology.org/2025.nlp4dh-1.50/. 12

Published as a conference paper at COLM 2026

Thomas Mussweiler and Fritz Strack. Hypothesis-consistent testing and semantic priming in the anchoring paradigm: A selective accessibility model. Journal of Experimental Social Psychology, 35(2):136–164, 1999. doi: 10.1006/jesp.1998.1364. Jeremy K. Nguyen. Human bias in ai models? anchoring effects and mitigation strategies in large language models. Journal of Behavioral and Experimental Finance, 43:100971, 2024. doi: 10.1016/j.jbef.2024.100971. URL https://www.sciencedirect.com/science/article/ pii/S2214635024000868. Tu Nguyen, Kevin Du, Alexander Miserlis Hoyle, and Ryan Cotterell. How persuasive is your context? In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 32097–32123, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main. 1633. URL https://aclanthology.org/2025.emnlp-main.1633/. OLMo Team, Pete Walsh, Luca Soldaini, et al. OLMo 2: The best fully open language model to date, 2025. URL https://arxiv.org/abs/2501.00656. Qwen Team, An Yang, Baosong Yang, et al. Qwen2.5 technical report, 2025. URL https: //arxiv.org/abs/2412.15115. Leonardo Ranaldi and Giulia Pucci. When large language models contradict humans? large language models’ sycophantic behaviour, 2025. URL https://arxiv.org/abs/2311. 09410. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=tvhaxkMKAn. Aaditya Singh, Adam Fry, et al. Openai gpt-5 system card, 2025. URL https://arxiv.org/ abs/2601.03267. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H. Chen, Nigam H. Shah, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomašev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Toward expert-level medical question answering with large language models. Nature Medicine, 31(3):943–950, March 2025. ISSN 1546-170X. doi: 10.1038/s41591-024-03423-7. URL https://doi.org/10.1038/s41591-024-03423-7. Fritz Strack and Thomas Mussweiler. Explaining the enigmatic anchoring effect: Mechanisms of selective accessibility. Journal of Personality and Social Psychology, 73(3):437–446, 1997. doi: 10.1037/0022-3514.73.3.437. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators, 2024. URL https://arxiv.org/abs/2405.01724. Yasuaki Sumita, Koh Takeuchi, and Hisashi Kashima. Cognitive Biases in Large Language Models: A Survey and Mitigation Experiments, November 2024. URL https://arxiv. org/abs/2412.00323. Yoshiki Takenami, Yin Jou Huang, Yugo Murawaki, and Chenhui Chu. How does cognitive bias affect large language models? a case study on the anchoring effect in price negotiation simulations. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 4481–4498. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025. findings-emnlp.240. URL https://aclanthology.org/2025.findings-emnlp.240/. 13

Published as a conference paper at COLM 2026

Predrag Teovanović. Individual differences in anchoring effect: Evidence for the role of insufficient adjustment. Europe’s Journal of Psychology, 15(1):8–24, 2019. doi: 10.5964/ejop. v15i1.1691. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. Do LLMs exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026, 2024. doi: 10.1162/tacl_a_00685. URL https://aclanthology.org/2024.tacl-1.56/. Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131, 1974. doi: 10.1126/science.185.4157.1124. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440– 9450, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.511. URL https://aclanthology.org/2024.acl-long.511/. Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Xuandong Zhao, Wenxuan Zhang, Dawn Song, and Bingsheng He. Assessing judging bias in large reasoning models: An empirical study. In Second Conference on Language Modeling, 2025. URL https:// openreview.net/forum?id=SlRtFwBdzP. Duane T. Wegener, Richard E. Petty, Brian T. Detweiler-Bedell, and W. Blair G. Jarvis. Implications of attitude change theories for numerical anchoring: Anchor plausibility and the limits of anchor effectiveness. Journal of Experimental Social Psychology, 37(1):62–69, 2001. doi: 10.1006/jesp.2000.1431. xAI. Grok-3, 2025. URL https://x.ai/blog/grok-3. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id= 3GTtZFiajM. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models, 2023. URL https: //arxiv.org/abs/2302.13439.

A

Appendix

A.1

Experimental details

Model panel and access.

Table 3 summarizes all fourteen models in one place.

Data and standardized inference. Each suite contains 360 items with five conditions (1,800 prompts per suite per model). Open-weight models were run on H100 GPUs using local vLLM (Kwon et al., 2023), while API models were queried through OpenRouter. Decoding settings were standardized across suites and models (temperature = 0, max tokens = 512). Suite-specific formatting and parsing. History uses a two-stage conversation where Stage 2 receives the model’s own Stage 1 response as context. Tool prompts use modelappropriate formats (structured tool messages when supported, otherwise equivalent plaintext tool context). Predictions are extracted with one deterministic parser shared across all conditions. 14

Published as a conference paper at COLM 2026

Model card Qwen/Qwen2.5-1.5B-Instruct Qwen/Qwen2.5-3B-Instruct Qwen/Qwen2.5-7B-Instruct meta-llama/Llama-3.2-1B-Instruct meta-llama/Llama-3.2-3B-Instruct meta-llama/Llama-3.1-8B-Instruct google/gemma-3-1b-it google/gemma-3-4b-it allenai/OLMo-2-1124-13B-Instruct allenai/OLMo-2-0325-32B-Instruct openai/gpt-5.4-mini anthropic/claude-haiku-4.5 google/gemini-2.5-flash x-ai/grok-3-mini-beta

Params

Access

Route

1.5B 3B 7B 1B 3B 8B 1B 4B 13B 32B — — — —

Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight Open-weight API API API API

Local Local Local Local Local Local Local Local Local Local OpenRouter OpenRouter OpenRouter OpenRouter

Table 3: Model panel used in A NCHOR B ENCH. Open-weight models were served locally on H100 GPUs; API models were accessed through OpenRouter. Stage 1: partial evidence

Stage 2: full evidence

A company is evaluating delivery reliability for a logistics partner.

This is the same case. Full evidence is now available.

Based on these preliminary indicators, provide your initial estimate (0–100).

Evidence: – Operations report A: 38 – Operations report B: 42 – Operations report C: 67 – Operations report D: 61 – Operations report E: 57

Model answer: 40

Given this complete evidence, provide your revised estimate (0–100).

Preliminary evidence: – Operations report A: 38 – Operations report B: 42

This Stage-1 answer becomes the self-generated anchor carried into Stage 2 through conversation history.

Gold: y∗ = 53 (mean of all five ratings). An anchored model

(a) Initial answer

(b) Revision after full evidence

may under-adjust from 40 toward 53.

Figure 4: Example A NCHOR B ENCH item for the History suite. The model first answers from partial evidence; that Stage-1 answer then serves as a self-generated anchor when the full evidence is revealed in Stage 2. In anchored conditions, this two-stage interaction creates the self-generated anchor pathway; the control instead presents the full evidence in a single turn. A.2

Prompt examples

A.3

Full per-suite results

Tables 4–8 report the full per-suite metrics for all fourteen models (ten open-weight plus four API-served). All metrics are computed on 360 items per model per suite (1,800 prompts). A.4

Cross-suite patterns

This section summarizes cross-suite patterns that help interpret Tables 4–8 without repeating all cell-level values. Table 9 reports suite-level means for UAI by relevance. Anchoring concentrates in E XTERNAL, H ISTORY, and T OOL; ICL (metadata) is near zero; RAG is moderate. History and Tool magnitudes are qualified by format confounds (Appendices A.10, A.12). API models show smaller but non-zero discrimination, mainly on E XTERNAL and RAG.

15

Published as a conference paper at COLM 2026

Shared few-shot scaffold (ICL suite)

Header line by condition

Three demonstrations precede the target item. Demo evidence and answers are identical across conditions. Below shows demo 1 of 3; demos 2 and 3 follow the same pattern.

Control: Example 1: Irrelevant (a=23): Example 1 [Case #23 , Batch 23 ]:

[condition-specific header inserted here]

Plausible (a=23):

– Market analyst A: 45 – Market analyst B: 51 Answer: 48

Example 1 [Prior estimate for similar cases:

23 ,

confidence: low]:

After 3 demos, the target item follows: Now estimate for a new case: (scenario + evidence + question, same as External)

The anchor value 23 appears only in demonstration metadata. The demo body and answer (48) are identical across conditions. The stronger ICL-dist variant instead embeds anchors in the demonstration answers themselves (Appendix A.11).

Demo answers and target evidence are unchanged.

(a) Shared scaffold

(b) Anchor conditions

Figure 5: Example A NCHOR B ENCH item for the ICL suite. Three demonstrations precede the target item. In the main-benchmark variant (ICL-metadata), the anchor appears only in the demonstration header metadata; demo evidence, demo answers, and target evidence are unchanged across conditions. Shared retrieved context (RAG suite)

Document 2 by condition

A three-document mini-corpus is concatenated into the prompt. Documents 1 and 3 are fixed; only Document 2 varies by condition.

Control: Assessment workflow note: no external benchmark data was located for this assessment case.

[Document 1] (fixed, no anchor) Industry analysis: The logistics sector has seen increased attention to delivery reliability metrics. . .

Administrative note: document batch reference #23 . This

Irrelevant (a=23):

[Document 2 inserted here]

identifier is assigned sequentially and is unrelated to assessment content or outcomes.

[Document 3] (fixed, no anchor) Data collection methodology: Ratings were gathered from independent assessors using standardized instruments. . .

Plausible (a=23): A recent survey of comparable organizations reported a delivery reliability index of approximately 23 .

After the documents: scenario + evidence (same five ratings as other suites) + question.

The anchor-bearing document occupies the middle slot by default. Only Document 2 changes across conditions; Documents 1 and 3 remain fixed.

Documents are concatenated directly (no retriever).

(a) Shared scaffold

(b) Anchor conditions

Figure 6: Example A NCHOR B ENCH item for the RAG suite. A frozen three-document mini-corpus is concatenated into the prompt. One document serves as the anchor slot, while the surrounding documents provide anchor-free context. This suite operationalizes retrieved-document context as an anchor pathway, without a live retriever. A.5

Statistical robustness checks

Reporting pipeline. Per-item UAI values are computed for each (model, suite, relevance, direction) cell (excluding ε-threshold items), then pooled across direction to give one UAIirr and one UAIpls mean per model–suite cell; Disc∆ is their difference. Suite-level means aggregate over models with equal weight, and bootstrap CIs and Wilcoxon tests treat each model–suite cell as one observation. Item-level uncertainty. For each model–suite run, we report 95% nonparametric bootstrap CIs (B=2000, seed 42) over items for control accuracy, UAI components, Disc∆ , and parse rate. These intervals complement the model-level summaries in Table 10. Paired cross-suite comparisons. History has higher Disc∆ than ICL on average across models (mean difference 0.24, bootstrap 95% CI [0.11, 0.41]; Wilcoxon pBH ≈ 0.01). Tool Disc∆ is not significantly different from the mean of the other suites on models with defined 16

Published as a conference paper at COLM 2026

Shared tool scaffold (Tool suite)

Tool 2 by condition

Two simulated tool calls are injected before the model answers. Tool 1 is anchor-free in all conditions; Tool 2 is the anchor slot.

Control: {"status": "no_data", "note": "No external reference available."}

Tool 1: get_evidence_summary {"mean": 60.0, "std": 3.7, "min": 54, "max": 65, "count": 5}

{ "request_id": 23 ,

Irrelevant (a=23): "status":

"no_data",

"note":

"Request ID is unrelated to assessment content."}

[Tool 2 inserted here]

Plausible (a=23):

After the tool outputs: scenario + evidence + question (same as other suites).

{ "reference_value": 23 ,

"source":

"industry_survey_2025", "confidence": "low"}

Tool 1 is fixed across conditions; only Tool 2 varies.

Qwen and Llama receive native role: "tool" messages; Gemma, OLMo, and API models receive equivalent plaintext rendering (Appendix A.12).

(a) Shared scaffold

(b) Anchor conditions

Figure 7: Example A NCHOR B ENCH item for the Tool suite. Two simulated tool outputs are injected before the model answers. The first is anchor-free; the second carries the conditiondependent anchor. This suite operationalizes tool-output context as an anchor pathway rather than full agentic tool use. MAEc ↓ Acc10 (%) ↑ UAIirr UAIpls TARirr TARpls Disc∆ Parse

Model 2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

17.93 9.62 7.74 16.79 9.21 5.61 15.43 16.33 3.64 13.01

34.2% 63.7% 72.5% 47.2% 69.4% 82.6% 36.8% 55.1% 89.4% 52.8%

0.02 0.01 0.01 0.13 0.14 0.07 0.02 0.20 −0.01 0.05

0.20 0.09 0.27 0.15 0.29 0.36 0.18 0.43 0.16 0.40

0.22 0.21 0.15 0.38 0.36 0.22 0.30 0.53 0.25 0.13

0.48 0.50 0.67 0.49 0.59 0.69 0.48 0.63 0.45 0.55

0.18 0.08 0.26 0.03 0.15 0.30 0.16 0.23 0.17 0.35

1.00 0.99 1.00 0.91 1.00 0.98 1.00 0.98 1.00 1.00

5.4- MINI H AIKU 4.5 2.5-F LASH 3- MINI

0.38 0.67 1.07 0.30

98.9% 100.0% 95.8% 99.2%

0.00 0.00 0.02 −0.00

0.14 0.12 0.07 0.16

0.10 0.16 0.10 0.04

0.45 0.57 0.34 0.40

0.14 0.12 0.05 0.16

1.00 1.00 0.99 1.00

Table 4: E XTERNAL suite full results (360 items, 1,800 prompts per model). Tool discrimination (pBH ≈ 0.56). Tool parse rate is modestly lower than the mean parse rate over External, History, ICL, and RAG (mean gap about 12 percentage points, bootstrap 95% CI [−0.31, −0.009]; pBH ≈ 0.05), consistent with higher format sensitivity. Interpretation limits. The model panel is not a random sample of all LLMs, so inferential statistics should be read as internal consistency checks rather than population claims. Cells are also structured by model and suite, so correlation intervals are descriptive. A.6

UAI exclusion-threshold sensitivity

The UAI metric (§3.4) excludes items where | a − yctrl | < ε to avoid division by small denominators. We report exclusion rates and Disc∆ stability across ε ∈ {1, 3, 5}. Exclusion rates. At the default ε=3, mean exclusion rates range from 3.8% (ICL) to 11% (History). History’s higher rate is expected: the self-generated Stage 1 anchor can closely approximate the gold answer for high-accuracy models. At ε=5, rates increase further but remain moderate.

17

Published as a conference paper at COLM 2026

Model

MAEc ↓ Acc10 (%) ↑ UAIirr UAIpls TARirr TARpls Disc∆ Parse

2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

19.02 9.56 7.11 17.23 9.20 5.74 15.81 15.78 5.14 12.40

32.5% 60.6% 78.6% 48.8% 72.2% 82.9% 39.2% 56.2% 85.8% 53.6%

0.29 −0.08 −0.06 0.48 0.15 0.13 −0.17 0.23 −0.00 −0.30

0.98 0.49 0.37 0.14 0.73 0.23 0.13 0.50 0.32 0.75

0.52 0.46 0.37 0.65 0.49 0.31 0.43 0.58 0.29 0.29

0.71 0.70 0.63 0.57 0.57 0.43 0.59 −0.35 0.70 0.58 0.42 0.10 0.43 0.31 0.70 0.27 0.56 0.32 0.63 1.05

1.00 1.00 1.00 0.92 0.99 0.99 1.00 0.99 1.00 1.00

5.4- MINI H AIKU 4.5 2.5-F LASH 3- MINI

2.73 1.17 1.93 0.58

94.4% 97.8% 94.7% 97.8%

−0.02 −0.02 0.03 0.10

0.04 −0.06 −0.00 0.10

0.32 0.23 0.17 0.15

0.38 0.06 0.24 −0.04 0.16 −0.03 0.14 −0.01

1.00 1.00 0.99 1.00

Table 5: H ISTORY suite full results (360 items, two-stage protocol). Anchor values are model-generated (Stage 1 answers). API models show near-zero discrimination, consistent with strong self-correction from full evidence in Stage 2. Model

MAEc ↓ Acc10 (%) ↑ UAIirr UAIpls TARirr TARpls Disc∆ Parse

2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

10.27 7.16 5.50 16.26 6.98 7.31 13.38 13.53 10.56 8.15

59.4% 77.5% 87.5% 54.5% 80.8% 76.7% 46.4% 38.6% 56.9% 69.7%

0.01 −0.04 −0.03 0.30 0.07 0.03 −0.00 0.00 0.03 −0.02

0.00 −0.04 −0.01 0.22 0.14 0.15 −0.03 −0.06 0.02 −0.01

0.23 0.15 0.17 0.42 0.36 0.40 0.16 0.31 0.18 0.14

0.25 0.21 0.23 0.49 0.44 0.51 0.24 0.33 0.29 0.19

−0.01 0.00 0.01 −0.09 0.06 0.12 −0.02 −0.07 −0.01 0.02

1.00 1.00 1.00 0.92 1.00 0.97 1.00 1.00 1.00 1.00

5.4- MINI H AIKU 4.5 2.5-F LASH 3- MINI

0.47 1.07 0.31 0.39

99.4% 96.9% 99.4% 99.2%

−0.00 −0.00 −0.00 −0.00

−0.01 −0.02 0.00 0.00

0.12 0.11 0.06 0.04

0.13 −0.01 0.23 −0.01 0.07 0.00 0.06 0.01

1.00 1.00 1.00 1.00

Table 6: ICL suite full results (360 items). Anchors appear only in demo-header metadata; demo evidence and answers are identical across conditions. Sign stability. Across 69 model–suite cells with computable Disc∆ , most (≈80%) preserve the same sign at all three ε values. The 13 cells with sign flips are concentrated in ICL (7 cells, where effects are near zero) and small-magnitude RAG/Tool cells. No External or History cell with |Disc∆ | > 0.05 shows a sign flip, confirming that the main findings are robust to the exclusion threshold. A.7

UAI distribution and extreme values

UAI can exceed 1 (overshoot past the anchor) or be negative (reverse shift). Table 11 summarizes the distribution across all per-item UAI values in the benchmark (44,261 irrelevant, 43,530 plausible). Robustness to aggregation method. The median UAI is zero for both relevance levels, reflecting that many items show no shift. However, the sign of Disc∆ = UAIpls − UAIirr is preserved for all five suites under both mean and median aggregation, and under capped [0, 1] means. Capping reduces the History Disc∆ from 0.40 to 0.23 (overshoot accounts for part of the magnitude) but does not change the direction or relative ranking of suites.

18

Published as a conference paper at COLM 2026

Model

MAEc ↓ Acc10 (%) ↑ UAIirr UAIpls TARirr TARpls Disc∆ Parse

2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

22.15 8.12 9.83 13.73 10.60 4.66 14.98 12.64 4.21 11.00

21.9% 74.1% 63.9% 60.5% 71.5% 85.1% 37.2% 47.5% 87.2% 58.1%

0.02 −0.02 −0.00 0.10 0.14 −0.02 −0.00 0.05 −0.00 0.01

0.23 0.08 0.20 0.09 0.22 0.12 0.04 0.01 0.05 0.45

0.18 0.15 0.11 0.29 0.29 0.22 0.12 0.23 0.29 0.07

0.49 0.21 0.39 0.10 0.48 0.20 0.35 −0.01 0.36 0.08 0.36 0.14 0.23 0.04 0.32 −0.05 0.35 0.05 0.57 0.44

1.00 1.00 1.00 0.96 0.99 0.99 1.00 1.00 1.00 1.00

5.4- MINI H AIKU 4.5 2.5-F LASH 3- MINI

0.57 1.10 5.55 0.26

98.6% 98.3% 84.2% 99.2%

0.01 0.00 0.08 −0.01

0.09 0.06 0.17 0.02

0.11 0.18 0.26 0.03

0.29 0.31 0.36 0.08

1.00 1.00 0.99 1.00

0.08 0.05 0.09 0.03

Table 7: RAG suite full results (360 items). Model

MAEc ↓ Acc10 (%) ↑ UAIirr UAIpls TARirr TARpls Disc∆ Parse

2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

26.17 4.62 0.67 — 2.33 12.15 20.51 14.21 9.28 7.98

45.9% 89.2% 98.9% — 94.5% 73.3% 25.8% 62.8% 70.9% 72.5%

0.26 0.08 0.00 — 0.03 0.31 −0.05 0.18 0.09 0.01

0.82 0.18 0.17 — 0.21 0.48 −0.03 0.21 0.07 0.02

0.28 0.24 0.08 — 0.17 0.28 0.23 0.48 0.37 0.11

0.72 0.57 0.37 0.10 0.51 0.17 — — 0.34 0.19 0.65 0.17 0.31 0.02 0.55 0.04 0.47 −0.02 0.15 0.01

0.97 0.98 1.00 0.01 0.80 0.96 1.00 0.97 1.00 1.00

5.4- MINI H AIKU 4.5 2.5-F LASH 3- MINI

0.52 1.27 1.12 0.45

99.4% 98.3% 98.3% 98.9%

0.00 −0.01 0.03 −0.01

0.06 0.01 0.06 0.02

0.13 0.15 0.12 0.01

0.43 0.27 0.19 0.11

1.00 1.00 0.98 1.00

0.06 0.02 0.03 0.02

Table 8: T OOL suite full results (360 items). Tool outputs are structured (role: "tool") for Qwen and Llama, and plaintext for Gemma, OLMo, and API models. Llama-3.2-1B metrics are undefined (parse rate 0.9%). A.8

Boundary clipping sensitivity

At δ=40, some anchors are clipped to the scale boundaries (a=0 or a=100) when θ < 40 or θ > 60. To check whether this clipping inflates the apparent attenuation at large offsets, we compare dose-response trends with and without boundary items (External + RAG, plausible UAI): δ=15 δ=25 δ=40 All items Excl. boundary

0.23 0.23

0.18 0.18

0.11 0.14

Excluding 1,731 boundary items (26% of δ=40 items) raises the mean UAIpls at δ=40 from 0.11 to 0.14. The monotonic attenuation trend is preserved, but roughly half of the drop from δ=25 to δ=40 is attributable to boundary clipping. At δ ∈ {15, 25}, no items hit the boundaries (all θ ∈ [30, 70]).

19

Published as a conference paper at COLM 2026

Suite

UAIirr UAIpls Disc∆

+ n

E XTERNAL H ISTORY ICL (metadata) ICL (ICL-dist) RAG T OOL

0.05 0.05 0.02 0.09 0.03 0.07

0.22 0.34 0.03 0.16 0.13 0.18

0.17 0.28 0.00 0.07 0.10 0.11

14/14 10/14 7/14 8/10 12/14 12/13

All (excl. ICL-dist)

0.04

0.18

0.13 55/69

Table 9: Mean anchoring influence by relevance and suite. +/n: cells with Disc∆ > 0. ICL-dist uses distribution-matching demonstrations (open-weight models only). Contrast

Est.

Plausible > irrelevant (mean ∆UAI) E XTERNAL H ISTORY ICL RAG T OOL

0.17 0.28 0.00 0.10 0.11

95% CI

pBH

[0.13, 0.22] <0.01 [0.12, 0.47] 0.02 [−0.02, 0.03] 0.81 [0.05, 0.17] <0.01 [0.04, 0.20] <0.01

Suite-mean Disc∆ range (open-weight panel) 0.40 [0.20, 0.60] Pearson r(Acc, Disc∆ ) −0.24 [−0.43, −0.00]

— —

Table 10: Statistical summary. Top: Wilcoxon signed-rank tests for UAIpls > UAIirr (n=14 models per suite, n=13 on Tool; p-values BH-corrected). Bottom: cross-suite discrimination range and Acc–Disc correlation. All CIs are bootstrap 95%. A.9

Anchored-condition prediction quality

We measure whether anchored conditions degrade prediction quality by comparing MAE in anchored vs. control conditions. Table 12 reports the mean ∆MAE = MAEanchored − MAEctrl across models. Plausible anchors increase MAE by 0.6–3.6 on External, RAG, and Tool, confirming that anchoring produces substantive accuracy degradation, not just directional shifts. History shows a near-zero plausible effect (+0.12) but a notable negative irrelevant shift (−1.61), likely an artifact of single-stage control noise (§A.10). The negative ∆MAEirr for Tool (−0.65) may reflect that irrelevant-condition prompts contain structured tool output that contextualizes the task, marginally improving predictions for some models despite the irrelevant anchor. ICL shows near-zero ∆MAE for both relevance levels, consistent with the weak manipulation. A.10

History suite matched-control analysis

The default History control is single-stage while anchored conditions use a two-stage protocol (§3.3). To assess whether format differences confound anchoring estimates, we compare the standard Disc∆ against a matched-control Disc∆ computed using a two-stage control baseline (where Stage 1 presents a different case, introducing the two-stage format without an item-specific anchor). Key observations. The two-stage control format substantially increases MAE (for example, 5.49 → 24.87 for Qwen-1.5B; mean across models: 8.1 → 16.0), confirming the format itself adds difficulty. Disc∆ drops substantially for 2/10 models: Qwen-1.5B drops from 0.89 to 0.27 (the standard estimate was severely inflated by the format confound), while Gemma-1B drops from 0.21 to 0.09. Most other models show increased Disc∆ under matched control, likely because the two-stage control is noisier (higher MAE), making the shift toward anchors relatively larger.

20

Published as a conference paper at COLM 2026

External History ICL RAG Tool

0.8 0.6 DiscΔ

r = -0.24 95\% CI [−0.43, −0.00]

Suite

1.0

0.4 0.2 0.0 Tier

−0.2

Open-weight API

−0.4 20

30

40

50 60 70 Acc10 (control, \%)

80

90

100

Figure 8: Task accuracy (Acc10 ) vs. discrimination (Disc∆ ) for all 69 computable model–suite cells. Circles: open-weight; diamonds: API; color: suite. The weak Pearson r (−0.24, 95% CI [−0.43, −0.00]) shows that accuracy and anchoring robustness are only weakly associated. UAIirr UAIpls In [0, 1] Overshoot (> 1) Reverse (< 0)

77.1% 4.2% 18.7%

73.4% 7.0% 19.7%

Mean Median Capped [0, 1] mean

0.04 0.00 0.10

0.17 0.00 0.22

−0.67 0.00 0.00 0.97

−0.78 0.00 0.33 1.32

5th pctl 25th pctl 75th pctl 95th pctl

Table 11: Distribution of per-item UAI values across all models and suites. The majority of items fall in [0, 1], but roughly 4–7% overshoot and ∼19% show reverse shifts. Overshoot concentrates in History (23.8% for plausible). Interpretation. The sign of Disc∆ is preserved for 10/10 models, supporting genuine anchoring above format effects. However, the magnitudes vary substantially between the two estimates. We recommend interpreting History Disc∆ as qualitative evidence for self-generated anchoring rather than as precise quantitative estimates, and note that the standard (single-stage control) likely overestimates the effect for small models with large format sensitivity. A.11

ICL distribution-matching comparison

The standard ICL suite uses metadata anchors (case IDs, batch numbers) that are transparently irrelevant to the estimation task (§3.3). To test whether stronger numeric priming produces measurable effects, we create a distribution-matching ICL variant where few-shot demonstration answers are numerically relevant to the target range.

21

Published as a conference paper at COLM 2026

∆MAEirr ∆MAEpls

Suite E XTERNAL H ISTORY ICL RAG T OOL

+0.08 −1.60 +0.14 −0.04 −0.67

+3.61 +0.12 −0.04 +0.59 +3.39

Table 12: Mean change in MAE from control to anchored conditions (averaged across all models per suite). Positive values indicate accuracy degradation. Plausible anchors consistently increase MAE; irrelevant anchors are generally small, though Tool shows a modest negative effect (−0.67). Model 2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

MAEstd MAEts Discstd Discmatch 5.49 3.65 3.77 15.71 9.46 7.02 14.71 12.03 5.40 4.13

24.87 17.67 10.33 20.87 15.42 8.05 27.45 17.32 9.86 8.51

0.89 0.18 0.18 0.01 0.22 0.08 0.21 0.24 0.29 0.12

0.27 0.51 0.97 0.39 0.20 0.33 0.09 0.57 0.82 0.69

Table 13: History suite: standard vs. matched-format (two-stage) control analysis. MAEstd : MAE on single-stage control; MAEts : MAE on two-stage control; Discstd : Disc∆ using singlestage control; Discmatch : Disc∆ using two-stage control. Both columns are computed from the dedicated matched-control evaluation run and should be compared against each other; Discstd values may differ from Table 1 because the matched-control run used a separate item subset and inference pass. Results. Eight of ten models show increased UAIpls under the distribution-matching variant (mean: 0.04 → 0.16, paired t-test p < 0.05). The largest increase is for Qwen-1.5B (0.00 → 0.43), consistent with smaller models being more susceptible to numeric priming in demonstrations. The distribution-matching variant also produces Disc∆ > 0 for 8/10 models (vs. 7/10 for standard ICL), confirming that relevance discrimination emerges when the anchoring pathway is sufficiently strong. Interpretation. The near-zero standard ICL result reflects the weakness of the manipulation (incidental metadata numbers) rather than model immunity to few-shot anchoring. When demonstrations contain numerically relevant information, models show substantial anchoring comparable to RAG and Tool suites. This supports the design choice to include ICL as a minimal-exposure baseline while demonstrating that stronger ICL manipulations produce expected effects. A.12

Tool suite format comparison

The Tool suite uses structured role: "tool" messages for Qwen and Llama models but plaintext context for Gemma, OLMo, and API models (§3.3). To isolate format effects, we re-evaluate all five structured-format models with tool outputs rendered as plaintext (identical to the Gemma/OLMo/API format). Results. Format effects are substantial and model-dependent. Three of five models show reduced discrimination in plaintext: Qwen-1.5B drops from 0.58 to 0.07 (the largest change in the benchmark), Qwen-7B from 0.17 to 0.02, and Llama-3B from 0.23 to 0.10. Two models are stable: Qwen-3B remains near 0.08–0.11 and Llama-8B retains 0.14–0.16. MAE increases

22

Published as a conference paper at COLM 2026

Model

UAIpls,std UAIpls,dist Discstd Discdist

2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-1B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT 3-1B- IT 3-4B- IT 2-13B-I NSTRUCT 2-32B-I NSTRUCT

0.00 −0.04 −0.01 0.22 0.14 0.15 −0.03 −0.06 0.02 −0.01

0.43 0.15 0.06 0.22 0.11 0.26 0.00 0.12 0.12 0.13

−0.01 0.00 0.01 −0.09 0.06 0.12 −0.02 −0.07 −0.01 0.02

0.09 0.07 0.07 0.04 0.10 −0.03 −0.03 0.19 0.12 0.09

Table 14: Standard ICL (metadata anchors) vs. ICL-dist (numerically relevant demonstrations). Model

Discstruct Discplain MAEstruct MAEplain

2.5-1.5B-I NSTRUCT 2.5-3B-I NSTRUCT 2.5-7B-I NSTRUCT 3.2-3B-I NSTRUCT 3.1-8B-I NSTRUCT

0.58 0.08 0.17 0.23 0.16

0.07 0.11 0.02 0.10 0.14

11.34 2.19 0.71 2.28 5.47

15.01 15.74 1.32 9.49 6.77

Table 15: Tool suite: structured (role: "tool") vs. plaintext format for all five models that used structured format in the main benchmark. Disc: Disc∆ ; MAE: control-condition MAE. in plaintext mode for all models, confirming that the structured format provides better task context. Implications. The structured tool format amplifies both task competence (lower MAE) and anchor sensitivity (higher Disc) for some models, particularly smaller Qwen models. Crossmodel Tool-suite comparisons in Table 1 mix formats (structured for Qwen/Llama; plaintext for Gemma/OLMo/APIs) and should be interpreted accordingly. The External and RAG suites, which use identical input formats across all models, provide cleaner cross-model comparisons. A.13

Gold-referenced error decomposition

Table 16 reports the aggregate decomposition of anchored responses into error-increasing, error-reducing, and neutral shifts relative to the evidence-only gold answer; this section adds the per-model breakdown and the anchor-proximity stratification. Suite

Err↑ (%) Err↓ (%) Irrerr↑ Plserr↑

E XTERNAL H ISTORY ICL RAG T OOL

33.9 39.2 22.9 22.2 28.6

20.0 44.0 20.0 20.2 17.5

19.9 29.5 20.3 16.4 17.6

48.0 48.7 25.4 27.9 39.9

All

28.7

23.2

20.2

37.3

Table 16: Error decomposition relative to the evidence-only target (aggregated over fourteen models). Err↑/Err↓: share of anchored items where the response shifts away from / toward the gold answer. Irrerr↑ /Plserr↑ : error-increasing rate by anchor relevance. Overall, errorincreasing shifts (29%) outnumber error-reducing ones (23%); the gap is driven by plausible anchors (37% vs. 20% for irrelevant), with the largest differences on External and Tool (≥22 pp).

23

Published as a conference paper at COLM 2026

Harmful error share by model and relevance Qwen-1.5B Qwen-3B Qwen-7B OLMo-32B OLMo-13B Gemma-1B Gemma-4B Llama-8B Llama-1B Llama-3B Claude-H4.5 Gemini-2.5-Flash GPT-5.4-mini

Irrelevant Plausible

Grok-3-mini

10

20

30

40

50

Harmful shift share (%)

Figure 9: Harmful shift share by model and relevance condition. Each point shows the percentage of anchored items on which the model’s error increased relative to the unanchored control. Plausible anchors (right) consistently produce a larger harmful share than irrelevant anchors (left), with the widest gaps for External and Tool suites. Suite-level patterns. External shows the sharpest relevance contrast: plausible anchors produce harmful shifts 48.0% of the time versus 19.9% for irrelevant anchors (∆ = 28 pp). Tool follows with a large gap (39.9% vs. 17.6%, ∆ = 22 pp). RAG shows a moderate gap (27.9% vs. 16.4%). ICL shows modest differentiation between relevance levels (25.4% vs. 20.3%), consistent with the weak discrimination in the standard ICL suite. History shows a 19 pp gap (48.7% vs. 29.5%). Proximity analysis. When the anchor is farther from the gold answer than the control response, 31.0% of items show harmful shifts. When the anchor is closer to gold, only 14.9% are harmful. This directional asymmetry is expected: anchors that pull responses away from gold are more likely to increase error, while anchors that lie between the control and gold may coincidentally improve predictions. Among the “closer” items, helpful shifts are common (59.3%), confirming that the helpful category largely reflects cases where the anchor direction happens to align with gold. Interpretation. The decomposition shows that plausible anchors increase prediction error more often than they reduce it (37.3% vs. 24.1%; the remaining 38.6% of items are unchanged), and substantially more often than irrelevant anchors. The 17 pp gap between plausible and irrelevant harmful rates (37.3% vs. 20.2%) mirrors the discrimination observed in the main UAI analysis and confirms that the anchor-induced shifts are not symmetric noise but directionally biased toward the anchor value. Breakdown by difficulty and offset. Hard items produce more non-neutral shifts overall: harmful 31.6% vs. 25.8% for easy items, and helpful 25.4% vs. 21.1%. This is consistent with noisier evidence leaving more room for anchor-driven shifts in either direction. Across anchor offsets, plausible harmful rates are similar at δ=15 (35.5%) and δ=25 (36.1%) but decline at δ=40 (33.9%), while irrelevant harmful rates remain stable (≈18.4%). The moderate decline at δ=40 is consistent with the dose-response attenuation, partly amplified by boundary clipping (Appendix A.8). A.14

Difficulty-conditioned analysis

Table 17 reports anchoring metrics separately for easy items (σ =8, all five ratings visible) and hard items (σ =15, two hidden, one conflicting). 24

Published as a conference paper at COLM 2026

Suite

Diff. Acc10 MAEc UAIirr UAIpls Disc∆

E XTERNAL easy hard H ISTORY easy hard ICL easy hard RAG easy hard T OOL easy hard All

77.0% 65.5% 77.1% 65.1% 76.7% 72.4% 77.7% 63.3% 79.0% 79.2%

7.2 9.6 7.5 10.1 6.7 7.8 6.8 10.3 8.0 7.6

0.05 0.04 0.03 0.08 0.03 0.02 0.02 0.03 0.09 0.05

0.16 0.11 0.28 0.23 0.36 0.33 0.31 0.23 0.02 −0.01 0.03 0.01 0.07 0.05 0.19 0.16 0.17 0.08 0.18 0.13

— —

— —

0.04 0.05

0.16 0.20

easy hard

0.12 0.15

Table 17: Anchoring metrics by item difficulty (mean across 14 models per suite, 13 on Tool). Bold highlights the higher Disc∆ within each suite. Hard items show stronger discrimination on External and RAG; History shows the opposite pattern. Findings. Hard items show higher overall susceptibility (mean Disc∆ : 0.15 vs. 0.12 for easy items), driven by External (0.23 vs. 0.11) and RAG (0.16 vs. 0.06). This is consistent with the expectation that models are more vulnerable to anchors when the evidence is sparser or more conflicting. History shows the opposite pattern (easy: 0.33 vs. hard: 0.23): easy items produce a stronger Stage 1 answer that serves as a clearer self-generated anchor, increasing susceptibility despite high task accuracy. ICL remains near zero for both difficulty levels. A.15

Sampling robustness

All main results use greedy decoding (temperature 0). To assess whether conclusions depend on the decoding strategy, we re-evaluate two models (Qwen-7B, Llama-8B) on three suites (External, RAG, ICL-dist) under stochastic sampling (temperature 0.7, top-p 0.9) with three seeds. Suite

Model

E XTERNAL Qwen-7B Llama-8B RAG Qwen-7B Llama-8B ICL-dist Qwen-7B Llama-8B

Disc∆greedy Disc∆sampled Sign 0.26 0.30 0.19 0.16 0.07 0.03

0.29 ± 0.02 0.24 ± 0.07 0.18 ± 0.02 0.06 ± 0.04 0.08 ± 0.02 0.00 ± 0.02

✓ ✓ ✓ ✓ ✓ ×

Table 18: Sampling robustness. Sampled values are mean ± SD over 3 seeds (temperature 0.7, top-p 0.9). Sign: whether greedy and sampled mean share the same Disc∆ sign. Findings. Qwen-7B produces nearly identical metrics under sampling (SD ≤ 0.02). Llama8B is more variable (SD up to 0.07) but preserves the sign of Disc∆ in 5 of 6 cells. The exception—ICL-dist, where greedy yields a small positive Disc∆ (0.03) that averages to zero under sampling—involves a near-zero-effect cell where sign instability is expected. Anchoring conclusions from the greedy main results are robust: plausible > irrelevant anchoring holds across decoding strategies for all suites with meaningful effect sizes. A.16

Mitigation headroom probe

We test whether simple prompt-based strategies can attenuate anchoring effects. Three strategies are compared against the unmodified baseline on the External and RAG suites for two open-weight models:

25

Published as a conference paper at COLM 2026

• Ignore: an instruction to disregard extraneous numbers and base the answer only on the task evidence. • Self-check: after producing an initial estimate, the model is asked to verify whether any extraneous numbers may have biased its answer, and correct if so. • CoT (chain-of-thought): the model is asked to list the relevant evidence, compute an estimate from that evidence only, then state its final answer. Suite

Strategy

Disc∆

MAEctrl

Qwen-7B Llama-8B Qwen-7B Llama-8B E XTERNAL baseline ignore self-check CoT

0.26 0.16 0.45 0.14

0.30 0.10 0.28 0.31

7.7 5.0 8.8 1.1

5.5 3.4 7.5 6.1

RAG

0.20 0.14 0.30 0.03

0.16 0.04 0.11 0.08

9.8 6.8 12.2 1.3

4.8 4.4 6.0 4.5

baseline ignore self-check CoT

Table 19: Mitigation headroom probe. Disc∆ and control MAE for three prompt-based strategies vs. unmodified baseline. Ignore consistently reduces discrimination across both models and suites. CoT strongly helps Qwen-7B but is inconsistent for Llama-8B. Self-check increases discrimination for Qwen-7B on both suites but slightly reduces it for Llama-8B. Results. The ignore strategy consistently reduces Disc∆ across all four model–suite cells (mean reduction: −0.12), with concurrent MAE improvement on External for both models. CoT strongly reduces discrimination for Qwen-7B (−0.12 on External, −0.17 on RAG) and moderately for Llama-8B on RAG (−0.08), but slightly increases it for Llama-8B on External (+0.01). The self-check strategy is model-dependent: it increases Disc∆ for Qwen-7B on both suites (External: +0.19; RAG: +0.10) while slightly reducing it for Llama-8B (External: −0.01; RAG: −0.05). Differences of this size should not be read directionally. Re-running the Llama-8B×External cell under an identical configuration moves Disc∆ by up to 0.06 and UAIpls by up to 0.08, because batched GPU inference is not bitwise reproducible even at temperature 0 (Appendix A.17.7, footnote 2). Only the larger effects (|∆| ≳ 0.10) in this table are resolved by a single run. Interpretation. Simple prompt-based mitigations can partially attenuate anchoring, but no single strategy eliminates it and effectiveness is model-dependent. The ignore strategy is the most reliable, reducing discrimination by 30–75% on External and RAG. CoT is highly effective for Qwen-7B (which also sees large MAE improvements, down to 1.1–1.3) but inconsistent for Llama-8B. Self-check backfires for Qwen-7B, likely because revisiting the context gives the anchor a second opportunity to bias the response; for Llama-8B, the effect is small. These results suggest headroom for targeted interventions but indicate that robust debiasing will require more than instruction-level changes. A.17

Additional analyses

This section collects supplementary analyses that probe the robustness, interpretation, and generality of the main findings. All use the standardized inference pipeline of §3.5 and, unless noted, the locked four-model open-weight panel (Gemma-4B, Llama-8B, OLMo-13B, Qwen-7B), chosen to span families and accuracy levels at manageable cost. A.17.1

A rational-updating reference for UAI

A bare UAI value does not say whether a shift is excessive. We give it a reference point with a simple Gaussian-conjugate Bayesian model: treat the visible evidence as n ratings of unit credibility and the anchor as worth w ratings. A rational updater then closes a fraction 26

Published as a conference paper at COLM 2026

Model

External History

Claude-H4.5 GPT-5.4-mini Gemini-2.5-Flash Gemma-1B Gemma-4B Grok-3-mini Llama-1B Llama-3B Llama-8B OLMo-13B OLMo-32B Qwen-1.5B Qwen-3B Qwen-7B

0.71 0.79 0.38 1.10 3.74∗ 0.97 0.91 2.09∗ 2.82∗ 0.94 3.27∗ 1.25 0.50 1.86∗

Rag

Tool

Icl

0.00 0.29 0.03 0.23 0.47 0.33 0.00 1.03 0.29 0.76 0.22 0.00 5.00∗ 0.03 1.37 0.53 0.11 0.08 0.79 0.51 — 13.32∗ 1.43 1.35 1.52 0.69 4.58∗ 2.30∗ 0.25 0.40 15.17∗ 4.04∗ 0.10 319.68∗ 1.46∗ 23.51∗ 4.88∗ 0.43 1.13 2.99∗ 1.22 1.05

0.00 0.00 0.00 0.00 0.00 0.02 1.38 0.78 0.88 0.08 0.00 0.02 0.00 0.00

Table 20: Implied anchor weight wimp on plausible anchors, defined as the smallest weight under a Gaussian-conjugate Bayesian model that would make the observed mean UAI rational (with n = 5 evidence ratings). Under the reference assumption that an anonymous anchor is worth one evidence item (w = 1), the rational UAI ceiling is 0.167; wimp > 1 means the model treats the anchor as more credible than a single evidence item. wimp > n means the model treats the anchor as more credible than all visible evidence combined. ∗ marks cells where the 95% CI for UAIpls lies entirely above the rational ceiling. Model

External History

Claude-H4.5 GPT-5.4-mini Gemini-2.5-Flash Gemma-1B Gemma-4B Grok-3-mini Llama-1B Llama-3B Llama-8B OLMo-13B OLMo-32B Qwen-1.5B Qwen-3B Qwen-7B

-0.042 -0.030 -0.096 +0.013 +0.261∗ -0.004 -0.013 +0.128∗ +0.194∗ -0.008 +0.228∗ +0.033 -0.076 +0.104∗

Rag

Tool

Icl

-0.226 -0.112 -0.161 -0.184 -0.123 -0.080 -0.105 -0.173 -0.171 +0.005 -0.111 -0.166 -0.034 -0.125 -0.195 -0.193 +0.333∗ -0.161 +0.048 -0.228 -0.071 -0.145 -0.151 -0.163 -0.031 -0.074 — +0.050 +0.560∗ +0.055 +0.046 -0.031 +0.067 -0.045 +0.311∗ -0.018 +0.149∗ -0.120 -0.093 -0.150 +0.585∗ +0.280∗ -0.148 -0.174 +0.818∗ +0.060∗ +0.658∗ -0.162 +0.327∗ -0.088 +0.018 -0.204 +0.207∗ +0.030 +0.007 -0.180

Table 21: Excess UAI on plausible anchors above the rational Bayesian ceiling 0.167 (assuming the anchor is worth one evidence item, w = 1, n = 5 evidence ratings). Positive values indicate the model shifted further toward the anchor than a rational Bayesian update could justify. ∗ marks cells where the 95% CI for UAIpls lies entirely above the ceiling. w/(n+w) of the control–anchor gap, so with n=5 and a reference weight w=1 the rational UAI ceiling is 0.167. For an irrelevant anchor the rational weight is 0, so the ceiling is 0 and any positive UAIirr is bias. For a plausible anchor, we invert the relation to read off the implied weight wimp = UAI/(1 − UAI) · n, the smallest weight that would make the observed shift rational (Table 20). Table 21 reports the matching excess over the ceiling. Across the 55 non-ICL model–suite cells, 16 have a UAIpls whose 95% CI lies entirely above 0.167, and 5 imply wimp > n=5—treating one anonymous anchor as more credible than all five ratings combined. These are the cells we flag as practically significant in §4.4; the rest sit inside the band that a rational update at w=1 would produce.

27

Published as a conference paper at COLM 2026

Model

UAIplacebo UAIirr UAIpls UAIauth

GPT-5.4-mini Gemma-4B Llama-8B OLMo-13B Qwen-7B

-0.01 +0.38 +0.08 +0.02 -0.00

+0.01 +0.20 +0.07 -0.01 +0.01

+0.13 +0.43 +0.36 +0.16 +0.27

+0.40 +0.58 +0.42 +0.49 +0.56

Mean

+0.09

+0.05

+0.27

+0.49

Table 22: Plausibility spectrum on External: mean UAI for each of four anchor types — placebo (anchor refers to an extraneous numeric property such as document length), irrelevant (case-number anchor), plausible (“a recent report suggested ∼23”), and authority (“a panel of senior analysts estimated ∼23”). A monotone increase along the plausibility axis would be consistent with rational use of source credibility; any positive UAI on the placebo column is unambiguously anchoring bias. Acc10

Model Gemma-4B Llama-8B OLMo-13B Qwen-7B Rational ceiling (w=1)

UAIpls

UAIirr

k=1

k=2

k=3 k=1 k=2 k=3 k=1

54% 47% 54% 42%

60% 61% 76% 64%

64% 69% 81% 75%

—

—

0.21 0.68 0.48 0.53

0.20 0.36 0.34 0.34

0.25 0.50 0.29 0.36

— 0.50 0.33 0.25

k=2 k=3

0.14 −0.13 0.13 0.25 −0.03 0.20 0.19 0.11 0.06 0.22 0.01 0.02 —

—

—

Table 23: Uncertain-judgment results. Only k of 5 ratings are shown to the model; the gold answer remains the full-5 mean. An ideal Bayesian updater treating the plausible anchor as one additional rating of equal credibility (w=1) would produce UAIpls = w/(k + w) (Rational ceiling row). Measured UAIpls above the ceiling indicates classical (super-rational) anchoring; below indicates the model weights the anchor less than one full additional evidence point. UAIirr should stay near zero for capable models at every k. A.17.2

Plausibility spectrum and a placebo floor

To separate anchoring on the numeric value from rational use of source credibility, we extend the binary relevance axis to a four-point spectrum on External: placebo (the number is an extraneous property such as a document’s age in days), irrelevant (a case number), plausible (“a recent report suggested ∼23”), and authority (“a panel of senior analysts estimated ∼23”). A purely rational model would ignore placebo and irrelevant values equally and move only for plausible/authority framings. Two observations argue against a purely rational reading. The placebo column is not zero (mean UAI 0.09) and sits close to irrelevant (0.05)—a shift toward a number that is literally a document length. And the increase along the axis is not clean: for some models the placebo shift rivals the plausible one. This is the behavior we expect if the numeric value itself is salient, not only its stated source. A.17.3

Uncertain judgment under partial evidence

The main task shows all five ratings, so the gold answer is a fully determined mean. To introduce genuine epistemic uncertainty, we re-render External items showing only k of the five ratings while still scoring against the full-five mean. Now the anchor could rationally carry information, and the rational ceiling rises to w/(k +w): 0.50, 0.33, 0.25 at k =1, 2, 3. Measured UAIpls frequently meets or exceeds the rational ceiling at each k (Table 23), and decreases as more evidence becomes visible—models do treat the anchor as one piece of evidence among several, but several cells still sit above what rational updating allows. UAIirr stays comparatively low at every k.

28

Published as a conference paper at COLM 2026

External

Model

RAG

History

Mild

Std Strong Mild

Std Strong Mild

Gemma-4B Llama-8B OLMo-13B Qwen-7B

0.45 0.26 0.07 0.23

0.42 0.39 0.16 0.31

0.40 0.42 0.29 0.49

0.00 −0.06 0.10 0.09 0.02 0.09 0.07 0.19

0.04 0.29 0.10 0.29

0.61 0.44 0.45 0.37

0.58 0.20 0.24 0.24

0.60 1.57 0.91 0.52

Mean

0.25 0.32

0.40

0.05

0.18

0.47 0.31

0.90

0.08

Std Strong

Table 24: Cross-pathway plausibility-intensity dose-response. The same mild/standard/strong source-credibility manipulation is applied across three pathways. Positive Strong > Mild gaps indicate models weight source credibility; near-flat curves at high UAI indicate numeric anchoring dominates regardless of source. UAI values are mean across the panel; details in supplementary CSV. UAIirr Suite

Model

bus med

UAIpls oth

external Gemma-4B 0.20 0.28 0.20 Llama-8B 0.07 0.03 0.20 OLMo-13B -0.01 0.08 -0.03 Qwen-7B 0.01 0.07 0.01 history Gemma-4B 0.23 0.17 0.03 Llama-8B 0.13 -0.23 0.42 OLMo-13B -0.00 0.43 0.46 Qwen-7B -0.06 -0.37 -0.02

Disc∆

bus med

oth

bus med

0.43 0.36 0.16 0.27 0.50 0.23 0.32 0.37

0.67 0.46 0.22 0.45 0.40 0.36 0.51 0.17

0.23 -0.07 0.46 0.30 0.21 0.26 0.17 0.01 0.24 0.26 0.32 0.44 0.27 0.24 0.37 0.10 0.48 -0.06 0.32 0.10 0.05 0.43 1.02 0.19

0.21 0.25 0.09 0.38 0.41 0.25 0.53 0.65

oth

Table 25: Eleven-domain robustness panel: 6 business domains (published benchmark), 3 medical pilot domains, and 2 further pilot domains (legal contract compliance, consumer purchase decision). The pattern UAIpls >UAIirr > 0 replicates across all three domain families on both External and History, demonstrating that anchoring is not a businessdomain artifact. A.17.4

Cross-pathway credibility intensity

We apply a matched three-level source-credibility manipulation (mild / standard / strong) to three pathways (External, RAG, History). If models weight source credibility, the curve should rise from mild to strong; a flat-but-high curve indicates numeric anchoring that ignores the source. External and RAG show a clean monotone increase, consistent with credibility-sensitive updating. History is non-monotone and much larger, because the credibility cue compounds with the model’s own self-generated Stage-1 anchor; we therefore read the History intensity numbers qualitatively. A.17.5

Domain extension beyond business scenarios

The six core domains are business-oriented. To check that anchoring is not a businessdomain artifact, we run an 11-domain panel that adds three medical domains and two further extension domains (legal contract compliance, consumer purchase decisions), keeping the same item structure. The ordering UAIpls >UAIirr > 0 holds in all three domain families on both External and History (Table 25). Magnitudes vary by family and model, but the qualitative pattern is stable, so the effect generalizes beyond the original domains.

29

Published as a conference paper at COLM 2026

UAIirr

UAIpls

Disc∆

Model

mean wmean mean wmean mean wmean

Gemma-4B Llama-8B OLMo-13B Qwen-7B

0.20 0.07 -0.01 0.01

0.20 0.06 0.03 0.01

0.43 0.36 0.16 0.27

0.47 0.34 0.20 0.27

0.23 0.30 0.17 0.26

0.28 0.28 0.17 0.26

Table 26: Anchor influence on External under the published mean gold standard (“mean”) vs. a weighted-mean gold standard with weights [1, 1, 1.5, 1.5, 2] (“wmean”). The bias structure (UAIpls ≫UAIirr ) is preserved under either scoring choice; the size of the anchoring effect is not an artifact of the simple-mean aggregation.

Suite

Model

UAIirr

UAIpls

Disc∆

base CoT

∆ base CoT

∆ base CoT

∆

external GPT-5.4-mini 0.01 -0.01 -0.01 0.13 0.14 +0.01 0.12 0.14 +0.02 Gemma-4B 0.09 0.01 -0.08 0.37 0.23 -0.14 0.28 0.22 -0.06 Llama-8B 0.08 0.06 -0.03 0.39 0.37 -0.03 0.31 0.31 -0.00 OLMo-13B 0.06 0.05 -0.01 0.15 0.18 +0.03 0.08 0.13 +0.04 Qwen-7B 0.01 0.00 -0.01 0.27 0.16 -0.12 0.26 0.15 -0.11 history GPT-5.4-mini -0.01 0.01 +0.01 0.02 -0.03 -0.05 0.03 -0.03 -0.06 Gemma-4B 0.24 0.31 +0.07 0.57 0.64 +0.07 0.32 0.33 +0.00 Llama-8B 0.08 0.27 +0.19 0.29 0.28 -0.01 0.21 0.01 -0.20 OLMo-13B -0.09 0.09 +0.18 0.51 0.17 -0.34 0.60 0.08 -0.52 Qwen-7B -0.10 -0.02 +0.08 0.42 0.13 -0.29 0.52 0.15 -0.37 rag GPT-5.4-mini 0.00 -0.00 -0.01 0.09 0.08 -0.01 0.09 0.08 -0.00 Gemma-4B -0.03 0.07 +0.09 -0.05 0.25 +0.31 -0.03 0.19 +0.22 Llama-8B 0.08 0.07 -0.02 0.20 0.12 -0.08 0.12 0.06 -0.06 OLMo-13B 0.08 0.04 -0.04 0.05 -0.01 -0.06 -0.03 -0.05 -0.02 Qwen-7B -0.00 -0.00 +0.00 0.19 0.03 -0.16 0.19 0.03 -0.16

Table 27: Reasoning-allowed (Chain-of-Thought) prompting vs. “final answer only” baseline across the locked 5-model panel and all three suites (External, RAG, History). CoT lowers plausible-anchor influence in 11 of the 15 model–suite cells, but the effect is model- and pathway-dependent rather than uniform: it lowers both UAIirr and UAIpls in only 6 cells, and raises UAIpls in 4. UAIpls remains positive in 13 of 15 cells, so anchoring is attenuated but not an artifact of the answer-only protocol. A.17.6

Robustness to the gold-standard choice

The gold answer is the simple mean of the ratings. One might worry that the anchoring magnitude is an artifact of this particular aggregation. We recompute UAI against a weighted-mean gold (weights [1, 1, 1.5, 1.5, 2] over the five ratings). The bias structure (UAIpls ≫UAIirr ) is preserved, and the mean change in UAIpls is only +0.02 (Table 26), so the effect is not an artifact of the simple-mean scoring. A.17.7

Reasoning-allowed (chain-of-thought) evaluation

The main protocol asks for a final integer only. Here we let the model reason first (chain-ofthought) and compare to the answer-only baseline across five models and three suites. CoT lowers anchor influence in 11 of 15 (model, suite) cells (mean ∆UAIpls = −0.06) but rarely removes it: UAIpls > 0 under CoT in 13/15 cells.2 The answer-only numbers in 2 The Llama-8B×External×CoT cell reported here (UAI pls 0.39 → 0.37, ∆ = −0.03) differs from the separate mitigation-headroom run in Appendix A.16 (0.336 → 0.338, ∆ = +0.00). We traced this: the two runs render byte-identical prompts, gold answers and anchor values, and both use greedy

30

Published as a conference paper at COLM 2026

UAIpls

Model

UAIirr

Baseline +Rule +Judgment Baseline +Rule +Judgment Gemma-4B Llama-8B OLMo-13B Qwen-7B

0.51 0.32 0.20 0.27

0.08 0.01 0.01 0.13

0.38 0.28 0.12 0.27

0.27 0.05 0.04 0.01

0.02 0.00 0.00 0.02

0.25 0.14 0.08 −0.00

Mean

0.33

0.06

0.26

0.09

0.01

0.12

Table 28: Task-specification ablation on External. Baseline = published prompt (return integer only); +Rule appends ‘Your estimate should be the unweighted arithmetic mean of the visible ratings’; +Judgment appends ‘Your estimate should be a weighted average ... weighting each piece according to which sources you find more or less credible’. A drop from Baseline to +Rule indicates the published anchoring effect is partly driven by task underspecification. the main paper are therefore an upper bound on a reasoning-allowed deployment, with Llama-8B and the History pathway as the residual stress test. A.17.8

Task specification: rule vs. judgment

To test how much of the anchoring effect is task underspecification rather than a deep numeric bias, we append two prompt suffixes on External: +Rule (“report the unweighted arithmetic mean of the ratings”) and +Judgment (“report a weighted average, weighting sources by credibility”). The explicit rule cuts mean UAIpls from 0.33 to 0.06, so a substantial share of the published effect is attributable to underspecification. But it does not vanish, and the residual under +Rule is the portion that confusion alone cannot explain. The +Judgment prompt, which invites credibility weighting, keeps UAIpls below the published baseline, consistent with the rational ceiling of Appendix A.17.1. This addresses the concern that weak-model behavior reflects task confusion: even when told exactly how to aggregate, models retain measurable anchoring. A.17.9

RAG realism: rank, distractors, and scores

The published RAG suite places the anchor document in a fixed slot of a three-document corpus. We vary the realism: move the anchor to the top (R1) or bottom of a five-document list with two plausible distractors (R5+D), and optionally expose synthetic per-document relevance scores (R5+D+Sc). Plausible-anchor influence persists across rank and distractor changes, and is slightly higher at the top of the ranking—models privilege top-ranked retrieved documents, as a real RAG system would. It falls toward zero only when explicit relevance scores let the model downweight a mid-ranked anchor, suggesting that production pipelines with good relevance signals would see weaker anchoring than the controlled suite. decoding, so the difference is not a prompt or protocol difference. It is run-to-run variation of batched GPU inference, which is not bitwise reproducible even at temperature 0 because batch composition changes the reduction order inside the attention and GEMM kernels. An independent third run of this cell under the configuration used here gives 0.32 → 0.35 (∆ = +0.03): across the three runs the baseline UAIpls spans 0.32–0.39 and ≈8% of parsed answers change. The sign of the CoT effect for this single cell is therefore not resolved by one run. The aggregate conclusions above are unaffected: they are driven by cells whose effects (∆ up to −0.34) are an order of magnitude larger than this variation, and UAIpls stays positive under CoT in all three runs. We report each run with its own numbers rather than re-normalizing across them.

31

Published as a conference paper at COLM 2026

Plausible (UAI)

Model Base

Irrelevant (UAI)

R1 R5+D R5+D+Sc

Gemma-4B −0.06 0.00 Llama-8B 0.09 0.12 OLMo-13B 0.09 0.02 Qwen-7B 0.19 0.23

0.01 0.14 0.21 0.14

Mean

0.13

0.08 0.10

Base

R1 R5+D R5+D+Sc

0.01 0.03 −0.04 0.01 −0.04 −0.02 0.01 0.01 0.04 0.04 0.09 0.13 0.11 0.01 −0.02 −0.02 0.03

0.02

0.01

0.03

0.02 0.01 0.05 0.00 0.02

Table 29: RAG realism ablation. Base = anchored doc at position 2 (published RAG layout); R1 = anchor at top of retrieval (rank 1); R5+D = anchor at bottom of a 5-doc list with 2 plausible distractor documents adjacent to the anchor; R5+D+Sc adds synthetic per-doc relevance scores. A drop from R1 to R5+D indicates models down-weight low-ranked retrieved anchors. Plausible (UAI)

Irrelevant (UAI)

Base Elicited Noisy

Base Elicited Noisy

Model Gemma-4B Llama-8B OLMo-13B Qwen-7B

0.13 0.62 0.06 0.17

0.28 0.29 0.31 0.54 0.58 0.45 0.06 0.15 0.06 0.02 −0.01 −0.00

0.35 0.39 0.40 0.45 0.02 −0.01 0.01 0.00

Mean

0.25

0.23

0.19

0.25

0.20

0.21

Table 30: Tool realism ablation. Base reproduces the published Tool suite (externally injected tool output); Elicited prepends a synthetic model-planned tool-call turn (provenance signal); Noisy wraps the anchor value in a realistic JSON envelope containing additional metadata fields. Closing the gap between Base and Elicited indicates the anchor effect is not driven by the externally-injected framing; reductions under Noisy indicate the anchor’s salience competes with surrounding metadata. A.17.10

Tool realism: provenance and noise

We vary two tool-realism axes: Elicited prepends a synthetic model-planned tool-call turn (a provenance signal), and Noisy wraps the anchor value in a realistic JSON envelope with extra metadata fields. On average the panel barely moves (mean UAIpls 0.25 → 0.23 → 0.25), so the effect is not an artifact of the externally-injected framing. But the per-model picture splits: Qwen-7B collapses to near zero under both variants while Gemma-4B strengthens, which is why we treat Tool numbers as robust on average but not reliable per model (§4.1). A.17.11

Extended large-model panel

To test whether scale removes anchoring, we evaluate five additional larger models on all five suites: two 70B-class open-weight models (Llama-3.3-70B, Qwen2.5-72B) and three frontier API models (GPT-5.4, Claude-Sonnet-4.6, Grok-4.3). The main-panel pattern persists at scale (Table 31): irrelevant anchors stay near zero, plausible anchors are positive on 21/25 cells, and even GPT-5.4 (near-ceiling accuracy on every suite) shows a small but strictly positive UAIpls on 4/5 suites. Llama-3.3-70B on External (UAIpls = 0.23) is the worst-case in this panel, comparable to Llama-3.1-8B in the main results. A.17.12

Per-item case studies

To show that the aggregate UAI pattern reflects interpretable per-item behavior rather than an averaging artifact, we surface, for each panel model, the External items where

32

Published as a conference paper at COLM 2026

E XTERNAL

Model

H ISTORY

ICL

RAG

T OOL

Acc10 UAIirr UAIpls Acc10 UAIirr UAIpls Acc10 UAIirr UAIpls Acc10 UAIirr UAIpls Acc10 UAIirr UAIpls 95% −0.02 91% 0.01

0.23 0.16

91% −0.02 93% 0.01

GPT-5.4 100% 0.01 C LAUDE -S ONNET-4.6 99% −0.00 G ROK -4.3 97% 0.05

0.19 0.19 0.25

99% 99% 99%

L LAMA -3.3-70B Q WEN 2.5-72B

0.01 0.12

87% 0.01 0.01 96% −0.01 −0.03

95% −0.02 88% 0.00

0.03 0.02 100% 0.00 0.00 100% 0.02 −0.02 72% −0.02 −0.01 95% 0.09 0.03 97% 0.04 0.05 96%

0.00 0.03 0.02

0.04 0.04

98% 99%

0.01 0.05 0.01 −0.00

0.09 100% 0.00 0.07 95% −0.00 0.18 91% 0.10

0.03 0.06 0.25

Table 31: Extended large-model panel. Task accuracy (Acc10 , control-only) and absolute anchor influence UAIirr /UAIpls across all five suites for two 70B-class open-weight models (top) and three frontier API models (bottom). The qualitative pattern of the main panel persists at scale: irrelevant anchors are near-zero, plausible anchors are positive. Model

Item

ahi

yc

yhi p

yihi

∆ p − ∆i

Qwen-7B Qwen-7B Qwen-7B Llama-8B Llama-8B Llama-8B Gemma-4B Gemma-4B Gemma-4B OLMo-13B OLMo-13B OLMo-13B

EXT-legal_policy-e-off40-005 EXT-resource_consumption-e-off15-002 EXT-pricing_wtp-e-off40-006 EXT-market_demographics-h-off40-005 EXT-resource_consumption-h-off25-010 EXT-transportation_logistics-h-off40-002 EXT-transportation_logistics-e-off40-010 EXT-resource_consumption-e-off15-010 EXT-operations_time-h-off40-002 EXT-market_demographics-e-off40-006 EXT-resource_consumption-e-off15-002 EXT-transportation_logistics-e-off40-009

85 50 83 70 65 97 71 66 98 75 50 82

43 75 44 23 100 38 0 0 100 27 35 43

75 44 74 100 14 100 90 100 9 100 100 100

43 75 44 26 56 45 0 57 58 27 35 43

+32 +31 +30 +74 +42 +55 +90 +43 +49 +73 +65 +57

Table 32: Per-item case studies (External): for each model we surface the top discriminative items where the plausible-anchor shift cleanly exceeds the irrelevant- anchor shift, illustrating that the aggregate UAI pattern is driven by interpretable per-item behaviour rather than averaging artifacts. Full scenarios and per-condition answers are tabulated in the supplementary material. the plausible-anchor shift most clearly exceeds the irrelevant-anchor shift (Table 32). In each case the control answer tracks the evidence mean, the irrelevant-anchor answer barely moves, and the plausible-anchor answer jumps toward the anchor value—the same value in both conditions. A.17.13

LLM anchoring on the human effect-size scale

To position LLM anchoring against the human literature, we express the External-suite effect in |Cohen’s d|, averaging the absolute per-direction effect (pooling high and low anchors, whose signed shifts cancel), following the convention of Furnham & Boo (2011). Suite

|dpls | |dirr | |dhi-lo |

E XTERNAL H ISTORY RAG T OOL

0.35 0.24 0.16 0.23

0.11 0.19 0.09 0.12

0.59 0.46 0.28 0.36

Table 33: LLM anchoring magnitude in |Cohen’s d| (open-weight panel mean). Read against the conventional small / medium / large bands of d = 0.2/0.5/0.8. On External, the panel reaches |dpls | in the 0.2–0.5 range and |dhi-lo | in the 0.3–0.9 range: small to medium by the conventional bands, and reaching large for the high-versus-low contrast on the strongest models. The pattern |dpls | > |dirr | replicates across RAG, Tool, and History at smaller magnitudes. We deliberately stop short of a numerical head-to-head with human effect sizes. Human anchoring is normally measured between subjects on a single judgment, whereas UAI and the d values above are within-item contrasts on matched prompts, so the two are not on a 33

Published as a conference paper at COLM 2026

common scale, and the classical studies most often quoted for anchoring magnitude report medians, percentages, or F-statistics rather than a pooled d that can be reused here. What does transfer is the qualitative structure, and our data reproduce it: numbers that carry no task information still move judgments, plausibility modulates the size of the shift, and presentation format modulates it further—the same regularities reported by Tversky & Kahneman (1974), Strack & Mussweiler (1997), and Furnham & Boo (2011). For studies that do report effect sizes on comparable scales, Teovanović (2019) spans d = 0.14–1.00 across 24 items and Li et al. (2021) finds stronger anchoring for related than random anchors (r = 0.42 vs. 0.21), which is the same relevance ordering we measure. A paired study running humans and models on identical A NCHOR B ENCH items is the clean way to close this gap and is our primary follow-up.

34

Related documents

• When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair#303019
• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#1011704
• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#965480
• See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs#1129575
• If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs#673594