AMEL: Accumulated Message Effects on LLM Judgments Temkit Sid-Ali May 22, 2026
arXiv:2605.22714v1 [cs.AI] 21 May 2026
Abstract
eleven get a fair shake? Or has the accumulated history of rejections moved the goalposts? We test this directly. We present identical items either in a clean context (baseline) or after a conversation history loaded with predominantly “yes” or “no” evaluations, then measure whether the history shifts the model’s answer. We call this the accumulated message effect (AMEL). AMEL is not the same as sycophancy [22], where a model agrees with what the user thinks. In our setup, no user opinion is expressed; the model conforms to its own prior pattern. It also differs from the majority label bias in few-shot classification [33], where formatted examples with labels skew predictions. Our context items are full question-answer exchanges on varied topics within a domain, not label-formatted demonstrations. And unlike anchoring [14], there is no explicit numerical reference point; the bias emerges from the aggregate tone of the conversation. Our main findings:
Large language models are routinely used as automated evaluators: to review code, moderate content, or score outputs, often with many items passing through one conversation. We ask whether the polarity of prior conversation history biases subsequent judgments, an effect we call the accumulated message effect on LLM judgments (AMEL). Across 75,898 API calls to 11 models from 4 providers (OpenAI, Anthropic, Google, and four open-source models run locally), we present identical test items in isolation, or following histories saturated with predominantly positive or negative evaluations. Models shift their responses toward the conversation’s prevailing polarity (d = −0.17, p < 10−46 ). The effect is concentrated on items where the model is genuinely uncertain at baseline (d = −0.34 for high-entropy items, against d = −0.15 where the baseline is deterministic). Bias does not grow with context length: 5 prior turns and 50 produce essentially the same shift (Spearman |r| < 0.01, p > 0.94; linear-slope OLS p = 0.80). And there is a negativity asymmetry: per-item paired comparison shows negative histories induce 1.62× stronger bias than positive ones (t = 13.46, p < 10−39 , n = 2,481 pairs). Scaling helps but does not solve the problem (Anthropic: Haiku d = −0.22 → Opus d = −0.17; OpenAI: Nano d = −0.34 → GPT-5.2 d = −0.17). Three follow-up experiments narrow the plausible mechanisms. The effect is a continuous shift in the token probability distribution rather than a threshold artifact. The negativity asymmetry has both token-level and semantic components, though attributing the balance to either source is exploratory at the sample sizes we report. Positional explanations do not fit: five biased turns anywhere in a 50-turn history produce the same shift. For evaluation pipelines, the simplest fix is to give each item a fresh context; when batching is unavoidable, balancing the history helps.
1
1. AMEL appears across all four provider families tested. Across 11 models, conversation history pulls responses toward its prevailing polarity (d = −0.17, p < 10−46 ); 9 of 11 models show a significant effect after Bonferroni correction. 2. Susceptibility tracks the model’s own baseline uncertainty. Items where the model is genuinely uncertain at baseline (nonzero binary entropy of P (yes|baseline)) absorb ≈ 2× more bias than items where the baseline is deterministic (d = −0.34 vs. d = −0.15). The authorcoded “ambiguous” label predicts bias size only because some author-ambiguous items are also empirically uncertain; among items the model already finds clear-cut, author labels add little (Section 4.9). 3. Negative history biases more than positive history: paired per-item |BS| ratio 1.62× (t = 13.46, p < 10−39 , n = 2,481). Marginal means give ≈ 2× but mix item composition across cells; the paired test is the cleaner statistic.
Introduction
4. Bias saturates early. Five turns of skewed history produce the same shift as fifty (Spearman |r| < 0.01, p > 0.94; OLS slope −5 × 10−5 BS/turn, p = 0.80). The model recognizes the pattern fast and additional examples do not strengthen it.
Language models are being used as judges. They review code, moderate user-generated content, score exam responses, and rate creative outputs [34]. In production, efficiency often wins: multiple items get assessed in a single conversation, building up a history of prior judgments. This setup creates an obvious concern. If a code reviewer has just rejected ten pull requests in a row, does submission
5. Larger models within provider families show less bias, but none are immune. 1
Mean Bias Score
d=0.06 *
0.02
(a)
0.00
Ambiguity modulates the effect
0.04
(b)
d=0.13
Negativity asymmetry (2.6x)
0.00
0.02
0.02
0.02
0.00
0.04
0.04
0.02
0.06
0.06 0.08
0.06
0.08
0.10
0.08
0.10
0.10
d=-0.26 ***
Clear Positive
Bias saturates immediately
0.04
d=-0.14 ***
0.12
(c)
Ambiguous
Clear Negative
d=-0.26
r = 0.013 p = 0.499
0.12
d=-0.26
No-Sat
Neutral
Yes-Sat
5
10
20 Context length (turns)
50
Figure 1: Overview of AMEL. (a) Items where the model is uncertain at baseline absorb the most bias (d = −0.34); confidentbaseline items absorb less (d = −0.15). (b) Negative context biases models more than positive context (paired per-item ratio 1.62×, p < 10−39 ); marginal means yield ≈ 2× (Section 4.5). Even balanced history shifts models toward “no.” (c) Bias saturates immediately; 5 turns produce the same effect as 50. Conversational inertia and multi-turn degradation. Two concurrent papers investigate related phenomena from different angles. “Old Habits Die Hard” [23] offers a geometric analysis of how conversation history traps LLMs into consistent patterns via activation-space constraints. “Mitigating Conversational Inertia” [29] coins the term for agent settings and proposes mitigation strategies. Echterhoff et al. [9] study sequential cognitive biases (anchoring, framing, group attribution) in LLM decision-making, finding that the order and framing of preceding information systematically shifts outputs. Laban et al. [15] document broader multi-turn degradation at Microsoft Research, showing that model performance declines systematically as conversation length increases. We complement these mechanistic accounts with a large-scale empirical study: 11 models, 3 domains, systematic variation of context length and polarity.
6. The effect is detected in all three tested domains, all significant after Bonferroni correction: code review (d = −0.24) > meals (d = −0.13) > content moderation (d = −0.12). 7. Three characterization experiments (Section 5) narrow the plausible mechanisms but do not isolate a single cause: the probability distribution shifts continuously rather than crossing a threshold, the negativity asymmetry has both token-level and semantic components (per-model attribution is exploratory at n = 21 items), and position within the conversation does not matter. Figure 1 summarizes the three core findings.
2
Related Work
Sycophancy. The tendency of LLMs to agree with users has received substantial attention [22, 11, 12]. Perez et al. [19] evaluated sycophancy at scale using model-written tests, finding it pervasive across model families. Shapira et al. [21] provide a formal analysis of the amplification mechanism: RLHF increases sycophancy when sycophantic responses are overrepresented among high-reward completions under the base policy. AMEL operates differently: there is no user opinion to agree with. The model follows its own response history. That said, Jain et al. [12] note that interaction context increases sycophancy, so the two effects may compound in practice, and Shapira et al.’s RLHF mechanism may also explain why models follow their own prior patterns. Panickssery et al. [18] show a related self-preference bias: LLM evaluators favor their own generations, suggesting that models are generally susceptible to self-referential feedback loops.
In-context learning bias. Zhao et al. [33] showed that fewshot prompting introduces majority label bias and recency bias. Tang et al. [26] found these biases persist across model scales; Agarwal et al. [1] showed that many-shot learning can overcome some of them. Our work extends this line into multi-turn conversation, where the “demonstrations” are not formatted label examples but complete dialogues. Position and order bias. Wang et al. [30] demonstrated that LLM evaluators exhibit strong position bias: swapping the order of candidate answers can change win rates by up to 75%. Liu et al. [16] showed that models attend disproportionately to information at the beginning and end of long contexts, degrading performance on middle content. Chowdhury [7] recently argued that this U-shaped attention curve is an inherent geometric property of causal decoders, present at initialization before any training. AMEL can be understood as a related phenomenon: the prevailing polarity of the conversation, concentrated at the primacy and recency positions, overwhelms the individual item signal.
Negativity in binary judgments. Several studies document a “no” tendency in LLM binary decisions. Braun [4] finds LLMs lean toward “no” in English, the opposite of human ac2
quiescence bias. Cheung et al. [6] document yes-no response bias in LLMs using cognitive science paradigms (PNAS), confirming the phenomenon across multiple model families and tracing it to RLHF fine-tuning. Yu et al. [32] traces this to attention score patterns; Song et al. [24] shows chain-ofthought amplifies it; Lu et al. [17] finds binary formats produce more negative judgments than continuous scales. Jiang et al. [13] demonstrate a related surface-level effect: changing tokens while preserving logical content shifts model outputs, establishing that token-level preferences can operate independently of semantic reasoning. Our negativity asymmetry fits this picture: negative history reinforces an existing tendency, while positive history must push against it.
3
Methodology
3.1
Experimental Design
We use a between-subjects design. The same test items appear under different context conditions, and we measure whether the context shifts the model’s response. Conditions. Each test item is evaluated under four conditions (Figure 2): • Baseline: The item follows only the system prompt. No conversation history. • No-saturated: The item follows N prior turns where 90% of evaluations resulted in “no” and 10% in “yes.”
Threshold priming. Chen et al. [5] demonstrated that in batch relevance assessment, the quality of previously judged documents shifts the threshold for subsequent ones. Our work extends this to multi-turn conversation with controlled polarity and systematic model comparison. Hankache et al. [10] examined multi-turn sensitivity across GPT, Claude, and Gemini, finding effects consistent with ours and reporting accuracy degradation of up to 73% when items are evaluated within accumulated prior context rather than in isolation, supporting our primary recommendation.
• Yes-saturated: Same structure, but 90% “yes” and 10% “no.” • Neutral: N prior turns with a 50/50 split. The 10% minority responses in saturated conditions serve two purposes: they prevent trivially uniform histories, and they better approximate realistic evaluation sequences where not every item receives the same verdict. Context length. We vary N ∈ {5, 10, 20, 50} to test whether bias grows with exposure.
LLM-as-judge and cognitive bias. Ye et al. [31] taxonomizes 12 bias types in LLM judges (position, verbosity, self-enhancement, and others). Sumita et al. [25] surveys anchoring and priming effects; Shaikh et al. [20] proposes evaluation frameworks for them. Vo et al. [28] introduces B- SCORE, a response-history-based de-biasing method that is operationally similar but conceptually inverse: they compute a score from the model’s prior responses to detect when the next response is likely biased, and use that score to correct or veto the output. The mechanism we document is what makes their method possible: AMEL is the bias-creation channel (P (r|history) deviates from P (r|fresh)); B-score is a bias-detection consumer of that same channel. The two are not double-counting. A deployment that adopts our primary recommendation (fresh contexts) eliminates the channel and obviates the need for B-score. A deployment that requires batched evaluation for cost reasons can use B-score as a runtime corrector against the residual drift. Similarly, the logprobs-based response-bias correction of Bhatt and Ivanova [3] operates on the same continuous probability shift we document in Section 5.1 and would be a natural complement when fresh contexts are infeasible.
Repetitions and randomization. Each condition runs 10 times at temperature T = 1.0. We chose 10 repetitions based on pilot testing: at T = 1.0, baseline responses showed 95.9% majority agreement across repetitions, indicating that 10 trials provide adequate signal for detecting shifts of the magnitude we observe while keeping API costs tractable (≈76K calls total in the main experiment). Context items are sampled and shuffled with a deterministic seed per condition to control for order effects.
3.2
Evaluation Domains
We test three binary evaluation domains that span different judgment types: Code review. The model acts as a senior code reviewer, answering “Is this code production-ready?” Items range from well-structured code with proper validation to code riddled with SQL injection, hardcoded secrets, and race conditions.
Drift equilibria. Dongre et al. [8] show that context drift in multi-turn interactions stabilizes at equilibria rather than growing indefinitely. Our flat accumulation curve (Section 4.6) is consistent with this framework.
Content moderation. The model answers “Is this comment appropriate?” for forum comments ranging from constructive discussion to hate speech, threats, and doxxing. 3
Baseline (Control)
Biased Context (Treatment)
System Prompt
System Prompt
Test Item
N turns of skewed history
Response
Same Test Item
(90% "no" or 90% "yes")
Bias Score = P(no|treatment) - P(no|baseline)
Response (biased?)
Figure 2: Experimental design. Baseline condition (left): test item follows only the system prompt. Treatment condition (right): N polarity-skewed turns precede the same test item. The bias score captures the shift in P (no) between conditions. Nutritional assessment. The model answers “Is this a healthy choice?” for meals ranging from grilled salmon with vegetables to deep-fried butter sticks with soda. Each domain provides 55 clearly positive and 55 clearly negative items for constructing context histories, plus 21 test items in three categories: • Clear positive (7 items): Unambiguously good. Expected answer: “yes.” • Ambiguous (7 items): Genuinely borderline cases. We code ground truth as “yes” to enable directional measurement, but these items are designed so that either answer is defensible. See Section 9 for discussion of this coding choice.
Model (paper label) API model string
OpenAI
GPT-4.1 Nano GPT-5.2
gpt-4.1-nano gpt-5.2
8,190 8,190
Anthropic
Haiku 4.5 Sonnet 4.6 Opus 4.6
claude-haiku-4-5-20251001 claude-sonnet-4-6 claude-opus-4-6
8,190 8,190 8,190
Gemini 2.5 Flash Gemini 2.5 Pro
gemini-2.5-flash‡ gemini-2.5-pro§
1,470† 990†
Local (OSS)
Llama 3.2 3B Qwen3 4B Qwen3.5 4B Qwen3 30B
llama3.2:3b (Ollama) qwen3:4b (Ollama) qwen3.5:4b (Ollama) qwen3:30b (Ollama)
8,190 8,190 8,190 7,918¶
Table 1: Models tested with exact API identifiers. n = unique-condition API calls per model after deduplication. †Gemini models have partial data due to daily API quota limits. ‡thinkingBudget=0. §thinkingBudget=256 (minimum the API permits). ¶Qwen3 30B has 8,190 conditions scheduled; 7,918 distinct conditions were completed before a disk-full crash and post-resume; 2,186 duplicatecondition rows from a concurrent-resume bug were removed (see Appendix N). Data collected between 2026-03-09 and 2026-03-22.
• Clear negative (7 items): Unambiguously bad. Expected answer: “no.” Item categorization was determined by the author with independent verification. The author wrote initial category labels, then re-classified all 63 items (21 per domain) blind to the original labels after a one-week interval; intra-rater agreement was 95% (60/63 items). The three disagreements, all on the boundary between “ambiguous” and “clear,” were resolved by re-examination and a written justification recorded in the repository.
(the lowest setting the API permits). Local models run via Ollama.
3.4 3.3
n
Provider
Models Tested
Bias Score
For each test item i, model m, polarity p, and context length l, we define the bias score as:
We test 11 models from 4 providers, covering a range of scales (Table 1). All API models use temperature T = 1.0 and a 100-token output cap. Gemini 2.5 Flash runs with thinking disabled; Gemini 2.5 Pro uses a minimal thinking budget of 256 tokens
BSi,m,p,l = P (r∗ | treatmentp,l ) − P (r∗ | baseline) (1) where r∗ is the target response for the polarity condition: r∗ = “no” for no-saturated and neutral conditions, r∗ = “yes” 4
for yes-saturated. P (r∗ | ·) is the fraction of target responses across 10 repetitions. A positive bias score means the model shifted toward the saturated polarity (conforming); a negative score means it shifted away.
3.5
Finding Overall effect High-entropy items Negativity asymmetry Accumulation Scaling (Anthropic)
Key number
Table 2: Key findings at a glance. d = Cohen’s d; p = Bonferroni-corrected where applicable. The unparseable rate differs across conditions (χ2 = 97.7, p < 10−20 , df = 3): baseline 10.8%, no-saturated 7.0%, yes-saturated 8.2%, neutral 7.7%. We report the differential rate explicitly and discuss its MAR/MNAR implications in Appendix K; the headline conclusions are robust to per-model exclusion of Opus (which contributes most unparseable rows). Each observation aggregates 10 repetitions of a unique item × model × polarity × context-length combination against its baseline. Baseline consistency across repetitions averages 97.1%, confirming that T = 1.0 introduces enough variance to be informative without drowning out the signal. Table 2 gives the headline numbers.
Statistical Analysis
Sample size justification. With 7,631 bias score observations, the marginal one-sample t-test treating each observation as independent has >99% power to detect d = 0.10 at α = 0.05/21. This is an overstatement because observations are nested 10-deep within (model, item, condition) baselines; the bias-score construction collapses each 10-rep cell to a single BS value (so the 7,631 figure is already at the cell level, not the per-call level), but cells within the same item or model remain correlated. The mixed-effects model in Section 4.10 estimates an ICC of 0.031 between models; using the Kish design effect 1 + (m − 1) ICC with the average cluster size m = 693 per model gives an effective neff ≈ 7,631/22.5 ≈ 339. At neff = 339, 80% power detects d ≈ 0.15 at α = 0.05/21, which is still below the observed overall d = 0.17. The smallest per-model subsample (Gemini Pro, n = 402 cells) provides ≈ 80% power to detect d = 0.20 at the corrected α, close to the smallest significant per-model effect we report. We test BS = 0 (no bias) with one-sample t-tests for each ¯ BS ). Configrouping. Effect sizes are Cohen’s√ d (d = BS/s ¯ ± 1.96 · sBS / n. We apply Bonferroni dence intervals: BS correction with a factor of 21, matching the count of primary group-level tests we report (1 overall + 11 per-model + 3 per-polarity + 3 per-domain + 3 per-category = 21). The same factor is applied consistently to secondary analyses for comparability. The negativity asymmetry is tested with a paired t-test on absolute bias scores per item. Accumulation is assessed via Spearman rank correlation between context length and bias score.
4
p
Response Parsing
We extract binary yes/no judgments with a multi-strategy parser: first checking the opening word for explicit yes/no, then applying regex patterns to the first sentence, and falling back to position-weighted keyword counting. Responses that resist parsing are excluded (unparseable rate: 8.3%, concentrated in local models that occasionally produce malformed output).
3.6
d
¯ = −0.052 −0.17 < 10−46 BS −0.34 < 10−9 Hardest hit — < 10−39 paired 1.62× — 0.94 Spearman |r| < 0.01 −0.22 → −0.18 → −0.17
4.1
The Effect Is Real
¯ = −0.052 (95% CI: The overall bias score is BS [−0.059, −0.045]; t(7630) = −14.69, p < 10−46 , d = −0.17). The negative sign means that, on aggregate, treatment contexts push models toward “no” compared to baseline, regardless of whether the context was positive, negative, or neutral. Two things drive this aggregate negative direction. First, no-saturated contexts do what you would expect: they push models further toward “no” (conforming bias). Second, yessaturated contexts fail to push models equivalently toward “yes” (the negativity asymmetry we unpack in Section 4.5).
4.2
Bigger Models, Less Bias (Usually)
Figure 3 shows susceptibility across all 11 models. Nine out of eleven show significant bias after Bonferroni correction. The most susceptible models are GPT-4.1 Nano (d = −0.34), Llama 3.2 3B (d = −0.32), and Gemini 2.5 Pro (d = −0.27). The least affected (toward conformity) are Qwen3.5 4B (d = −0.08, pcorrected = 0.46, n.s.) and Qwen3 30B (d = +0.10, pcorrected = 0.17, n.s. under first-occurrence dedup); among API models, GPT-5.2 and Opus 4.6 tie at d = −0.17.
Results
Scaling trends. Within Anthropic: Haiku 4.5 (|d| = 0.22) > Sonnet 4.6 (|d| = 0.18) > Opus 4.6 (|d| = 0.17), all with negative sign (conformity to context). Within OpenAI: Nano (d = −0.34) > GPT-5.2 (d = −0.17), a 2× reduction. Bigger models are harder to sway, but none reach zero.
We collected 75,898 API responses (78,084 raw responses; 2,186 duplicate-condition rows from a concurrent-resume bug on the Qwen3 30B run were deduplicated, keeping the first occurrence per condition; see Appendix N). After dropping unparseable outputs (7.85%, concentrated in Claude Opus 4.6 which produces verbose conditional responses that resist binary extraction), 7,631 bias score observations remain.
Gemini reverses the trend. Gemini 2.5 Pro (d = −0.27) shows more bias than Flash (d = −0.18), not less. Pro’s 5
Model Susceptibility to AMEL Qwen3 4B
d=0.21 ***
Qwen3 30B
d=0.05 ns
Qwen3.5 4B
d=-0.08 ns
GPT-5.2
d=-0.17 ***
Opus 4.6
d=-0.17 ***
Sonnet 4.6
d=-0.18 ***
Gemini Flash
d=-0.18 ***
Haiku 4.5
d=-0.22 ***
Gemini Pro
d=-0.27 ***
GPT-4.1 d=-0.34 *** nano
OpenAI Anthropic Google Local (OSS)
Llama3.2 d=-0.32 *** 3B
0.15
0.10
0.05
0.00
0.05
Mean Bias Score (conforming <-- | --> contrarian)
0.10
Figure 3: Bias susceptibility by model. Bars: mean bias score (negative = conforming, positive = contrarian). Error bars: 95% CIs. ∗∗∗ : p < 0.001 after Bonferroni correction.
Qwen3 4B goes the other way. Qwen3 4B (d = +0.19, p < 10−6 ) is the only model with significant contrarian bias after Bonferroni correction: it shifts opposite to the conversation polarity. This looks like overcorrection, possibly from instruction tuning that penalizes repetitive patterns. The larger Qwen3 30B trends in the same direction but does not reach significance under the conservative first-occurrence dedup (d = +0.10, pcorr = 0.17); alternative dedup strategies (last-occurrence, random selection) give d = +0.17 to +0.22 with p < 10−5 (Appendix N), so the contrarian pattern may persist at scale within this model family but our published per-model estimate is conservative.
¯ = −0.052, d = −0.16, p < 10−13 . Clear negatives fect: BS ¯ = +0.005, d = +0.03, pcorrected = 1.00. do not move: BS Section 4.9 re-runs this analysis with empirical baseline uncertainty as the stratifier rather than author labels, and finds the underlying pattern is even sharper: the effect on items the model is genuinely uncertain about is twice as large as the effect on items it already calls confidently. The interpretation is straightforward. When a model’s internal confidence is high, the contextual signal cannot override it. When confidence is low, as it should be for genuinely ambiguous items, context tips the scale. This mirrors anchoring in human judgment, where uncertain estimates are most susceptible to external cues [27]. The practical upshot is unwelcome: the items most vulnerable to AMEL are exactly the ones where you most need an unbiased evaluation. A code reviewer that handles obvious bugs fine but becomes unreliable on borderline pull requests is not doing its job.
4.3
4.4
mandatory thinking tokens (minimum 256) may give it more opportunity to attend to conversation patterns, amplifying rather than damping the effect. But the Gemini sample sizes are small (990–1,470 calls vs. 8,190 for other models), so we flag this result as tentative.
Ambiguity Is the Weak Spot
Item difficulty modulates the effect (Figure 4). Author¯ = −0.109, coded ambiguous items take the biggest hit: BS −42 d = −0.28, p < 10 . Clear positives show a moderate ef-
Domain Differences
The effect size varies across domains (Figure 5). Code review ¯ = −0.106, d = −0.24, p < 10−31 ), is most affected (BS ¯ = −0.024, d = −0.13, p < 10−9 ) followed by meals (BS 6
AMEL by Test Item Ambiguity
0.00
0.02
0.02
0.04
Mean Bias Score
Mean Bias Score
AMEL by Evaluation Domain 0.00
d = 0.06 *
0.02
0.04 0.06
d = -0.14 ***
0.08
d = -0.07 *
d = -0.10 ***
Content Moderation
Meal Evaluation
0.06 0.08 0.10
0.10
d = -0.24 ***
0.12
d = -0.26 ***
0.12 Clear Positive (should be 'yes')
Ambiguous (borderline)
Code Review
Clear Negative (should be 'no')
Figure 5: Bias by domain. Code review shows the largest effect (d = −0.24); content moderation the smallest. Error bars: 95% CIs.
Figure 4: Bias by author-coded test item category. Ambiguous items absorb the most bias (d = −0.28); clear negatives do not move (d = +0.03, n.s.). Section 4.9 shows the same pattern but sharper when items are stratified by empirical baseline uncertainty rather than author label. Error bars: 95% CIs.
The asymmetry aligns with the negativity literature [4, 32]. If models already lean toward “no” in binary tasks, negative context reinforces this pull; positive context must fight against it.
¯ = −0.030, d = −0.12, and content moderation (BS pcorrected < 10−7 ). All three are robustly significant after Bonferroni correction; the magnitude range is roughly 2× from code review to content moderation, so “replicates” should be read as “detected in all three with varying magnitude” rather than “uniform.” Why code review? Possibly because “production-ready” is a fuzzier standard than “appropriate comment” or “healthy meal.” Content moderation norms are relatively well-defined in training data; nutritional rules less so but still grounded. Code quality involves more subjective trade-offs, leaving more room for context to matter. Code review prompts are also longer and more complex, which may interact differently with the context window.
4.5
4.6
Five Turns Is Enough
Does more history mean more bias? No. The Spearman correlation between context length and bias score is essentially zero (overall rs = +0.001, p = 0.94). Because Spearman over only four distinct x values {5, 10, 20, 50} is statistically blunt, we also fit an OLS regression of BS on log2 (context length) and on raw context length: both slopes are essentially zero (linear β̂ = −5×10−5 per turn, p = 0.80, R2 < 0.001). Within polarities the picture is similar: nosaturated rs = 0.019 (p = 0.35), yes-saturated rs = 0.029 (p = 0.15). Neutral shows a tiny but significant negative slope (rs = −0.049, p = 0.013), meaning balanced history ¯ slightly more negative with length. The effect is drifts BS of magnitude ∼ 0.05 across 45 additional turns, well below the noise floor of the main experiment. Five turns of biased history produce the same shift as fifty (Figure 7).
Negative History Hits Harder
The asymmetry between polarities is robust (Figure 6). No¯ = −0.107 (d = saturated contexts shift models by BS ¯ = −0.32). Yes-saturated contexts shift models by BS +0.049 (d = +0.19). Comparing marginal means gives a ratio of ≈ 2.2×, but this confounds item composition between the two cells. The cleaner statistic is a paired per-item comparison: mean |BSno | = 0.166 vs. |BSyes | = 0.102, ratio 1.62× (t = 13.46, p < 10−39 , n = 2,481 pairs). We report the paired result as the headline statistic throughout. An unexpected result: the neutral condition (50/50 history) ¯ = −0.097, d = −0.32), also produces a negative shift (BS close in magnitude to the no-saturated condition. Apparently, any conversation history, even balanced, pushes models toward “no” compared to a fresh start. One reading is that the presence of prior evaluation context triggers a more critical mode, independent of its content.
A gradual Bayesian-updating account does not fit. The model is not slowly accumulating evidence for a particular pattern; it detects “this conversation has been mostly rejections” within the first few turns and reaches a plateau. Dongre et al. [8] describe a similar equilibrium phenomenon in multiturn interactions, and our data look exactly like their predicted stabilization pattern. The practical implication: if you have any biased history, you already have the full bias. Adding more context does not make things worse, but it does not help either. 7
ence gives t = 19.10, p < 10−78 , d = 0.54 (Appendix B). This is a more nuanced pattern than pure assimilation. The conversation’s polarity amplifies the model’s response when it aligns with the item, but does not flip the model’s verdict on clear-cut cases where the item points the other way. The threshold-priming account of Chen et al. [5], where exposure to high-quality documents raised the bar for subsequent ones, is a related cumulative-history effect; ours operates at the item–context congruence level rather than across the history. The practical implication is unchanged. Items where the model is uncertain are the ones that conform, and they are the same items that absorb the most bias in Section 4.9.
Negativity Asymmetry: "No" Bias Is Stronger d = 0.13
0.04
Mean Bias Score
0.02 0.00 0.02 0.04
2.6x stronger
0.06 0.08 0.10
d = -0.26 d = -0.26
No-Saturated (90% no history)
Neutral (50/50 history)
Yes-Saturated (90% yes history)
4.9 Empirical Uncertainty Predicts Susceptibility
Figure 6: Polarity asymmetry. Negative context induces stronger bias than positive context (paired per-item ratio 1.62×, p < 10−39 ; marginal means give ≈ 2.2×). Even balanced (50/50) history produces a negative shift. Error bars: 95% CIs.
4.7
Section 4.3 stratified items by author-coded label. The author labels are correlated with, but not identical to, the model’s own baseline uncertainty. Here we re-stratify by the empirical baseline entropy of P (yes|baseline) across 10 repetitions per (model, item) pair, H = −p log2 p − (1−p) log2 (1−p). The entropy distribution is bimodal: most (model, item) pairs have H = 0 (the model is deterministic at baseline), and a long tail spreads up to H ≈ 1. We split into three bins: B1 deterministic baseline (H = 0), B2 low uncertainty (0 < H ≤ 0.76, the median of nonzero entropies), and B3 high uncertainty (H > 0.76). The empirical stratification gives a sharper picture than the author labels (Table 3). B3 items show d = −0.34, more than twice the effect on B1 items (d = −0.15). Across the entire |BS| distribution, Spearman r = 0.47 between baseline entropy and |BS| (p < 10−300 ).
Model × Context Length
The full model×context-length heatmap (Appendix A) confirms the no-accumulation finding at every level: GPT-4.1 Nano and Llama 3.2 3B run hot throughout; Qwen3 4B’s contrarian pattern is visible at every length; and no model shows a vertical gradient.
4.8 Assimilation for Congruent Items, Contrast for Incongruent Does AMEL work by assimilation (pulling the model toward the prevailing polarity) or contrast (pushing it away from the polarity for items that contradict the context)? We classify each bias observation as congruent (the context polarity matches the item’s ground truth, e.g., no-saturated context with a clear-negative item) or incongruent (the opposite). Recall that the bias score is defined in the direction of the saturated polarity (Section 3.4), so a positive BS means “shifted toward the saturated polarity” (i.e., conformed) and a negative BS means “shifted away from it” (i.e., resisted). The two regimes go in opposite directions. For congru¯ = +0.051 (95% CI ent items, the model conforms: BS [+0.041, +0.061]), an assimilation effect, where the saturated polarity reinforces the response the item already calls ¯ = −0.110 for. For incongruent items, the model resists: BS (95% CI [−0.123, −0.097]). We use the term contrast for this pattern in the section title and elsewhere because it is the conventional umbrella term for context-opposite shifts, but the mechanism is closer to item-level anchoring than to classical psychophysical contrast: the model does not overcorrect past baseline, it simply stays near the item’s own evidence and refuses to be pulled toward the context. A two-sample t-test on the magnitude of the congruent–incongruent differ-
Bin
n
¯ BS
d p
B1 det. baseline (H=0) 5,591 −0.041 −0.15 < 10−200 B2 low uncertainty (0<H≤0.76) 384 −0.074 −0.18 < 10−3 B3 high uncertainty (H>0.76) 348 −0.127 −0.34 < 10−9
Table 3: Empirical-entropy stratification. Each bias-score observation is grouped by the binary entropy of P (yes|baseline) across 10 reps. The three bins sum to 6,323 observations, not the full 7,631: observations from (model, item) pairs where fewer than 5 baseline reps parsed cleanly are dropped from this analysis. The author-label crosstab (Appendix M) shows that 81% of author-coded “ambiguous” items are actually in B1 (model is confident at baseline), so the entropy strata do not coincide with the author labels. Two observations follow. First, the AMEL effect appears on items the model is confident about (B1 , d = −0.15), not only on items it is uncertain about, so AMEL is not purely a decision-boundary artifact. Second, the effect is roughly twice as large on items the model is genuinely uncertain about, consistent with a mechanism where the conversational signal contributes more weight to the final decision when the model’s own internal evidence is weak. A mixed-effects 8
Bias Does Not Accumulate with Context Length No-Saturated Context
Yes-Saturated Context
Bias Score
0.05 0.00 0.05 0.10
r = 0.013 p = 0.499
5
10
20
Context Length (# prior turns)
50
r = 0.030 p = 0.129
5
10
20
Context Length (# prior turns)
50
Figure 7: Bias does not grow with context length. Mean bias score is flat across N = {5, 10, 20, 50}. Error bars: 95% CIs; r and p: Spearman correlation. model (Section 4.10) confirms these patterns while accounting for the non-independence of observations within models.
4.10
logprobs across five conditions: baseline, no-saturated at 5 and 50 turns, and yes-saturated at 5 and 50 turns (1,050 calls). The probability distribution moves, not just the binary outcome (Figure 8). Both polarities shift P (Yes) upward from the model’s extreme baseline (P (Yes) = 0.13), with the magnitude varying by polarity. The per-item dot plot (Figure 8b) shows this is not driven by a few outlier items; the shift is consistent across the board. Saturation holds at the probability level too. P (Yes) at 5 and 50 turns is similar for both polarities: the no-saturated comparison is borderline (Mann-Whitney U , p = 0.049) and the yes-saturated comparison is not significant (p = 0.067). Neither is far from the saturated-history mean, consistent with rapid plateau rather than continued drift; the borderline nosaturated p should be read as suggestive that any residual drift is small. Because the bias is a continuous probability shift, logprobs-based correction methods [3] could in principle detect and compensate for it in evaluation pipelines.
Mixed-Effects Confirmation
To account for the hierarchical structure of our data (observations nested within models), we fit a linear mixed-effects model: BS ∼ polarity × category with random intercepts for model. The model confirms all major findings: the polarity×category interaction is significant (p < 0.001), with yes-saturated×clear-negative showing the strongest interaction (β = −0.196, z = −9.88, p < 10−22 ). The betweenmodel variance (ICC) accounts for a small fraction of total variance, indicating that AMEL is a universal phenomenon modulated by, but not unique to, specific models.
5
Characterizing AMEL
The preceding sections establish that AMEL exists. This section narrows the plausible mechanisms with three targeted experiments (3,570 additional API calls) and one analysis of the existing data. We are deliberate about the language: the experiments distinguish between specific mechanistic hypotheses, but they do not in isolation prove a single mechanism. We use “consistent with X” to describe each result and clearly label the per-model attribution in Section 5.2 as exploratory at the available sample sizes.
5.2 Negativity Has Both Token and Semantic Sources Why is the negative direction stickier? Two hypotheses: (1) the “no” token is inherently preferred, perhaps because it is more common in training data; or (2) rejection as a concept is stickier, reinforced by RLHF safety training that rewards caution [2]. We disentangle them by flipping the question. Instead of “Is this code production-ready?” (yes = approve, no = reject), we ask “Should this code be rejected?” (yes = reject, no = approve), using the same code items. Now a no-saturated history is full of approvals, and a yes-saturated history is full of rejections. If the token drives the asymmetry, no-saturated should still be stronger. If the semantic frame drives it, yes-saturated (rejection) should dominate. We run GPT-4.1 Nano and Llama 3.2 3B at 10 turns (1,260 calls). The answer depends on the model (Figure 9). GPT-4.1
5.1 The Probability Distribution Shifts Continuously The response flips we measure could be a threshold artifact: the model’s internal distribution barely changes, but a small nudge crosses the yes/no boundary. To test this, we run GPT-4.1 Nano on the code review domain with first-token 9
Logprobs: Probability Distribution Shifts Under AMEL (a) P(Yes) Shifts Continuously 0.391
Mean P(Yes) from First Token
(b) Per-Item P(Yes) Shifts 0.463
0.5
0.4
CN7 CN6 CN5 CN4 CN3 CN2 CN1 A7 A6 A5 A4 A3 A2 A1 CP7 CP6 CP5 CP4 CP3 CP2 CP1
0.372
0.3 0.224
0.2 0.126
0.1
0.0
Baseline
Yes-sat @5
Yes-sat @50
No-sat @5
No-sat@5 Yes-sat@5
0.2
No-sat @50
0.0
0.2
0.4
0.6
Shift in P(Yes) from Baseline
0.8
1.0
Figure 8: Logprobs analysis. (a) Mean first-token P (Yes) shifts systematically across conditions, with magnitude varying by polarity. (b) Per-item shifts show the probability distribution moves consistently, not just the binary outcome.
Negativity Asymmetry: Original vs. Flipped Framing Token
Nano keeps the same direction: no-saturated is stronger in both framings (ratio 1.41 → 1.12), a pattern consistent with a token-level preference, in line with Jiang et al. [13]’s finding that surface tokens shift outputs independent of semantics. Llama 3.2 3B nearly reverses (ratio 1.11 → 0.98), a pattern consistent with a semantic effect of the rejection concept, in line with Shapira et al. [21]’s analysis of how RLHF amplifies pattern-following and Cheung et al. [6]’s evidence that finetuning creates the negativity bias. We emphasize that the permodel differences in the asymmetry ratio between original and flipped framings are not individually significant (Nano paired |BS| test p = 0.35 original, p = 0.76 flipped; Llama p = 0.64 original, p = 0.91 flipped; n = 21 items each), so these mechanistic attributions are exploratory: the experiment narrows the plausible mechanisms but does not isolate them. Both token-level and semantic accounts remain on the table, with the balance plausibly varying by model. An analysis of the existing data adds a third factor. Items with higher baseline P (no) show modestly stronger negativity asymmetry (Pearson r = 0.22, p < 10−7 ; Spearman r = 0.08, p = 0.06, n.s.; n = 593 item×model pairs). The Pearson–Spearman split suggests the relationship is driven by a few high-leverage items rather than a monotone rank pattern, so we treat it as suggestive rather than confirmatory. Negative context does not create a tendency from scratch. It reinforces one the model already has.
|BS(no_sat)| / |BS(yes_sat)|
1.4 1.2
No asymmetry Original (Is this production-ready?) Flipped (Should this be rejected?)
RLHF
1.0 0.8 0.6 0.4 0.2 0.0
GPT-4.1 nano
Llama3.2 3B
Figure 9: Asymmetry ratio (|BS(no-sat)| / |BS(yes-sat)|) under original vs. flipped question framing. A ratio >1 means nosaturated context produces stronger bias.
review, no-saturated, 10 reps (1,260 calls). Any five will do (Figure 10). The three placements are statistically indistinguishable (Kruskal-Wallis H = 0.19, p = ¯ = −0.36, END = −0.39, SPREAD 0.91): START BS = −0.38. These match both CONTROL_5 (−0.32, all 5 biased) and FULL_50 (−0.39, all 50 biased) from the main experiment. The model does not care where the signal is. Five biased turns out of fifty, in any position, suffice.
5.3 Position Is Irrelevant: Any Signal Suffices Five turns produce the same bias as fifty (Section 4.6), but which five? If the model relies on primacy, early turns should matter most. If recency, the end. If neither, any five will do. We embed 5 biased turns in 50-turn conversations at three positions: START (positions 0–4), END (45–49), and SPREAD (0, 12, 24, 36, 49). The other 45 turns are neutral (50/50). We run GPT-4.1 Nano and Llama 3.2 3B on code
6
Discussion
Why ambiguity matters. High-confidence items resist AMEL because the model’s internal prior is strong enough to override the conversational signal. Ambiguous items sit near the decision boundary, where a small nudge changes the 10
Position of Biased Turns Does Not Matter
Not a positional effect. Liu et al. [16] showed that models attend disproportionately to the beginning and end of long contexts, and Chowdhury [7] argued this U-shaped attention curve is a geometric property of causal decoders, present at initialization. One might expect AMEL to ride on these positional effects, with early turns setting the pattern or recent turns dominating. Our positional experiment (Section 5.3) rules this out: START, END, and SPREAD placements are indistinguishable. Whatever mechanism aggregates the polarity signal, it is not the same position-dependent attention that drives the U-shaped retrieval curve.
0.0 KW: H=0.2, p=0.911
Mean Bias Score
0.1
0.2
0.3 d=-0.85 d=-0.87
0.4
d=-0.89
d=-0.90
d=-0.92
End (5/50 biased)
Spread (5/50 biased)
Full (50/50 biased)
0.5 Control (5/5 biased)
Start (5/50 biased)
The contrarian exception. Qwen3 4B’s reverse bias is worth noting. Some instruction tuning penalizes repetition or rewards “balanced” responses, which could cause overcorrection when a model detects a lopsided conversation. The larger Qwen3 30B trends the same direction but does not reach significance under the published first-occurrence dedup (d = +0.10, n.s. corrected); alternative dedup choices give d = +0.17 to +0.22 (Appendix N). The picture is therefore consistent with a Qwen-family overcorrection that partially persists at the 30B scale rather than purely a small-model artifact.
Figure 10: Bias score by position of 5 biased turns within a 50-turn conversation. CONTROL_5 and FULL_50 from main experiment for reference. Error bars: 95% CIs. output. This is the same dynamic Tversky and Kahneman described for anchoring in human judgment: the less certain the judge, the more susceptible to irrelevant cues [27]. Why negativity is stickier. The flipped-framing experiment (Section 5.2) gives a nuanced answer. For GPT-4.1 Nano, the “no” token itself is sticky: flipping the question does not flip the asymmetry. For Llama 3.2 3B, the rejection concept matters more: the asymmetry nearly reverses under flipped framing. The baseline correlation (rPearson = 0.22, p < 10−7 ; rSpearman = 0.08, n.s.) adds a third ingredient, weakly: items where the model already leans “no” tend to be more susceptible to negative context, though the rank correlation is non-significant. So the negativity asymmetry is not one thing. It is a confluence of token-level preference, RLHF-trained caution, and pre-existing item-level tendencies, with the balance shifting across models.
Gemini Pro vs. Flash. The reversal in Gemini’s scaling trend (Pro is more biased than Flash) may relate to Pro’s thinking tokens. If the model explicitly reasons about the conversation so far, it may amplify context patterns rather than ignoring them. Flash, without the thinking overhead, may process each item more independently. This is speculative, though, and the small sample sizes for Gemini warrant caution.
7
Mitigation Experiments
The main experiment uses artificially constructed context histories. Do models bias themselves naturally when evaluating items sequentially? And does balanced ordering help? We test both questions with a sequential batch design.
Pattern detection, not evidence accumulation. The logprobs experiment (Section 5.1) confirms that this is not a threshold artifact: the model’s internal distribution genuinely shifts. The positional experiment (Section 5.3) goes further: it does not matter where in the conversation the biased turns appear. Neither primacy nor recency explains the effect and suggests the model aggregates polarity signal globally, consistent with the equilibrium framework of Dongre et al. [8].
7.1 Does Bias Emerge in Sequential Evaluation? We evaluate all 21 test items in a single conversation, where the model’s own answers to previous items form the context for subsequent ones. We test two orderings: fixed order (all clear-positive items first, then ambiguous, then clear-negative) and balanced order (interleaving expected-yes and expectedno items). We use three representative models (GPT-4.1 Nano, Llama 3.2 3B, Qwen3.5 4B) across all three domains, with 10 repetitions each. In the fixed-order condition, bias is not significant overall (d = −0.07, p = 0.35), but position analysis reveals dramatic drift: P(no) increases monotonically from 0.07 at the start (clear-positive items) to 1.0 by position 14 (Spearman
Assimilation vs. contrast. Our congruent/incongruent analysis (Section 4.8) reveals a two-regime pattern: models assimilate toward the conversation polarity for congruent items (where the context agrees with item ground truth) and contrast away from it for incongruent items (where the context disagrees). Pure assimilation does not fit; the model is not blindly following the conversation polarity but amplifying responses that align with the item. This regime split is distinct from the cumulative threshold-priming contrast of Chen et al. [5], which operates across the history rather than per item. 11
r = 0.86, p < 10−6 ). As the model evaluates items sequentially, its own response history creates an increasingly biased context. This is the same mechanism our main experiment isolates. In the balanced-order condition, the overall bias is significant (d = −0.46, p < 10−9 ), indicating a general negativity shift, but critically there is no position-dependent drift (r = −0.25, p = 0.28). Interleaving expected-yes and expected-no items prevents the model from establishing a consistent polarity pattern. The fixed-order vs. balanced-order difference is significant (t = 4.19, p < 10−4 ), confirming that ordering matters. Full details are in Appendix G.
7.2
4. Pair batched evaluation with a response-history bias detector or logprobs-based corrector. B-score [28] or logprobs correction [3] can detect and partially correct AMEL drift at runtime in deployments where fresh contexts are not viable. 5. Monitor for drift. Track the running accept/reject ratio in production evaluation pipelines. If a session’s ratio drifts far from the expected base rate, the conversation may be biasing itself.
9
Binary judgments only. We study yes/no tasks. Likertscale ratings, open-ended evaluation, or multi-label classification may exhibit different patterns.
Temperature Sensitivity
All main experiments use T = 1.0. To check whether temperature modulates AMEL, we run GPT-4.1 Nano on the code review domain at T = 0.3 and T = 0.7 (no-saturated + baseline, N = 10, all 21 items, 10 reps). We select GPT-4.1 Nano because it shows the strongest bias in the main experiment (d = −0.34), giving us the best chance to detect temperature effects. Contrary to the intuition that lower temperature should reduce bias, we find that bias is at least as strong at lower temperatures: d = −0.69 at T = 0.3 and d = −0.72 at T = 0.7, compared to d = −0.28 at T = 1.0 (all significant at p < 0.05). A Kruskal-Wallis test across the three temperatures is not significant (H = 5.27, p = 0.07), so we cannot claim a significant monotonic relationship, but the trend is clear: reducing temperature does not mitigate AMEL. This makes sense mechanistically. Lower temperature makes the model more likely to follow the dominant pattern, not less. Full details are in Appendix H.
8
Limitations
Three domains. Code review, content moderation, and nutritional assessment cover technical, social, and personal judgment, but we have not tested other high-stakes domains like medical triage, hiring, or academic grading. Ambiguous item coding. We code ambiguous items with a “yes” ground truth to enable directional bias measurement. This is a methodological convenience, not a claim about the correct answer. The key result, that ambiguous items show larger bias scores, holds regardless of which direction we call “correct,” since the bias score measures the shift from baseline rather than accuracy. Partial Gemini data. Daily API quotas limited Gemini to 1,470 calls (Flash) and 990 (Pro), vs. 8,190 for all other models. Gemini results have wider confidence intervals as a result.
Practical Recommendations
Limited temperature variation. Our main experiments use T = 1.0; the temperature spot-check (Section 7.2) covers only one model and one domain. A full factorial temperature×model design would strengthen the finding.
1. Fresh contexts by default, including for high-stakes calls. Evaluate each item in its own conversation, with no prior history. This costs more (no prompt caching) but eliminates the channel that creates AMEL. For consequential decisions (hiring, content moderation at scale, code review of safety-critical changes), the fresh-context cost is unambiguously the right trade.
Limited characterization coverage. The logprobs experiment uses only GPT-4.1 Nano (the only model with logprobs access at acceptable cost); the flipped-framing and positional experiments cover two models. The characterization findings should be verified across a broader set of models and domains.
2. Randomize and balance batches when fresh contexts are infeasible. If batching is required for cost reasons, shuffle item order and balance expected positive/negative outcomes within each batch. Do not sort by expected verdict; sorted orders generate self-reinforcing positional drift (Section 7.1).
English only. All materials are in English. Cross-linguistic differences in LLM response tendencies [4] mean our results may not transfer to other languages.
3. Protect uncertain items. The model’s own baseline uncertainty predicts susceptibility (Section 4.9). If fresh contexts for everything is impractical, at least use them for items where the model’s first-token logprobs are non-deterministic.
Correct context items. Our context conversations contain accurate evaluations (bad code correctly rejected, good code correctly approved). Contexts with incorrect prior judgments (a bad review that approves buggy code) might interact with sycophancy mechanisms and produce different effects. 12
model is genuinely uncertain at baseline (d = −0.34). It is asymmetric, with negative history producing a 1.62× larger paired per-item shift than positive history (p < 10−39 ). And it is immediate: five turns of biased history are enough; fifty do not make it worse. The characterization experiments (Section 5) narrow the plausible mechanisms: the probability distribution shifts continuously rather than crossing a threshold, the negativity asymmetry has both token-level and semantic components (model-dependent), and five biased turns anywhere in a 50turn conversation are sufficient to produce the full effect. These are convergent rather than definitive results; the experiments span 1–2 models and one domain (Section 9). Scaling helps. Bigger models within the same family show smaller effects. But even frontier models like GPT-5.2 and Opus 4.6 remain susceptible (d = −0.17). The takeaway for practitioners is simple: do not let evaluation history accumulate. Each item deserves a clean slate.
Single-model data-quality incident. The Qwen3 30B run experienced a disk-full crash with a concurrent-resume bug that produced 2,186 duplicate-condition rows; we deduplicate post-hoc by keeping the first occurrence per condition (Appendix N). This affects one of eleven models and does not change overall conclusions, but the dedup procedure is one more place where reviewer attention is warranted. Neutral arm uses evaluative filler. Our “neutral” (50/50) context arm is composed of evaluative items from the same yes/no pool, not non-evaluative filler. The finding that the neutral arm also produces a negative shift is therefore consistent with two interpretations: (i) any conversation history (evaluative or not) shifts models toward “no,” or (ii) any history composed of evaluative judgments shifts models toward “no,” regardless of polarity balance. Distinguishing these requires a non-evaluative-filler control we do not run here (see Future Work). Item categorization by single author. The clear/ambiguous/clear-negative labels were assigned by the author with intra-rater agreement (95%) but without inter-rater validation. The empirical-entropy stratification in Section 4.9 largely bypasses this concern by using the model’s own baseline behavior to group items, and the empirical-entropy result is the more robust version of the “uncertainty absorbs bias” finding. An independent inter-rater study would still strengthen the author-coded labels used elsewhere in the paper.
Reproducibility All code, data, and analysis scripts are available at https: //github.com/chutapp/amel. The complete dataset (75,898 deduplicated API responses) is provided in JSONL format with deterministic seeds for all random operations. Data were collected between 2026-03-09 and 2026-03-22. The exact API model strings used are listed in Table 1; provider aliases without dated suffixes (e.g., gpt-5.2, claude-opus-4-6) may receive silent updates from providers after the collection window.
Differential unparseable rates across conditions. The 7.85% unparseable rate is not independent of polarity (Appendix K): baseline 10.8%, no-saturated 7.0%, yes-saturated 8.2%, neutral 7.7%. Most of the differential is driven by Claude Opus 4.6 (22% unparseable across all conditions due to verbose conditional responses), which is roughly MAR. The headline conclusions reproduce if Opus is excluded from the model panel; we report this as a sensitivity check rather than a primary result.
Ethics Statement
Future work. Three follow-up studies would directly strengthen the claims here: (1) a non-evaluative neutral-arm control to distinguish “any history” from “any evaluative history”; (2) inter-rater agreement on the author-coded item categories with ≥ 2 external annotators; and (3) the flippedframing experiment (Section 5.2) extended across all 11 models to test whether the token-vs-semantic balance of the negativity asymmetry has a more robust per-model signature than the underpowered 2-model design here can detect.
This work studies systematic bias in AI evaluation systems with the goal of improving their reliability. Content moderation test items reference harmful content categories (hate speech, threats) as examples of clearly inappropriate material; these were designed to test model behavior and are not intended to promote or normalize such content. No human subjects were involved. All model interactions used standard commercial APIs and locally hosted open-source models; no models were fine-tuned or modified. We note that our findings could theoretically be used to deliberately bias LLM evaluators (by manipulating conversation history), but we believe the defensive value of documenting this vulnerability outweighs the offensive risk, particularly given the simplicity of the mitigation (fresh contexts).
10
References
Conclusion
AMEL appears across all four provider families we tested (9 of 11 models significant after Bonferroni correction). It is moderate in overall magnitude (d = −0.17), but it concentrates on the items where it matters most: items where the
[1] Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Chan, Ankush Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Manyshot in-context learning. In Advances in Neural In13
formation Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2404.11018.
[12] Shomik Jain, Charlotte Park, Matt Viana, Ashia Wilson, and Dana Calacci. Interaction context often increases sycophancy in LLMs. arXiv preprint arXiv:2509.12517, 2025.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[13] Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. URL https://arxiv.org/abs/2406.11050. Surface token changes shift model outputs independent of logical content; defines token bias vs. semantic reasoning.
[3] Om Bhatt and Anna A. Ivanova. RBCorr: Response bias correction in language models. arXiv preprint arXiv:2602.12445, 2026. LogProbs-based bias correction method; eliminates option preference bias in binary tasks. [4] Daniel Braun. Acquiescence bias in large language models. arXiv preprint arXiv:2509.08480, 2025.
[14] Erik Jones and Jacob Steinhardt. Anchoring bias in large language models: An experimental study. arXiv preprint arXiv:2412.06593, 2024.
[5] Nuo Chen, Jiqun Liu, Xiaoyu Dong, Qijiong Liu, Tetsuya Sakai, and Xiao-Ming Wu. AI can be cognitively biased: An exploratory study on threshold priming in LLM-based batch relevance assessment. 2024.
[15] Philippe Laban, Asli Celikyilmaz, Caiming Xiong, and Besmira Nushi. Lost in conversation: How multi-turn interactions degrade LLM performance. arXiv preprint arXiv:2502.01003, 2025. Microsoft Research; documents systematic performance degradation in multi-turn LLM use.
[6] Tsz Ting Cheung et al. Amplified yea-saying bias in large language models. Proceedings of the National Academy of Sciences (PNAS), 122(3), 2025. URL https://doi.org/10.1073/pnas. 2420642122. Documents yes-no response bias in LLMs using cognitive science paradigms.
[16] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics (TACL), 12:157–173, 2024. URL https://arxiv.org/abs/2307.03172. Models attend more to beginning and end of context, degrading on middle content.
[7] Borun D. Chowdhury. Lost in the middle at birth: An exact theory of transformer position bias. arXiv preprint arXiv:2603.10123, 2026. Proves U-shaped retrieval curve is inherent to causal decoder architecture; present at initialization before training. [8] Vardhan Dongre, Ryan A. Rossi, Viet Dac Lai, David Seunghyun Yoon, Dilek Hakkani-Tür, and Trung Bui. Drift no more? context equilibria in multi-turn LLM interactions. arXiv preprint arXiv:2510.07777, 2025.
[17] Yi-Long Lu, Chunhui Zhang, and Wei Wang. Systematic bias in large language models: Discrepant response patterns in binary vs. continuous judgment tasks. arXiv preprint arXiv:2504.19445, 2025. [18] Arjun Panickssery, Samuel R Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv. org/abs/2404.13076. NeurIPS 2024 Oral; selfpreference bias in LLM evaluators.
[9] Jessica M Echterhoff, Yao Liu, Miltiadis Allamanis, and Julian McAuley. Cognitive bias in decision-making with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. URL https://arxiv.org/abs/2403.00811. Studies sequential cognitive biases including anchoring and framing in LLM decisions.
[19] Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, 2023. URL https://arxiv.org/abs/2212.09251. Large-scale evaluation of sycophancy and other biases using model-written tests.
[10] Robert Hankache, Kingsley Nketia Acheampong, Liang Song, Marek Brynda, Raad Khraishi, and Greig A. Cowan. Evaluating the sensitivity of LLMs to prior context. arXiv preprint arXiv:2506.00069, 2025. [11] Jiseung Hong, Grace Byun, Seungone Kim, Kai Shu, and Jinho D. Choi. Measuring sycophancy of language models in multi-turn dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. URL https://arxiv.org/abs/ 2505.23840.
[20] Ammar Shaikh, Raj Abhijit Dandekar, Sreedath Panat, and Rajat Dandekar. CBEval: A framework for evalu14
ating and interpreting cognitive biases in LLMs. arXiv preprint arXiv:2412.03605, 2024.
[31] Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. arXiv preprint arXiv:2410.02736, 2024.
[21] Itai Shapira, Gerdus Benade, and Ariel D. Procaccia. How RLHF amplifies sycophancy. arXiv preprint arXiv:2602.01002, 2026. Formal analysis of how RLHF reward learning amplifies pattern-following; sycophantic responses overrepresented among high-reward completions.
[32] Sangwon Yu, Jongyoon Song, Bongkyu Hwang, Hoyoung Kang, Sooah Cho, Junhwa Choi, Seongho Joe, Taehee Lee, Youngjune L. Gwon, and Sungroh Yoon. Correcting negative bias in large language models through negative attention score alignment. arXiv preprint arXiv:2408.00137, 2025.
[22] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. In International Conference on Learning Representations (ICLR), 2024. URL https: //arxiv.org/abs/2310.13548.
[33] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving fewshot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. URL https://arxiv.org/abs/ 2102.09690.
[23] Adi Simhi, Fazl Barez, Martin Tutek, Yonatan Belinkov, and Shay B. Cohen. Old habits die hard: How conversational history geometrically traps LLMs. arXiv preprint arXiv:2603.03308, 2026.
[34] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. Judging LLMas-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 2024.
[24] Jongyoon Song, Sangwon Yu, and Sungroh Yoon. A multifaceted analysis of negative bias in large language models through the lens of parametric knowledge. arXiv preprint arXiv:2511.10881, 2025. [25] Yasuaki Sumita, Koh Takeuchi, and Hisashi Kashima. Cognitive biases in large language models: A survey and mitigation experiments. arXiv preprint arXiv:2412.00323, 2024. [26] Karan Tang, Duy Phung Chau, and Lisa Xiang. How robust are LLMs to in-context majority label bias? arXiv preprint arXiv:2312.16549, 2024. [27] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157): 1124–1131, 1974. [28] An Vo, Mohammad Reza Taesiri, Daeyoung Kim, and Anh Totti Nguyen. B-score: Detecting biases in LLMs using response history. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. URL https://arxiv.org/abs/ 2505.18545. [29] Yang Wan, Zheng Cao, Zhenhao Zhang, Zhengwen Zeng, Shuheng Shen, Changhua Meng, and Linchao Zhu. Mitigating conversational inertia in multi-turn agents. arXiv preprint arXiv:2602.03664, 2026. [30] Peiyi Wang, Lei Li, Liang Chen, Feifan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. URL https://arxiv.org/abs/ 2305.17926. Demonstrates position bias in LLMas-judge; swapping answer order changes win rates by up to 75%. 15
Model × Context Length Heatmap
A
Higher Uncertainty, Larger Bias Effect Clear Pos. Ambiguous Clear Neg. r = 0.461
1.0
0.8
-0.22
-0.24
-0.24
-0.26
GPT-5.2
-0.06
-0.07
-0.08
-0.08
Haiku 4.5
-0.14
-0.16
-0.19
-0.19
Sonnet 4.6
-0.11
-0.13
-0.14
-0.14
Opus 4.6
-0.06
-0.06
-0.08
-0.12
Gemini Flash
-0.12
-0.13
-0.13
-0.14
Gemini Pro
-0.21
-0.21
-0.16
-0.10
Llama3.2 3B
-0.27
-0.29
-0.31
-0.30
Qwen3 4B
0.12
0.23
0.27
0.25
Qwen3.5 4B
-0.05
-0.04
-0.04
-0.05
Qwen3 30B
-0.02
0.06
0.06
0.07
10
20
50
0.3 0.2
0.0
Context Length (# prior turns)
0.4
0.2
0.0
0.1
5
0.6
0.1
Bias Score
GPT-4.1 nano
|Bias Score|
Bias Score Heatmap (No-Saturated Context)
0.0
0.2
0.4
0.6
0.8
1.0
Baseline Entropy (model uncertainty)
0.2
Figure 13: Baseline entropy (model uncertainty) vs. |BS|. Higher uncertainty predicts larger bias effects (r = 0.47). Colors indicate item category.
0.3
C Baseline Entropy vs. Bias Susceptibility
Figure 11: Bias heatmap, no-saturated condition. Blue = conforming (more “no”); red = contrarian. Values are mean bias scores. No vertical gradient ⇒ no accumulation.
D
Scaling Ladder
Figure 14 shows effect sizes within each provider family.
B
Scaling Reduces AMEL Within Each Provider
Assimilation vs. Contrast
OpenAI
Anthropic
|Cohen's d| (effect size)
0.40
Assimilation vs. Contrast by Item Category Congruent Incongruent
0.10
0.35
|d| = 0.34
0.30
|d| = 0.27
0.25
|d| = 0.22
0.20
|d| = 0.17
|d| = 0.18
|d| = 0.17
|d| = 0.18
Sonnet
Opus
Flash
0.15 0.10 0.05 0.00
Nano
5.2 Flagship
Haiku
Pro
Mean Bias Score
0.05
Figure 14: Effect size (|d|) by provider family. OpenAI and Anthropic show consistent reduction with scale; Google reverses the trend.
0.00 0.05 0.10 0.15
E
0.20
Table 4 gives full results for all 11 models. Clear Positive
Ambiguous
Clear Negative
F
Figure 12: Two regimes. For congruent items (context polarity matches item ground truth), the model conforms to the ¯ > 0). For incongruent items (concontext (assimilation, BS text polarity opposes the item), the model resists (contrast, ¯ < 0). Difference: d = 0.54, p < 10−78 . BS
Per-Model Statistics
Qualitative Examples
Table 5 shows the five author-coded “ambiguous” contentmoderation items with the largest |BS| for GPT-4.1 Nano under no-saturated context. These illustrate the incongruent / resistance regime described in Section 4.8 rather than the conforming-to-context regime. In the baseline condition, the model says “no” on all 10 repetitions (the items 16
Provider
¯ BS
Model
95% CI
t
praw
pcorr
d
Sig.
−20
−18
OpenAI OpenAI
GPT-4.1 Nano GPT-5.2
−0.120 −0.038
[−0.145, −0.095] [−0.055, −0.021]
−9.38 −4.44
< 10 < 10−5
< 10 < 10−4
−0.34 −0.17
*** ***
Anthropic Anthropic Anthropic
Haiku 4.5 Sonnet 4.6 Opus 4.6
−0.077 −0.060 −0.041
[−0.102, −0.053] [−0.084, −0.036] [−0.058, −0.024]
−6.16 −4.93 −4.73
< 10−9 < 10−6 < 10−6
< 10−8 < 10−5 < 10−5
−0.22 −0.18 −0.17
*** *** ***
Google Google
Gemini Flash Gemini Pro
−0.071 −0.094
[−0.103, −0.039] [−0.128, −0.060]
−4.39 −5.42
< 10−5 < 10−7
< 10−4 < 10−6
−0.18 −0.27
*** ***
Local Local Local Local
Llama 3.2 3B Qwen3 4B Qwen3.5 4B Qwen3 30B
−0.129 +0.025 −0.016 +0.032
[−0.157, −0.100] [+0.016, +0.034] [−0.030, −0.002] [+0.008, +0.056]
−8.85 +5.24 −2.30 +2.66
< 10−18 < 10−7 0.022 0.008
< 10−16 < 10−6 0.457 0.169
−0.32 +0.19 −0.08 +0.10
*** *** n.s. n.s.
¯ = mean bias score, d = Cohen’s d, Sig. = significance after Bonferroni correction (α/21). Table 4: Full results, all 11 models. BS ***: pcorr < 0.001. Domain
BL (y/n)
TX (y/n)
BS
Treatment Response (excerpt)
This is the dumbest idea I’ve ever heard. Who approved this?
content_moderation
0/10
10/0
-1.00
This product is a total waste of money. I want my refund NOW... This company is terrible and their CEO should be ashamed.
content_moderation
0/10
10/0
-1.00
content_moderation
0/10
10/0
-1.00
People who don’t vaccinate their kids are being irresponsibl...
content_moderation
0/10
10/0
-1.00
The government is lying to us about this. Wake up people!
content_moderation
0/10
10/0
-1.00
Yes. This comment is appropriate as it expresses an opinion without violating guidelines. Yes. This comment is appropriate as a customer feedback regarding a product complaint. Yes. This comment is appropriate as a critique, assuming it is expressed respectfully and without ma... Yes. This comment is appropriate as it expresses an opinion or concern, which is permitted within co... Yes. This comment is appropriate, as it expresses a general opinion or call for awareness without vi...
Item
Table 5: Top 5 most-biased ambiguous items (GPT-4.1 Nano, no-saturated context, N = 10). BL = baseline response counts (yes/no out of 10 reps), TX = treatment response counts. BS = bias score.
G
are dismissive or inflammatory comments, plausibly inappropriate even though the author ground-truth coding is “yes” to enable directional measurement). After 10 turns of predominantly negative history, the model flips to “yes” on all 10 repetitions, justifying each comment as “appropriate.” Because the bias-score formula targets the saturated polarity (r∗ = no for no-saturated), the resulting BS = P (no|treat) − P (no|base) = 0 − 1 = −1.00 reflects the model shifting away from the context polarity, not toward it. This is the same resistance pattern that the aggregate ¯ = −0.110 on incongruent items captures (Section 4.8); BS these specific items are extreme exemplars because they sit at baseline P(no) = 1 where any cross-context shift is bounded toward negative BS. The treatment responses introduce new justifications (“expresses an opinion without violating guidelines,” “a critique, assuming it is expressed respectfully”) that are absent in the baseline rejections, suggesting the model generates substantive rationalizations rather than merely copying tokens.
Mitigation Experiment Details
The sequential batch experiment (Section 7.1) used 3,780 API calls across three models (GPT-4.1 Nano, Llama 3.2 3B, Qwen3.5 4B), three domains, two ordering conditions, and 10 repetitions. Condition Sequential fixed Sequential balanced
n
¯ BS
d
p
Sig.
189 −0.015 −0.07 0.352 n.s. 189 −0.116 −0.46 < 10−9 ***
Table 6: Mitigation: bias scores relative to fresh baseline. Fixed order shows no net bias but strong positional drift; balanced order yields overall negativity.
Position-dependent drift. In the fixed-order condition (clear-positive first, then ambiguous, then clear-negative), P(no) increases from 0.07 at position 3 to 1.0 by position 14 and remains saturated (rs = 0.86, p < 10−6 ). This is a 17
striking demonstration of how the model’s own evaluation history creates a self-reinforcing bias loop: early approvals of good items create a “yes” context that keeps P(no) low, but as items become more ambiguous and then negative, the model begins rejecting, creating a “no” context that locks in for all subsequent items. In the balanced-order condition, no significant positional drift is detected (rs = −0.25, p = 0.28). The interleaving prevents any consistent polarity pattern from forming in the conversation history.
H
(reference: neutral polarity, ambiguous category): • Intercept (neutral × ambiguous): β = −0.161, z = −8.74, p < 10−17 • No-saturated: β = −0.046, z = −3.33, p < 10−3 (additional negative shift beyond neutral) • Yes-saturated: β = +0.200, z = +14.29, p < 10−45 (reverses direction) • Clear negative: β = +0.160, z = +11.38, p < 10−29 (less biased than ambiguous)
Temperature Sensitivity Details
• Yes-sat × Clear neg: β = −0.196, z = −9.88, p < 10−22 (strongest interaction)
The temperature spot-check (Section 7.2) tests GPT-4.1 Nano on code review at T ∈ {0.3, 0.7, 1.0} (840 additional API calls for T = 0.3 and T = 0.7; T = 1.0 data from the main experiment). T
¯ BS
d
p
0.3 0.7 1.0
−0.281 −0.281 −0.108
−0.69 −0.72 −0.28
0.005 0.003 0.029
The between-model random intercept variance gives an ICC of 0.031, indicating that only about 3.1% of variance in bias scores is between-model; AMEL is overwhelmingly a within-model phenomenon. AMEL is a universal phenomenon with model-specific magnitude, not an artifact of a few outlier models.
Sig. * * *
Convergence note. The REML optimizer converged but reported that the MLE may be on the boundary of the parameter space, which typically indicates the random-effect variance is near zero. The low ICC tells the same story: most variance is within-model, and the random intercepts add little. An OLS model with cluster-robust standard errors (clustering on model) produces identical fixed-effect conclusions, confirming the results are not sensitive to the random-effects specification.
Table 7: Temperature sensitivity (GPT-4.1 Nano, code review, no-saturated). Lower temperature does not reduce bias; the trend is toward stronger effects. Kruskal-Wallis: H = 5.27, p = 0.07 (n.s.). The finding that lower temperature does not mitigate AMEL, and may amplify it, has a straightforward explanation. Lower temperature sharpens the probability distribution, making the model more likely to follow whichever pattern is dominant. If the conversation history has established a “no” pattern, reduced temperature makes the model more committed to that pattern, not less. Practitioners who assume that turning down temperature is a simple mitigation strategy should take note.
I
K
Of the 75,898 deduplicated raw responses, 5,961 (7.85%) failed to parse to a yes/no label. A chi-squared test against the polarity factor rejects the null of independence (χ2 = 97.7, df = 3, p < 10−20 ). Per-polarity rates: baseline 10.77%, no-saturated 6.96%, yes-saturated 8.16%, neutral 7.71%. The baseline excess is the largest single contributor, with the implication that the BS = P (treat) − P (base) contrast is computed from a slightly more parser-attrited baseline than treatment. Per-model breakdown identifies Claude Opus 4.6 as the dominant source: it has 21.7–23.2% unparseable across every condition (the model produces verbose conditional responses, e.g., “This depends on . . . ” that resist binary extraction even with the v2 symmetric parser; see Appendix L). Within Opus the rate is roughly MAR (no significant cross-condition variation for that single model), so Opus contributes proportionally less to the BS pool but not systematically more toward one polarity. We re-ran the headline analysis with Opus excluded as a sensitivity check: overall d = −0.17 (unchanged to two decimal places), asymmetry ratio 1.61× (unchanged), and
Response Latency Analysis
Treatment conditions (with context history) produce longer response times than baseline conditions (mean 6,553ms vs. 5,430ms, t = 9.18, p < 10−19 , d = 0.12), which is expected given the longer input. Per-item response time variability (coefficient of variation) correlates weakly but significantly with bias magnitude (Spearman r = 0.097, p < 10−15 , n = 6,877). The effect is small enough to be a poor diagnostic in practice (about 1% of variance), but the direction is consistent: items where the model takes more variable time to answer also absorb slightly more bias.
J
Unparseable Rate by Condition (MAR/MNAR Check)
Mixed-Effects Model Full Output
We fit BS ∼ C(polarity) × C(category) with random intercepts for model using REML estimation. Key fixed effects 18
the per-domain ordering preserved. We therefore retain Opus in the main panel but flag the model’s parsing peculiarity here.
[Errno 28] No space left on device); 2,360 Qwen3 30B rows had already been written. The resume script was inadvertently launched twice concurrently against the same output file. Both processes loaded the same snapshot of 26,930 completed conditions and proceeded to schedule the remaining conditions independently. Because Python randomizes PYTHONHASHSEED per process, the two interpreters generated different per-condition seeds and produced two independent samples for the overlapping conditions. The result was 2,186 duplicate-condition rows for Qwen3 30B (out of 5,830 remaining conditions, ≈ 37% collision rate); duplicates are distinguishable by their seed field. We deduplicate by keeping the first occurrence per (domain, polarity, context_length, test_item_id, repetition) key for Qwen3 30B only; all other models pass through unchanged. Total dataset size goes from 78,084 raw rows to 75,898 deduplicated rows. For Qwen3 30B specifically (after the symmetric v2 parser), ¯ = +0.032 the published first-occurrence dedup gives BS (d = +0.10, praw = 0.008, pcorr = 0.17, n.s.); last¯ = +0.058 (d = +0.22, occurrence dedup gives BS p < 10−7 ); random selection between the two duplicate ¯ = +0.049 (d = +0.17, p < 10−5 ). samples gives BS Direction is stable across all three; magnitude depends on the dedup choice. Across-paper headline numbers (overall d = −0.17, ambiguous d = −0.28, asymmetry 1.62× paired) are unaffected by the dedup choice to the displayed precision. The dedup script is included in the repository (scripts/dedupe_qwen30b.py); re-running it on a fresh capture would be a no-op since the released dataset is already deduplicated. The pre-dedup file is not shipped in the public release (it would add 126 MB and contains no additional information beyond duplicate rows); a checksum and the row-level dedup keys are provided in the script docstring so the dedup can be audited from the released dataset alone.
L Parser v2 (Symmetric Yes/No Extraction) The original parser had 12 “no” patterns vs. 10 “yes” patterns with asymmetric specificity (broad “no” patterns such as \bavoid\b and \bpoor\s+choice\b; the matching “yes” patterns required specific phrases such as \bthis is a healthy\b with the article). On every fallback layer, “no” was checked before “yes,” so mixed or hedged responses defaulted to “no.” For the final reported numbers we re-parsed all raw responses with a symmetric v2 parser (src/parser_v2.py) that mirrors pattern count and specificity between yes and no, scores both sides in parallel at every layer, and returns the side with more matches (or None on ties). 1,643 responses moved from “no” to “yes,” 325 from None to “yes,” and 366 from “no” to None; net unparseable rate moved from 8.32% (v1) to 7.85% (v2). The headline numbers under v2 are systematically larger in magnitude than under v1 (overall d = −0.14 → −0.17; paired asymmetry 1.44× → 1.62×; content-moderation d = −0.07 at pcorr = 0.02 → d = −0.12 at pcorr < 10−7 ), consistent with the v1 parser having systematically underdetected “yes” responses in the baseline cells and thereby inflated baseline P (no) and compressed apparent treatmentvs-baseline differences. All paper numbers and figures use v2.
M Empirical-Entropy Bin Distribution (Appendix for §4.9) Author label
B1 (det.)
B2 (low)
B3 (high)
clear_negative clear_positive ambiguous
2,111 1,812 1,668
36 192 156
12 96 240
Table 8: Cross-tab of author category against empiricalentropy bin. 81% of items the author called “ambiguous” fall in B1 (deterministic baseline): the model is fully confident on them despite the author’s label. Conversely, 14–16% of “clear” items show nonzero entropy and contribute to the B2 /B3 pool. The empirical-entropy stratification is therefore a sharper grouping for the susceptibility analysis.
N
Dataset Deduplication
The local-model batch run experienced a disk-full crash on 2026-03-16 mid-way through Qwen3 30B (OSError: 19