What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations Xiaonan Xua,∗, Wenjing Wub a College of Computing, Georgia Institute of Technology, Atlanta, GA, 30332, USA

arXiv:2608.17719v1 [cs.SE] 18 Aug 2026

b Department of Computer Science, University of Colorado

Boulder, Boulder, CO, 80309, USA

Abstract Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, we query 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following) 50 times per item per model, classify each item as reliably improved, reliably regressed, practically equivalent, or inconclusive under falsediscovery-rate control and a practical-significance threshold, and calibrate the results against a label-permutation null. Results: Across all nine migration– benchmark cells, reliable improvements and reliable regressions coexist. Edges with aggregate gains of up to 7.3 percentage points contain up to 8.3% reliably regressed items; edges with aggregate losses contain up to 10.7% reliably improved items. On the instruction-following benchmark, the gap between strict and loose scoring widens by 3.9 percentage points on the latest migration: a 3.9-point regression under strict scoring shrinks to 0.04 points under loose scoring. Conclusion: Migration decisions based on aggregate scores alone miss ∗ Corresponding author.

Email addresses: xiaonanxu5@gmail.com (Xiaonan Xu), wuwenjing256@gmail.com (Wenjing Wu)

substantial bidirectional item-level change. The complete response-level archive and per-item scoring outputs are released. Keywords: LLM evaluation, backward compatibility, regression testing, API migration, repeated sampling

1. Introduction Commercial large language model (LLM) APIs have become external software dependencies in a growing share of production systems [1]. These APIs are controlled by the vendor: models are retired on published deprecation schedules, and downstream consumers must migrate to successor versions or lose access [2]. Library-migration studies show that forced dependency updates carry real cost for client projects [3], and that updates preserving interfaces can still break client behaviour [4]. Each migration replaces the model behind every downstream call site. Whether the new version preserves the test-case-level behaviour of the old one, a property known in software engineering as backward compatibility, determines whether the migration is safe. Standard benchmark reporting compresses the answer to that question into aggregate scores. A net gain of two percentage points on a benchmark may consist of 100 items answered more reliably and 80 answered less reliably; the aggregate reports only the net balance. This compression has practical consequences: in July 2026, OpenAI’s Thibault Sottiaux publicly stated that, before the GPT-5.6 Sol launch, the team had focused on average and median usage and missed cases in which long-tail usage was substantially higher [5]. The stochastic nature of LLM outputs adds a second difficulty. A single correctto-incorrect flip between two model versions may reflect sampling noise alone. Ma et al. [2] identify non-determinism as one of three fundamental obstacles to applying regression testing to LLM APIs, and in empirical measurements on open-weight models, single-draw evaluation missed 42% of reliably changed items [6]. Backward compatibility at the item level therefore requires estimating pass-probability changes and calibrating them against a permutation null.

2

A longitudinal study of 18 GPT models fits ability trajectories to singledraw responses, so estimated probability shifts share one direction within each comparison [7]. A repeated-sampling study on open-weight 7–8B models measures bidirectional reliable churn at K=10, leaving frontier commercial models as an open question [6]. An industry evaluation compares GPT-5.5 and GPT5.6 on 711 enterprise workflows at 5 runs per task with Holm-corrected significance tests [8]. Our study extends this repeated-sampling approach to frontier commercial API migrations at K=50 with permutation-null calibration of the complete classification procedure. We compare three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence on 900 public benchmark items spanning graduate-level knowledge, olympiad mathematics, and instruction following, with all request parameters held constant except the model identifier. Each model–item pass probability is estimated from 50 independent trials, and item-level judgements are calibrated against a permutation null at matched sample size. On the instruction-following benchmark we also compare the official strict and loose verifier scores to test whether measured regressions depend on exact output compliance. We find that reliable improvements and reliable regressions coexist in all nine migration– benchmark cells: edges with a positive aggregate change contain up to 8.3% reliably regressed items, and edges with a negative aggregate change contain up to 10.7% reliably improved items. On the instruction-following benchmark, the strict–loose scoring gap widens by 3.9 percentage points on the latest migration: the same edge shows a 3.9-point regression under strict scoring and a 0.04-point regression under loose scoring. The contributions of this paper are as follows. 1. We measure item-level backward compatibility across the GPT-5.4 to GPT5.6 Sol product sequence, querying 900 public benchmark items 50 times per item per model. Item-level judgements are calibrated against a permutation null that reruns the complete classification procedure at matched sample size. 2. We show that reliable improvements and reliable regressions coexist in all nine migration–benchmark cells, including edges where the aggregate score 3

improved. 3. We quantify how migration conclusions change between strict and loose scoring: on the instruction-following benchmark, the latest migration widens the gap between strict and loose scoring, causing the regression observed under strict scoring to shrink under loose scoring. 4. We release the complete response-level archive with per-item scoring outputs, enabling verification and alternative rescoring without re-querying mutable commercial APIs.

2. Related Work Aggregate evaluation and its limits. The dominant practice in LLM evaluation is to report aggregate scores on static benchmarks. Chang et al. [9] survey evaluation practices and identify recurring concerns: sensitivity to prompt format, inconsistency across evaluation pipelines, and the limited diagnostic value of a single aggregate figure. Benchmark saturation and data contamination amplify these concerns as models and training corpora co-evolve [10], and a mapping of generative-AI metrics to software quality characteristics concludes that no single aggregate captures the relevant quality dimensions [11]. Aggregate scores remain useful summaries of overall capability, but they reduce heterogeneous item-level behaviour to a single net figure that conflates uniform improvement with a mixture of gains and losses. Our study takes this compression as its object of measurement and quantifies what the aggregate hides. Model-update regression in LLM APIs. Classical regression testing assumes deterministic test outcomes, an assumption already strained within conventional software by flaky tests [12]; stochastic LLM outputs invalidate it entirely. Chen et al. [13] provided early large-scale evidence that commercial GPT behaviour shifts across time-stamped snapshots, comparing two snapshots at the task level. Ma et al. [2] reframed the phenomenon as a software engineering problem, identifying three properties of commercial LLM APIs that break classical regressiontesting assumptions: vendor-controlled updates, prompt sensitivity, and non4

deterministic outputs. Echterhoff et al. [14] formalised negative flips (items correct before an update and incorrect after) and proposed compatibility-aware training to reduce them, addressing the problem from the model developer’s side. A production-oriented migration framework [15] uses Bayesian candidate comparison to support replacement decisions when a model reaches end-oflife. Dong et al. [16] showed that instruction-following accuracy on GPT-4o snapshots drops when prompts are paraphrased, decomposing aggregate scores along the prompt-variation axis. Our work shares the decomposition goal but varies the model version rather than the prompt, and measures pass-probability changes from repeated sampling rather than from prompt variants. Item-level measurement and repeated sampling. Three recent studies address item-level version comparison directly. A longitudinal study covering 18 GPT models from GPT-3.5 to GPT-5.2 [7] fits a dynamic item response theory model to single-draw binary responses, estimating latent ability trajectories and localising probability changes across difficulty and discrimination regions. Because the model assigns a single ability parameter per snapshot, all item-level probability shifts share the same sign within a comparison; the authors note that heterogeneous directions require analysis of observed response flips, which the singledraw design does not support. The repeated-sampling approach of [6] addresses this directly: each item is queried K=10 times, and a reliable-change index with a permutation null separates true changes from sampling noise on open-weight Llama and Qwen models (7–8B parameters). That study reports 21–28% total reliable churn; single-draw evaluation missed 42% of reliably changed items and falsely flagged 25% of unchanged items. It identifies frontier commercial models and additional benchmarks as open questions. Toloka [8] evaluates GPT5.5 against GPT-5.6 on 711 frozen enterprise workflows with 5 runs per task, applying stratified bootstrap and Holm-corrected significance tests to detect aggregate regressions and identify strong per-task flips. The present study extends the repeated-sampling line to frontier commercial API migrations, raises the per-item trial count to K=50, applies false-discovery-rate control with a

5

practical-significance threshold, and calibrates item-level judgements against a permutation null. Output variability. Repeated queries to the same commercial model return varying answers. Ouyang et al. [17] measure this directly for ChatGPT code generation, finding semantic variation across identical requests even at temperature zero, and Kim and Ming [18] compare output reliability and similarity across models on software-development tasks. A systematic catalogue attributes such divergence to sampling, silent updates, numerical rounding, and expert routing [19]. Estimating per-item pass probabilities from multiple generations, rather than judging single draws, follows the direction formalised by Zhang et al. [20]. Community guidelines for LLM-based empirical studies likewise identify output non-determinism and model evolution as threats to reproducibility and call for archived interaction traces [21], a practice the released response archive follows.

3. Method 3.1. Research questions We study whether aggregate benchmark scores reliably reflect the backward compatibility of commercial LLM API upgrades. Three research questions structure the study. RQ1. On each migration edge in the current GPT product line, what proportion of benchmark items shows reliable improvement, and what proportion shows reliable regression, beyond what sampling noise alone would produce? RQ2. How does the aggregate score change on each edge relate to the underlying item-level changes, and how much bidirectional change does the aggregate figure conceal? RQ3. How do migration conclusions change between strict and loose scoring of the same responses?

6

3.2. Study design The unit of comparison is a migration edge: an ordered pair of models that an API consumer moves between when following the vendor’s product line. We measure three edges over three models, GPT-5.4, GPT-5.5, and GPT-5.6 Sol: the two consecutive flagship migrations (5.4→5.5 and 5.5→Sol) and the direct migration 5.4→Sol, which the vendor’s guidance also supports. The three models are served under the API identifiers gpt-5.4, gpt-5.5, and gpt-5.6-sol. All request parameters are held constant across models; only the model identifier varies. This mirrors a dependency upgrade in which client code is unchanged and only the dependency version moves. Reasoning effort is set explicitly to medium for every model, the shared tier across the three models’ effort ranges. Temperature is not accepted by these models. No output-length cap is imposed. Each item is queried K = 50 times per model in independent single-turn calls. 3.3. Benchmarks and item selection Measuring item-level change requires items on which the models under study retain headroom: an item answered correctly, or incorrectly, in every trial by every model carries no information about migration-induced change. We therefore selected benchmarks by a screening procedure defined before the main collection. Eleven public, automatically scorable candidates spanning knowledge, mathematics, and instruction following were screened with 30 randomly drawn items each, three models, and K = 10 repetitions per item. For each candidate we computed the fraction of items on which at least one model’s observed accuracy fell in [0.1, 0.9]. One benchmark per category was selected, requiring that at least 35% of items meet this criterion, and taking the candidate with the highest proportion per slot. Widely used benchmarks, including GPQA Diamond, MMLU-Pro, and IFEval, fell below the threshold on these models, with proportions between 7% and 27%; the full screening table is reported in the appendix. Screening data served benchmark selection only and did not enter the main analysis. 7

The selected benchmarks are SuperGPQA [22] (knowledge; 500 items drawn at random from the full pool before collection), Omni-MATH hard [23] (mathematics; 100 items drawn at random from the 452-item pool before collection), and IFBench [24] (instruction following; all 300 prompts, with the prompt as the unit of analysis), for a total of 900 items. The sampled item lists were fixed before collection and are included in the released archive. Prompting follows the benchmark type. Knowledge and mathematics items use a minimal task statement and a final-answer-line convention, frozen after piloting. IFBench prompts are used verbatim from the official release, since their wording constitutes the task constraints; they are scored by the official verifier [24, 25]. 3.4. Data collection Collection proceeded in per-model phases in release order (GPT-5.4, then GPT-5.5, then GPT-5.6 Sol), with item–repetition order randomised within each phase. Collection ran from 30 July to 13 August 2026. All phases ran at a fixed request concurrency of 36. Transient API errors were retried with exponential backoff until success; the final response matrix is complete. 3.5. Scoring Scoring is decoupled from collection and deterministic given the archived responses. Every response is assigned to one of six mutually exclusive categories: semantically correct, semantically wrong, format failure, refusal, truncation, and API error; given the retry policy, every response in the final matrix falls in the first five categories. API errors are excluded from the accuracy denominator and their occurrence rate is reported separately; refusals, format failures, and model-side truncations remain in the denominator as behavioural failures. For knowledge and mathematics items, a response is scored correct when a two-stage parser extracts an answer from the final answer line and that answer matches the reference; mathematical equivalence is established by exact match

8

followed by symbolic normalisation. IFBench responses are scored by the official verifiers [24], which provide a strict and a loose reading; the loose reading re-applies each verification function after a fixed set of surface-format normalisations, so the two readings differ only in format tolerance. The strict reading is primary. Responses that a downstream system could not parse under the agreed format are counted as failures, since for an API consumer an unparseable reply is operationally indistinguishable from a wrong one. 3.6. Analysis For item i under model m, the observed accuracy is p̂m,i , the fraction of the K trials scored correct under the primary (strict) rule. For each edge, source s to target t, we report per benchmark the aggregate change ∆ = meani (p̂t,i ) − meani (p̂s,i ); the reliable-improvement share P + ; the reliable-regression share P − ; and the reliable-change share P chg = P + + P − . Item-level judgements require both statistical significance and a minimum effect size. A Fisher exact test on the 2 × K correct/incorrect counts, with Benjamini–Hochberg control of the false discovery rate at 5% within each migration– benchmark cell, establishes significance. A practical-significance threshold |p̂t,i − p̂s,i | ≥ ε with ε = 0.2 (a 20-percentage-point shift in pass probability) establishes a minimum effect size. Items meeting both criteria are classified as reliably improved or reliably regressed by sign. Items whose 95% confidence interval for the difference lies entirely within (−ε, ε) are classified as practically equivalent; the remainder are classified as inconclusive. Observed shares are calibrated against a permutation null. For each item we pool the 2K outcomes of the two models on an edge, randomly reassign version labels with K outcomes per side, and rerun the complete classification procedure; 1,000 replications yield the null distribution of P + , P − , and P chg under zero true change at matched sample size. Reported shares are presented alongside the 95th percentile of this null, so that reliable change is claimed only where it exceeds what label noise alone produces. For the secondary analysis on IFBench, let Rm denote the mean difference 9

Table 1: Aggregate change and item-level classification for each migration edge and benchmark. ∆ is the change in mean strict accuracy in percentage points. P + , P − , and P chg are the shares of items classified as reliably improved, reliably regressed, and reliably changed under the Fisher/BH/ε=0.2 criteria; Equiv. and Incon. are the practically-equivalent and inconclusive shares. Null95 is the 95th percentile of P chg under the permutation null (1,000 replications). All shares in percent; ∆ is computed before rounding

Edge

5.4→5.5

5.5→Sol

5.4→Sol

Benchmark

∆

P+

P−

P chg

Equiv.

Incon.

Null95

SuperGPQA

+2.3

8.2

5.0

13.2

77.0

9.8

0.0

Omni-MATH hard

+7.3

17.0

6.0

23.0

59.0

18.0

0.0

IFBench

+1.9

11.3

8.3

19.7

55.7

24.7

0.0

SuperGPQA

+1.4

7.6

4.4

12.0

80.4

7.6

0.0

Omni-MATH hard

−4.1

9.0

15.0

24.0

67.0

9.0

0.0

IFBench

−3.9

6.7

13.3

20.0

57.0

23.0

0.0

SuperGPQA

+3.8

10.8

4.8

15.6

76.8

7.6

0.0

Omni-MATH hard

+3.2

10.0

5.0

15.0

70.0

15.0

0.0

IFBench

−2.0

10.7

13.3

24.0

54.3

21.7

0.0

between the loose and strict verifier scores across prompts. Each edge reports ∆R = Rt − Rs , the change in this gap across the migration.

4. Results 4.1. Aggregate scores and item-level changes Mean strict accuracy was 63.4%, 45.7%, and 62.8% for GPT-5.4 on SuperGPQA, Omni-MATH hard, and IFBench; 65.7%, 53.0%, and 64.6% for GPT-5.5; and 67.1%, 48.9%, and 60.7% for Sol. Table 1 summarises the aggregate change and the full item-level classification for each migration edge and benchmark; Figure 1 shows the reliable-improvement and reliable-regression shares alongside the aggregate change. On the 5.4→5.5 edge, aggregate scores rose on all three benchmarks (+2.3, +7.3, and +1.9 percentage points on SuperGPQA, Omni-MATH hard, and IFBench). All three benchmarks also show reliable regression: 5.0% of SuperG10

GPT-5.4 → GPT-5.5 SuperGPQA Omni-MATH hard IFBench GPT-5.5 → GPT-5.6 Sol SuperGPQA Omni-MATH hard IFBench GPT-5.4 → GPT-5.6 Sol SuperGPQA Omni-MATH hard

P + reliably improved P − reliably regressed

IFBench

Δ aggregate change

−15

−10

−5 0 5 Share of items / aggregate change (percentage points)

10

15

Figure 1: Item-level reliable improvement and regression alongside aggregate change for each migration edge and benchmark. Bars extending right (left) show the share of items reliably improved (regressed); the central marker shows the aggregate change ∆

PQA items, 6.0% of Omni-MATH items, and 8.3% of IFBench items regressed reliably despite a positive aggregate change. On the 5.5→Sol edge, the three benchmarks diverge. SuperGPQA continued to rise (+1.4) while Omni-MATH hard (−4.1) and IFBench (−3.9) declined. Both declining benchmarks still contain reliably improved items: 9.0% of OmniMATH items and 6.7% of IFBench items improved reliably even as the aggregate fell. On the direct 5.4→Sol edge, SuperGPQA showed the largest gain (+3.8) and the highest improvement share (10.8%), while IFBench showed a net loss (−2.0) with 13.3% of items reliably regressed and 10.7% reliably improved. Reliablechange shares on this edge differ from the sums of the two consecutive edges, reflecting item-level movements that cancel across consecutive migrations. 4.2. Permutation-null calibration Under zero true change, the 95th percentile of the reliable-change share was 0.0% in every migration–benchmark cell: across 1,000 label permutations at the study’s sample sizes, the Fisher/BH/ε criteria produced zero spurious reliable 11

changes. Every reliably changed item in Table 1 therefore lies strictly above the noise floor. Prior work on open-weight 7–8B models reported total churn of 21–28% at K=10 [6]; the shares of 12.0–24.0% observed here are of comparable magnitude yet rest on a calibrated zero baseline. 4.3. Strict–loose scoring gap on IFBench IFBench’s official verifiers provide a strict and a loose reading of every response. The gap Rm between the two readings was 6.3 percentage points for GPT-5.4, 5.1 for GPT-5.5, and 9.0 for Sol, yielding a ∆R of −1.2 points on the 5.4→5.5 edge, +3.9 on 5.5→Sol, and +2.7 on 5.4→Sol. The widening on the 5.5→Sol edge accounts for nearly the entire aggregate decline: Sol’s strict score falls 3.9 points below GPT-5.5’s, while its loose score falls 0.04 points; the regression is concentrated in exact constraint compliance. On SuperGPQA and Omni-MATH hard, responses are scored by answer extraction and exact match; extraction failures were negligible, and the six-category breakdown in the appendix reports their rates. 4.4. Supplementary analyses The appendix reports five supplementary analyses on the same data: a single-draw comparison that contrasts raw correct-to-incorrect and incorrectto-correct flips from one randomly selected trial per item with the K=50 classification; a sensitivity analysis of the practical-significance threshold over ε ∈ {0.10, 0.15, 0.20, 0.25}; token-cost distributions per model and edge; the sixcategory response breakdown; and difficulty stratification using metadata provided by each benchmark.

5. Discussion Across all nine migration–benchmark cells, reliable improvements and reliable regressions occur together. The permutation null confirms that the observed shares exceed what sampling noise alone produces at K=50. Aggregate scores therefore provide a necessary but insufficient basis for migration decisions: 12

on the six cells where the aggregate improved, a gate based on the aggregate alone would have accepted the migration while 4.4–8.3% of items regressed reliably. The IFBench results raise a separate concern. Because the two readings differ only in format tolerance, the widening of the strict–loose gap on the 5.5-to-Sol edge locates part of the observed strict-scoring regression in exact format compliance rather than in the underlying task. For a system that parses outputs programmatically, format non-compliance is a functional regression; a consumer that tolerates format variation faces a smaller compatibility cost from the same migration. Migration risk therefore depends on the acceptance criteria applied to model outputs, and capturing both perspectives requires reporting both readings. The repeated-sampling study on open-weight models [6] asked whether its findings extend to frontier commercial models; the present measurements provide a direct comparison point at K=50 on three GPT versions. The Toloka evaluation [8] identified per-task regressions on the same product line at K=5 with Holm-corrected tests; our higher per-item trial count and permutation null complement that evidence by separating reliable item-level changes from sampling artefacts at higher statistical resolution. The appendix compares single-draw evaluation with the repeated-sampling judgements directly: a single binary observation per model cannot reach significance under the item-level test, so none of the 457 reliable item-level changes across the nine cells is detectable from one draw, and the appendix additionally reports how raw single-draw flips distribute over the K=50 classes. The gap between single-draw and K=50 judgements sets a lower bound on the information lost when migration decisions rely on single-draw evaluation. Organisations that treat model upgrades as dependency updates can use the item-level regression shares reported here as reference rates when designing acceptance tests for their own workloads.

13

6. Threats to Validity Three limitations bound the scope of these measurements. The benchmark screening step selects benchmarks on which the studied models retain headroom; the reported reliable-change shares are therefore estimates for informative benchmarks, not population rates over arbitrary workloads. The study covers one vendor’s product line (GPT-5.4 through GPT-5.6 Sol) and three public benchmarks spanning knowledge, mathematics, and instruction following; other vendors, task types, and production workloads may show different patterns. The practical-significance threshold ε = 0.2 and the trial count K=50 together determine which item-level changes are detectable: smaller genuine changes remain in the inconclusive category and are reported as such.

7. Conclusion Across three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, with 900 public benchmark items, 50 trials per item, and calibration against a permutation null, every migration–benchmark cell contains both reliably improved and reliably regressed items; aggregate gains of up to 7.3 percentage points accompany up to 8.3% reliably regressed items. The strict–loose scoring gap widens on the latest migration for instruction following, indicating that part of the measured regression concentrates on format compliance. The complete response-level archive and per-item scoring outputs are released to support verification and alternative analyses.

Data Availability The complete response-level archive (request parameters, raw responses, returned model identifiers, usage metadata, and timestamps) and the per-item scoring outputs are available at https://github.com/WenJing95/gpt-regression-data. All reported statistics follow the procedures specified in Section 3 and can be recomputed from this archive without re-querying commercial APIs.

14

Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this work the authors used Claude (Anthropic) and ChatGPT (OpenAI) in order to improve the language and readability of the manuscript. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.

CRediT authorship contribution statement Xiaonan Xu: Conceptualization, Methodology, Investigation, Data curation, Writing – original draft. Wenjing Wu: Software, Validation, Formal analysis, Writing – review & editing.

References [1] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language models for software engineering: A systematic literature review, ACM Transactions on Software Engineering and Methodology 33 (8) (2024) 220:1–220:79.

15

[2] W. Ma, C. Yang, C. Kästner, (Why) is my prompt getting worse? Rethinking regression testing for evolving LLM APIs, in: Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering – Software Engineering for AI (CAIN), 2024, pp. 166–171. [3] R. G. Kula, D. M. German, A. Ouni, T. Ishio, K. Inoue, Do developers update their library dependencies? an empirical study on the impact of security advisories on library migration, Empirical Software Engineering 23 (1) (2018) 384–417. [4] D. Jayasuriya, V. Terragni, J. Dietrich, K. Blincoe, Understanding the impact of APIs behavioral breaking changes on client applications, Proceedings of the ACM on Software Engineering 1 (FSE) (2024) 1238–1261. [5] T. Sottiaux, Sol community update: GPT-5.6 Sol usage quotas, X (formerly Twitter),

https://x.com/thsottiaux/status/2082317452755751098,

post of 29 July 2026. Accessed 16 August 2026 (2026). [6] J.-P. Cacioli, Beyond the mean: Within-model reliable change detection for LLM evaluation, arXiv preprint arXiv:2604.27405 (2026). [7] Anonymous, Longitudinal evaluation of large language models, under review for Transactions on Machine Learning Research (TMLR), Paper 8871. Submitted 2026-05-11. https://openreview.net/forum?id=INuSvLC7Bq (2026). [8] Toloka AI, GPT-5.6 got smarter. Then it kept acting., https://toloka. ai/blog/gpt-5.6-got-smarter-then-it-kept-acting/, blog post, July 2026. Accessed 16 August 2026 (2026). [9] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al., A survey on evaluation of large language models, ACM Transactions on Intelligent Systems and Technology 15 (3) (2024) 39:1–39:45.

16

[10] S. Chen, Y. Chen, Z. Li, Y. Jiang, Z. Wan, Y. He, D. Ran, T. Gu, H. Li, T. Xie, B. Ray, Benchmarking large language models under data contamination: A survey from static to dynamic evaluation, in: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 10080–10098. doi:10.18653/v1/2025.emnlp-main.511. [11] L. Yu, E. Alégroth, P. Chatzipetrou, T. Gorschek, Measuring the quality of generative AI systems: Mapping metrics to quality characteristics— snowballing literature review, Information and Software Technology 186 (2025) 107802. [12] Q. Luo, F. Hariri, L. Eloussi, D. Marinov, An empirical analysis of flaky tests, in: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014, pp. 643–653. [13] L. Chen, M. Zaharia, J. Zou, How is ChatGPT’s behavior changing over time?, arXiv preprint arXiv:2307.09009 (2023). [14] J. M. Echterhoff, F. Faghri, R. Vemulapalli, T.-Y. Hu, C.-L. Li, O. Tuzel, H. Pouransari, MUSCLE: A model update strategy for compatible LLM evolution, in: Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 7320–7332. [15] E. Casey, D. Roberts, D. Sim, I. Beaver, When your LLM reaches endof-life: A framework for confident model migration in production systems, arXiv preprint arXiv:2604.27082 (2026). [16] J. Dong, Y. Zhang, Y. Liu, Z. Zhong, T. Wei, C. Zhang, H. Qiu, Revisiting the reliability of language models in instruction-following, in: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 7784–7812. doi:10.18653/v1/2026.acl-long.354. [17] S. Ouyang, J. M. Zhang, M. Harman, M. Wang, An empirical study of the

17

non-determinism of ChatGPT in code generation, ACM Transactions on Software Engineering and Methodology 34 (2) (2025) 42:1–42:28. [18] D.-K. Kim, H. Ming, Assessing output reliability and similarity of large language models in software development: A comparative case study approach, Information and Software Technology 185 (2025) 107787. [19] G. Coqueret, J. Llull, F. Oswald, C. Pérignon, C. Scheuch, L. Vilhuber, Randomness in large language models: What researchers need to know (and report), arXiv preprint arXiv:2607.24372 (2026). [20] W. Zhang, H. Cai, W. Chen, Beyond the singular: Revealing the value of multiple generations in benchmark evaluation, in: Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 10033–10043. [21] S. Baltes, F. Angermeir, C. Arora, M. M. Barón, C. Chen, L. Böhme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald, et al., Evaluation guidelines for empirical studies in software engineering involving LLMs, arXiv preprint arXiv:2508.15503 (2025). [22] M.-A.-P. Team, SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines, arXiv preprint arXiv:2502.14739 (2025). [23] B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, et al., Omni-MATH: A universal olympiad level mathematic benchmark for large language models, arXiv preprint arXiv:2410.07985 (2024). [24] V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, H. Hajishirzi, Generalizing verifiable instruction following, in: Advances in Neural Information Processing Systems 38 (NeurIPS 2025) Datasets and Benchmarks Track, 2025. [25] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, L. Hou, Instruction-following evaluation for large language models, arXiv preprint arXiv:2311.07911 (2023).

18

Appendix A. Benchmark screening Table A.2 reports the benchmark screening described in Section 3.3: for each of the eleven candidate benchmarks, 30 randomly drawn items were queried K=10 times by each of the three models, and an item counts as estimable when at least one model’s observed accuracy falls in [0.1, 0.9]. The screening run for ComplexBench did not complete. Table A.2: Benchmark screening: estimable items out of 30 per candidate (K=10, three models). One benchmark per slot was selected, requiring a share of at least 35% and taking the highest-share candidate per slot

Slot

Candidate

Estimable

Share (%)

Outcome

SuperGPQA

13/30

43.3

Selected

GPQA Diamond

8/30

26.7

Not selected

SuperGPQA hard

7/30

23.3

Not selected

MMLU-Pro

2/30

6.7

Not selected

Omni-MATH hard

21/30

70.0

Selected

OlymMATH

16/30

53.3

Not selected

OlympiadBench text math

11/30

36.7

Not selected

IFBench

17/30

56.7

Selected

ComplexBench

11/30

36.7

Incomplete

IFEval

8/30

26.7

Not selected

IFEval++

3/30

10.0

Not selected

Knowledge

Mathematics

Instruction following

Appendix B. Single-draw comparison For each item and model, one trial was drawn uniformly at random from the K=50 archived trials under a fixed seed, and the classification procedure of Section 3.6 was applied to the resulting single observations. A comparison of two single binary observations never reaches significance under the Fisher/Benjamini– Hochberg criteria, so every item on every edge is classified as inconclusive and none of the 457 reliable item-level changes is recovered. Table B.3 reports 19

the raw correct-to-incorrect and incorrect-to-correct flips observed in the single draws; Table B.4 cross-tabulates these flips against the K=50 classification, aggregated over the nine migration–benchmark cells. Table B.3: Single-draw comparison. Reliable changes under the full K=50 procedure, reliable changes detected from single draws, and raw flips between the single draws of source and target model (I→C: incorrect to correct; C→I: correct to incorrect)

Edge

5.4→5.5

5.5→Sol

5.4→Sol

Benchmark

Items

K=50 rel. changes

Single-draw rel. changes

I→C

C→I

SuperGPQA

500

66

0

27

20

Omni-MATH hard

100

23

0

14

4

IFBench

300

59

0

42

19

SuperGPQA

500

60

0

28

19

Omni-MATH hard

100

24

0

6

10

IFBench

300

60

0

25

47

SuperGPQA

500

78

0

38

22

Omni-MATH hard

100

15

0

9

3

IFBench

300

72

0

32

31

Table B.4: Raw single-draw flips cross-tabulated against the K=50 classification, aggregated over the nine migration–benchmark cells (2,700 item–edge pairs)

K=50 classification

I→C

C→I

Unchanged

Total

Reliably improved

120

8

127

255

Reliably regressed

10

87

105

202

Practically equivalent

25

26

1817

1868

Inconclusive

66

54

255

375

Total

221

175

2304

2700

Appendix C. Sensitivity to the practical-significance threshold Table C.5 reports the reliable-improvement, reliable-regression, and reliablechange shares of each migration–benchmark cell when the practical-significance 20

threshold varies over ε ∈ {0.10, 0.15, 0.20, 0.25}, holding the Fisher/Benjamini– Hochberg criterion fixed. The rows with ε = 0.20 correspond to Table 1.

Appendix D. Token consumption Table D.6 summarises per-request token counts for each benchmark and model over the 135,000 archived responses; Table D.7 reports the change in mean total tokens per request on each migration edge.

Appendix E. Response category breakdown Table E.8 reports the six-category classification of Section 3.5 for all archived responses. Truncation and API error do not occur in the final response matrix; refusals occur only on IFBench.

Appendix F. Difficulty stratification Table F.9 reports mean strict accuracy per model and the per-edge change within strata defined by benchmark-native metadata: SuperGPQA provides difficulty and discipline labels, and Omni-MATH hard provides a difficulty rating. IFBench provides no comparable difficulty metadata and is therefore not stratified.

21

Table C.5: P + , P − , and P chg per migration–benchmark cell for four values of the practicalsignificance threshold ε. All shares in percent Edge

5.4→5.5

5.5→Sol

5.4→Sol

Benchmark

ε

P+

P−

P chg

SuperGPQA

0.10 0.15 0.20 0.25

8.4 8.4 8.2 8.0

5.2 5.2 5.0 4.6

13.6 13.6 13.2 12.6

Omni-MATH hard

0.10 0.15 0.20 0.25

19.0 19.0 17.0 15.0

6.0 6.0 6.0 3.0

25.0 25.0 23.0 18.0

IFBench

0.10 0.15 0.20 0.25

12.7 12.7 11.3 10.0

9.0 9.0 8.3 7.3

21.7 21.7 19.7 17.3

SuperGPQA

0.10 0.15 0.20 0.25

8.0 8.0 7.6 6.6

4.4 4.4 4.4 3.8

12.4 12.4 12.0 10.4

Omni-MATH hard

0.10 0.15 0.20 0.25

10.0 10.0 9.0 6.0

16.0 16.0 15.0 12.0

26.0 26.0 24.0 18.0

IFBench

0.10 0.15 0.20 0.25

7.0 7.0 6.7 6.0

14.0 14.0 13.3 12.7

21.0 21.0 20.0 18.7

SuperGPQA

0.10 0.15 0.20 0.25

11.4 11.4 10.8 10.2

5.0 5.0 4.8 4.6

16.4 16.4 15.6 14.8

Omni-MATH hard

0.10 0.15 0.20 0.25

11.0 11.0 10.0 8.0

5.0 5.0 5.0 3.0

16.0 16.0 15.0 11.0

IFBench

0.10 0.15 0.20 0.25

11.7 11.3 10.7 9.3

14.0 14.0 13.3 12.3

25.7 25.3 24.0 21.7

22

Table D.6: Per-request token counts by benchmark and model. Mean input and output tokens, and mean, median, 5th and 95th percentile of total tokens

Mean Benchmark

SuperGPQA

Omni-MATH hard

IFBench

Total tokens

Model

Input

Output

Mean

Median

P5

P95

GPT-5.4

543.2

555.2

1098.4

758

468

2692

GPT-5.5

543.2

543.5

1086.7

824

494

2767

Sol

543.8

409.7

953.5

702

476

2244

GPT-5.4

444.8

7653.3

8098.1

5732

1521

20779

GPT-5.5

444.8

3705.3

4150.1

3684

991

8751

Sol

444.8

3766.1

4211.0

3527

1139

9128

GPT-5.4

375.1

1030.2

1405.3

997

461

3699

GPT-5.5

375.3

981.8

1357.1

992

466

3339

Sol

394.1

1452.9

1847.0

1118

443

5613

Table D.7: Mean total tokens per request on each migration edge

Edge

5.4→5.5

5.5→Sol

5.4→Sol

Benchmark

Source

Target

Change

SuperGPQA

1098.4

1086.7

−11.7

Omni-MATH hard

8098.1

4150.1

−3948.0

IFBench

1405.3

1357.1

−48.1

SuperGPQA

1086.7

953.5

−133.2

Omni-MATH hard

4150.1

4211.0

+60.9

IFBench

1357.1

1847.0

+489.8

SuperGPQA

1098.4

953.5

−144.9

Omni-MATH hard

8098.1

4211.0

−3887.2

IFBench

1405.3

1847.0

+441.7

23

Table E.8: Six-category response classification by benchmark and model (counts; N is items × K=50)

Benchmark

SuperGPQA

Omni-MATH hard

IFBench

N

Correct

Wrong

Format

Refusal

Trunc.

API err.

GPT-5.4

25000

15841

8913

246

0

0

0

GPT-5.5

25000

16424

8205

371

0

0

0

Sol

25000

16780

7932

288

0

0

0

GPT-5.4

5000

2285

2689

26

0

0

0

GPT-5.5

5000

2648

2326

26

0

0

0

Sol

5000

2445

2555

0

0

0

0

GPT-5.4

15000

9418

4635

945

2

0

0

GPT-5.5

15000

9696

4528

771

5

0

0

Sol

15000

9111

4534

1352

3

0

0

Model

24

Table F.9: Mean strict accuracy (%) per model and change per edge (percentage points) within benchmark-native strata

Benchmark

Items

5.4

5.5

Sol

∆5.4→5.5

∆5.5→Sol

∆5.4→Sol

Difficulty easy

156

64.0

69.3

72.4

+5.3

+3.1

+8.4

Difficulty middle

204

69.8

70.8

72.6

+0.9

+1.8

+2.7

Difficulty hard

140

53.2

54.3

53.3

+1.0

−1.0

+0.1

Agronomy

10

23.2

23.4

27.4

+0.2

+4.0

+4.2

Economics

17

74.0

76.7

78.6

+2.7

+1.9

+4.6

Education

10

50.6

67.2

72.8

+16.6

+5.6

+22.2

Engineering

135

66.5

70.3

70.3

+3.8

−0.0

+3.7

History

16

65.1

84.2

82.9

+19.1

−1.4

+17.8

Law

6

37.3

61.3

55.3

+24.0

−6.0

+18.0

Literature and Arts

41

55.9

59.9

64.0

+4.0

+4.2

+8.2

Management

12

57.2

57.2

64.0

0.0

+6.8

+6.8

Medicine

57

71.3

68.1

74.4

−3.2

+6.2

+3.0

Military Science

4

54.5

27.0

43.0

−27.5

+16.0

−11.5

Philosophy

11

71.1

72.0

78.0

+0.9

+6.0

+6.9

Science

180

62.8

63.4

62.9

+0.6

−0.5

+0.1

Sociology

1

100.0

100.0

100.0

0.0

0.0

0.0

Difficulty 8.0

61

43.8

52.0

47.6

+8.2

−4.4

+3.8

Difficulty 8.5

2

49.0

50.0

49.0

+1.0

−1.0

0.0

Difficulty 9.0

32

50.4

53.8

52.9

+3.4

−0.9

+2.5

Difficulty 9.5

5

37.2

60.4

39.2

+23.2

−21.2

+2.0

Stratum

SuperGPQA

Omni-MATH hard

25

Related documents

• Architecting Conversational Data Systems for Stateless LLM APIs: The Hydration Proxy Pattern#822257
• Trace Grafting: Cross-Model Reasoning in Large Language Model Code Generation#1092500
• Trace Grafting: Cross-Model Reasoning in Large Language Model Code Generation#1092484
• When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering#1083623
• LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation#6010