Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints Haoyuan Zhu

hzhu51@sheffield.ac.uk School of Electronic and Electrical Engineering, University of Sheffield, Sheffield S10 2TN, UK

arXiv:2609.04198v1 [cs.AI] 3 Sep 2026

Jie Zhang∗

jie.zhang@ranplanwireless.com R&D Department, Ranplan Wireless Network Design Ltd., Cambridge CB23 3UY, UK R&D Department, Cambridge AI+ Ltd., Cambridge CB23 3UY, UK

Abstract Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns built to test whether an observer can read solution progress from partial reasoning traces; every threshold was fixed in advance, and neither campaign got past validating its instrument. Across 52,988 audited request attempts, samewindow repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99. Those attempts are audit volume, not sample size: the analyses rest on 31 valid task groups, 100 replay pairs, ten windows per supplementary arm, and 3,060 constructed-error judgments. The engineering was not the problem: delivery, schema validity, request hashes, and recorded metadata all sat at ceiling. Three mechanisms explain the gap: a label-to-meaning mapping that biased readouts as strongly as the signal itself; candidate gaps seven orders of magnitude below the instrument’s own noise floor; and byte-identical inputs returning different rankings, a noise that exact-permutation readouts compound. Neither metric substitution nor sampling repaired it on the tested grid (a 748,000-call simulated design passed 0 of 500 times). Preregistered follow-ups bound the problem. Waiting did not help on the days sampled (0.805 same-day versus 0.800 cross-day; replicated over five further days, one day’s own floor lower still). Switching providers did not help: four providers in three jurisdictions share the floor (medians 0.74 to 0.88), predicted by none of the metadata fields they expose. Self-hosting on batch-invariant kernels helped only while the server was quiet: concurrent load raised disagreement 8.4-fold, back to shared-endpoint magnitude. And the readout is no magnitude meter: on constructed errors with known gaps, its separation tracks error type, not size. We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist; a pilot at roughly 2% of the study’s call volume would have exposed both unreachable gates in advance. All results concern externally measured behaviour on shared serving infrastructure; they say nothing about model internals and do not assess any provider’s service quality. On a shared endpoint, a model name is not a frozen instrument; a preregistered evaluation must measure its instrument before freezing any gate on it.

1

Introduction

Language models now sit inside the measurement loop of their own field: as judges they rank candidate responses, score reasoning traces, gate training data, and drive leaderboards. Using a model as an instrument ∗ Corresponding author.

1

carries an assumption so basic it is rarely stated: that the instrument is stable. The same request, sent to the same model name, should read the same tomorrow; a frozen evaluation criterion should test the system under study rather than the fluctuating state of its measuring device. On shared inference endpoints this assumption is not innocent. The systems literature documents why: serving infrastructure batches concurrent requests, common inference kernels are not batch-invariant, and deployments change behind fixed model names (§2); and none of this is visible in the response metadata an evaluator can record, so a black-box study can observe the resulting instability without being able to attribute it. The question this paper answers is the metrological one that the judge paradigm has mostly skipped: when an LLM observer is held to the standard of a scientific instrument (preregistered thresholds, frozen protocols, no post-hoc repair), does it pass? And when it does not, can the platform’s own metadata even reveal why? We did not choose this question; we collided with it. A preregistered research programme (measuring correctness-probability paths over visible reasoning prefixes) required its observer to pass instrumentvalidation gates before any scientific claim could be tested. It never got further. Across two campaigns and 52,988 audited request attempts [index@cf4ea148 §8] , both terminated at the instrument layer: a same-window repeat-ranking gate failed at Spearman 0.400 against a frozen 0.90, and a next-day byte-identical replay gate failed at 0.78 agreement against a frozen 0.99, each time with the execution record at ceiling. Every request delivered, every schema valid, every request body hash-identical to its frozen plan, every recorded metadata field constant. The engineering record showed nothing wrong; the preregistered criteria declared the instrument unusable. This paper is the full audit of that contradiction, and of what it teaches about using black-box observers on shared endpoints. Four literatures surround the question, surveyed in §2: judge reliability (biases catalogued early; selfconsistency recently measured (Zheng et al., 2023; Haldar & Hockenmaier, 2025; Choi et al., 2026)), the nondeterminism mechanism (batch-dependent kernels, batch-invariant remedies (Thinking Machines Lab, 2025; vLLM project, 2026)), reproducibility machinery (checklists and preregistration proposals (Pineau et al., 2021; Kapoor et al., 2024; van Miltenburg et al., 2021)), and a measurement-theoretic turn (Jacobs & Wallach, 2021). Each stops short of the instrument-audit setting. Self-consistency is reported descriptively: no study we found lets the number act as a preregistered gate with authority to terminate a research line, ties it to a byte-identity audit chain, or decomposes it into same-day floor versus cross-day change. The nondeterminism literature explains the mechanism but does not quantify it for structured ranking readouts under frozen criteria, where one token flip voids a whole record. The checklists require versions and seeds but no snapshot-identity level and no measured metadata semantics, while vendors themselves document that a fixed model ID does not fix observable behaviour (Anthropic, 2026). And the design-time discipline of analytical chemistry and industrial inspection (measure the noise floor and the gap distribution first, run the criterion’s operating characteristic, only then freeze a threshold (Currie, 1999; Automotive Industry Action Group (AIAG), 2010; International Organization for Standardization, 1999)) has, as far as our retrieval found, never been applied to an LLM evaluation gate. Our own campaigns sat inside that last gap, which is precisely how a threshold comes to test nothing at full ceremony. We turn the failed programme into the audit the field lacks, in four moves: publish the failure whole, under an append-only audit chain with every excluded window disclosed and every number resolving to a SHA256-pinned archive member (§3–§4, §12); dissect the mechanism into three modes that one permutation readout amplifies into a single observable (§5–§7); attribute the instability prospectively, with preregistered same-day/cross-day, four-provider, self-hosted, and constructed-error supplements (§8, §10); and convert the evidence into reusable discipline (§9, Appendix D). The ingredient facts are not new: judge inconsistency, temperature-zero nondeterminism and its kernel-level mechanism, position bias, and reporting checklists are all established (§2, Table 1). What this paper adds is the combination none of that work executed: a fully preregistered audit of the stability assumption with frozen thresholds and no post-hoc repair, the mechanism decomposition of its failure, prospective attribution and calibration supplements, and the instrument-first discipline the failure licenses. The claims are scoped to the tested configurations throughout; the transferable product is the discipline, not a universal law. Contributions.

2

1. A fully audited preregistered negative result (§3–§4, §12): two independent instrument generations failing their frozen gates with execution records at ceiling, establishing with unusual evidential hygiene that engineering-layer perfection carries no implication about measurement-layer reliability, and that the two must be reported separately. 2. A mechanism decomposition of judge instability (§5–§7): instrument bias, near-degenerate score separations, and platform nondeterminism, unified by the amplifier property of permutation readouts; the two natural escapes (metric substitution and sample-size scaling) closed on the tested grid; and the resolution question checked from outside the instrument’s own loop with a constructederror battery (§10.3). 3. Attribution experiments (§10): a preregistered same-day/cross-day variance decomposition of judge replay stability with dependence-aware uncertainty, a four-provider characterisation showing the floor at comparable magnitude on every tested platform, and a direct measurement of whether the system_fingerprint field predicts stability (it does not, in three distinct failure modes). 4. The audit boundary, stated as levels (§8): the L0–L2 snapshot-identity ladder; a taxonomy of boundary forms found by instrumenting the audit itself (documentation contradicted by measured behaviour in four instances, a protocol-level ban on deterministic settings, bimodal rate-limit observability, aliases as moving targets); and the reporting implication that snapshot identity belongs beside temperature and seed as mandatory metadata. 5. An instrument-first evaluation discipline (§9, Appendix D): the pilot–simulate–freeze pipeline with operating-characteristic calibration, priced by this study’s own history at pilot scale (about a thousand calls against the fifty-three-thousand-attempt sequence it would have redirected); and the reporting checklist extending existing community checklists with the instrument-status items they lack. What this paper does not claim. None of the results here bear on the internal states or semantic dynamics of the observed models: every quantity is an external measurement over visible records, and the theory-testing programme that motivated the instrument lives in a separate white-box track with its own preregistration. Nor is any result a quality assessment of a named provider: the behaviours we document are consistent with the public engineering properties of shared inference services, and our task family manufactured its own worst case (§11.2). The claim is narrower and, we believe, more useful: on a shared endpoint, a model name is not a frozen instrument, and a preregistered evaluation must measure its instrument before freezing any gate on it. Sections 3–8 establish that sentence empirically; sections 9–13 turn it into practice.

2

Background and related work

Judge reliability and nondeterminism. The judge paradigm entered wide use with MT-Bench and Chatbot Arena (Zheng et al., 2023), and its systematic defects were catalogued early: position bias strong enough that reordering two candidates flips a verdict (Wang et al., 2024; Shi et al., 2025), verbosity and self-enhancement biases (Zheng et al., 2023), self-preference measurable at the mechanism level (Panickssery et al., 2024); surveys organise the space (Gu et al., 2024; Li et al., 2024), with agreement with human judgment, measured once, as the dominant criterion (Bavaresco et al., 2025). A recent line measures the judge’s agreement with itself : repeated-call self-consistency is generally low across NLG tasks (Haldar & Hockenmaier, 2025), large-scale audits separate reliability from validity (Norman et al., 2026), and itemresponse theory has been applied to judge reliability (Choi et al., 2026). Those studies characterise reliability descriptively across repeated runs, typically jointly with human agreement or fidelity, and their numbers inform but do not bind: no reported consistency value carries authority over the study that measures it. Here the same quantity is a preregistered gate with the authority to terminate a research programme, every request is byte-auditable, and the instrument is calibrated before interpretation; that enforcement difference, not the measurement of inconsistency, is the contribution. The instability’s mechanism is settled engineering knowledge: shared endpoints batch concurrent requests, kernels are not batch-size-invariant, 3

and greedy decoding atop drifting logits diverges, with batch-invariant kernels restoring bitwise equality at known cost (Thinking Machines Lab, 2025; vLLM project, 2026). Empirically, instability at nominally deterministic settings is documented for code generation (Ouyang et al., 2025) and task suites (Atil et al., 2024), contaminates security evaluations (Monteuuis et al., 2026), and cross-version drift is established for dated snapshots months apart (Chen et al., 2024); prompt-sensitivity work varies the input and measures spread (Sclar et al., 2024), while our replays hold the input fixed to the byte, the orthogonal axis. We claim no novelty for the phenomenon. Our contribution to both lines is the instrument-audit setting (Table 1): preregistered thresholds that a research programme enforced, byte-level identity that makes the instability attributable, its decomposition into bias, separation, and platform terms, and frozen-band quantification for structured ranking readouts (§5, §10). M1 is the readout-level face of position bias, adding the interaction and ranking cascade that literature rarely quantifies (§5.1); and where (Chen et al., 2024) compares different dated snapshots over months, we show exact replay failing within one dated snapshot on one afternoon (0.805), bracketing the audit problem from the opposite side. Reproducibility machinery and measurement science. Community checklists (NeurIPS’s (Pineau et al., 2021), NLP’s (Magnusson et al., 2023), REFORMS (Kapoor et al., 2024)) require versions, hyperparameters, code, and compute; none requires what §8 shows is load-bearing for API-hosted evaluation: a snapshot-identity level, the response-side model distribution, or measured metadata semantics. Harness builders document API-side irreproducibility from experience (Biderman et al., 2024), vendors document that fixed model IDs do not fix observable behaviour (Anthropic, 2026), and the nearest governance work gates model updates on the supply side (Chishti et al., 2026); our checklist (§9.7, Appendix D) extends these with the instrument-status items they lack. (Miller, 2024) supplies report-time uncertainty; our gate calibration (§9.2) is the design-time counterpart. Preregistration has been proposed and debated for NLP (van Miltenburg et al., 2021; Søgaard et al., 2023) but remains rare in executed form; this paper is among the few cases where preregistered gates were enforced to a terminal negative verdict with the full audit trail published (§11.5). The questions themselves were formalised long ago in analytical chemistry and industrial quality control: detection limits (Currie, 1999; IUPAC), gauge R&R (Automotive Industry Action Group (AIAG), 2010), operating characteristics in acceptance sampling (International Organization for Standardization, 1999), equivalence bands (Lakens, 2017). ML’s measurement-theoretic turn (Jacobs & Wallach, 2021; Freiesleben & Zezulka, 2025; Choi et al., 2026) concentrates on construct validity and report-time variance; the instrument-first discipline (noise floor and gap distribution first, operating characteristic next, threshold last) has, as far as our retrieval found, no precedent in LLM evaluation. Sections 5, 7, and 9 import it whole, with this study’s two campaign failures as the price of learning it the other way around. Positioning. This paper contributes no new judging method and no new benchmark. It contributes a fully audited negative result, the mechanism decomposition that explains it, prospective supplements that attribute it, and the design rules and checklist the evidence supports. Its nearest neighbours each hold one piece: self-consistency measurement without preregistered enforcement, nondeterminism mechanisms without readout-level audit, update governance without downstream reporting levels, measurement theory without gate design. The claim to attention is the assembled whole, checked against its own archive at every number; Table 1 states the division line by line. Table 1: Contributions relative to prior work.

4

Established before this paper

This paper adds

Judge biases: position, verbosity, self-preference (Zheng et al., 2023; Wang et al., 2024; Panickssery et al., 2024) Judges’ self-agreement across repeated calls, reported descriptively alongside human agreement or fidelity (Haldar & Hockenmaier, 2025; Norman et al., 2026; Choi et al., 2026) Temperature-zero drift traced to batch-dependent kernels; batch-invariant inference as remedy (Thinking Machines Lab, 2025; vLLM project, 2026)

a bias diagnostic inside the readout (bmap ), measured against the signal it contaminates, and the symmetrisation that cancels its first order (§5.1) the same quantity as an enforced preregistered gate (frozen threshold, no-repair rule, authority to terminate), byte-auditable, and decomposed into bias, separation, and platform terms (§4–§5) what the documented mechanism does to a preregistered evaluation: the permutation-readout amplifier, and same-day/cross-day plus four-provider floors under frozen bands (§6, §10) drift under byte-identical requests within one dated snapshot name, with every recorded metadata field constant (§5.3, §8) the instrument-status items they lack (noise floor, gap distribution, fingerprint status, snapshot-identity level), each traced to a measured failure here (§9, App. D) an executed instrument-validation campaign under that framing: gates, calibration, detection limits, audit boundary, and the full negative result disclosed (§3–§8, §12)

Behavioural drift across dated snapshots months apart (Chen et al., 2024) Reporting checklists and preregistration proposals (Pineau et al., 2021; Kapoor et al., 2024; van Miltenburg et al., 2021; Biderman et al., 2024) Judges and benchmarks framed as measurement instruments (Jacobs & Wallach, 2021; Freiesleben & Zezulka, 2025; Choi et al., 2026)

3

Measurement setup, preregistration discipline, and audit chain

The failures reported in this paper are only informative if the instrument that produced them was specified before execution, executed exactly as specified, and audited byte-by-byte afterwards. This section describes that instrument. All design constants below were frozen in configuration snapshots and preregistered gate definitions before any paid call was placed; the audit chain that enforces this is described in §3.5. Figure 1 shows the measurement chain end to end, and the terminology map in Appendix A.12 collects the paper’s recurring names so no reader needs the project history to follow the argument.

3.1

Measured object and operationalisation

The measured system is a two-model pipeline over deterministic arithmetic tasks. A generator produces, for each task, four candidate solutions written as explicit step-by-step traces (4–8 visible steps) ending in one exact rational answer. Each task has a unique canonical answer, and the only supervision label in the entire study is exact correctness of a candidate’s final answer, Yqn ∈ {0, 1}, decided by canonical rational comparison, never by a model. An external observer is then shown the task together with a visible prefix of one candidate’s trace, truncated at progress level j ∈ {1, . . . , 4} (25%, 50%, 75%, 100% of the visible steps), and is read out as a Bernoulli judgement pqnj = Pobs (Yqn = 1 | task, visible prefix). Two declarations bound everything that follows. First, every quantity in this paper is an operationalisation of an external measurement applied to a visible record; nothing observes, or is claimed to observe, an internal state of the generator (§11.3). Second, the study never reached its scientific objective: both preregistered campaigns terminated at the instrument-validation layer, because the gates below failed. The data analysed here are therefore instrument data, and are used for nothing else. 5

3.2

Three readout protocols (two instrument generations)

Generation

Protocol

Readout

Legacy D1

Single-mapping exact two-label logprob: the observer must emit exactly A or B; both exact labels are read at the single answer-token position Dual-mapping adjacent pairs: each measurement point is asked twice, with the label-to-meaning mapping reversed between arms; probabilities are clipped at ε = 10−4 before the logit Structured full-ranking output: one JSON object whose ranking field is an exact permutation of ten presented candidate aliases

r = eℓ+ /(eℓ+ + eℓ− )

D1-S (v2.3a)

I0-R judge

zsym = (z (+) + z (−) )/2, with mapping-bias diagnostic bmap = (z (+) − z (−) )/2

the permutation itself

The symmetrised protocol exists because the legacy protocol failed its stability gate in a way traceable to label semantics (§5.1); the judge protocol exists because the second campaign selected it as primary observer protocol on frozen development data (§3.4). The progression matters for what follows: each generation repaired the previous generation’s diagnosed failure mode, and each then failed on a different preregistered criterion.

3.3

Frozen configuration

Every design constant was sealed in a run-local, hash-pinned configuration snapshot before execution; the full frozen configuration of the I0-S run (task design, generator, observer, protocol, request plan, budget) and the judge-protocol campaigns’ shared wire body are reproduced in Appendix A.9. Three constants matter for what follows: the observer ran at temperature 0.0 with one-token exact-label readouts (top_logprobs 5, thinking disabled) over 80 tasks × 4 candidates, on a fixed plan of 3,312 calls under a hard cap of 3,632. The cap is enforced by a counter that reserves before each network attempt and counts retries and transport failures, so a run cannot exceed it by construction. Arm order within pairs was balanced by frozen seed: of the 1,496 pairs, 748 ran forward-first and 748 reverse-first [phase1-i0s-fresh@ac06bb97] .

3.4

Preregistered gates

Each campaign froze a battery of instrument gates before execution; the full definitions are reproduced in Appendix A. The two that decide this paper’s narrative: Gate I-3S (instrument stability, I0-S). Five subcriteria, all frozen: single-arm exact two-label coverage ≥ 0.995; pair completeness ≥ 0.990; arm label mass median ≥ 0.95 with 1st percentile ≥ 0.80; repeated-pair scalar stability, median |∆rsym | ≤ 0.03; and repeated candidate-ranking stability, within-task Spearman median ≥ 0.90 over the 10% repeated subset. Gate I-5R (next-day replay, I0-R). After a mandatory ≥ 24 h gate, 100 preregistered judge requests are replayed with byte-identical request bodies against the same model name; the frozen criterion is full-ranking agreement (exact permutation match; recorded as "structured-field agreement" in the frozen artifacts) ≥ 0.99, with every preregistered canary kept in the denominator. Missing labels invalidate the affected arm and pair; no tail-mass allocation, no imputation, and no posthoc metric substitution are permitted anywhere in the battery [phase1-i0s-fresh@ac06bb97] . One preregistration deviation in the second campaign is disclosed and bounded in §11.5: the calibration-stage freeze postdates two development results, so this paper never describes I0-R as fully pre-frozen. 6

Task

Generator

Visible prefix

Observer (black box)

exact rational arithmetic; unique canonical answer

4 candidate solutions, step-by-step traces

truncated at a progress level; label withheld

LLM on a shared endpoint; frozen request body

Frozen gate

Stability statistic

Repeat / replay

Readout

threshold fixed before execution; three-state verdict

Spearman; full-ranking agreement

same request again: same window, next day

probability or full ranking

the audited assumption: the same request, sent to the same model name, reads the same tomorrow

Figure 1: The measurement chain. A generator writes four candidates per exact-arithmetic task; a black-box LLM observer sees the task plus a truncated trace prefix and returns a probability or full ranking; repeats and replays feed a stability statistic against a threshold fixed before execution. The paper audits the italicised assumption.

3.5

Audit chain

Every network attempt appends, before and after the call, to an append-only JSONL event log inside the run directory: request body with canonical and wire-order SHA-256s, raw response with its SHA-256, UTC timestamps, verbatim provider metadata, retry count (the full field reference is Appendix C). Interrupted runs may resume only if the existing log is a strict prefix of the frozen plan. On completion a manifest records the SHA-256 of every artifact, the directory becomes immutable, and verification tooling re-hashes every sealed run, failing on any drift. The I0-S full run and the F2 replay window are anchored by manifests [phase1-i0s-fresh@207cd6eb] and [i0r-round@22c05608] .

4

Engineering correctness does not imply measurement reliability (C1)

Both preregistered campaigns produced the same shape of event: an execution record with no defect anywhere a software engineer would look, and a scientific gate that failed exactly on its frozen threshold. This section establishes that shape as a fact about the instrument rather than about our code, because every downstream claim (C2–C6) interprets it.

4.1

Two clean failures, side by side

Table 2: Execution integrity versus preregistered verdict. Left column: the I0-S full run (samewindow repeat stability). Right column: the I0-R Stage 3-F2 window (next-day exact replay). Sources: [phase1-i0s-fresh@ac06bb97] [i0r-round@55674b09] [i0r-round@e8420c2f] , , . 7

I0-S (same-window repeat)

I0-R Stage 3-F2 (next-day replay)

Planned calls Calls with a provider response Retries / transport errors / refusals Schema-valid responses Request bodies vs frozen plan Response model field system_fingerprint

3,312 fixed (cap 3,632) 3,312 / 3,312 0/0/0 pairs stored 1,496 / 1,496 plan enforced by construction constant across run —

Finish reason Exact-label coverage / pair completeness Arm label mass, median (p01) Scalar repeat stability

uniform 1.0 / 1.0

100 fixed (cap 200) 100 / 100 0/0/0 100 / 100 SHA-256 match 100 / 100 identical 100 / 100 identical 100 / 100 (both windows: null) identical 100 / 100 n/a (structured branch)

Preregistered primary verdict

0.999475 (0.997172) median |∆rsym | = 0.006710 (threshold ≤ 0.03: passes with 4.5× margin) ranking Spearman median 0.400 < 0.90 → Gate I-3S fail

n/a —

full-ranking agreement 78/100 = 0.78 < 0.99 → Gate I-5R fail

Read each column top to bottom. Everything an execution audit can check (delivery, schema validity, byteidentity of requests, constancy of the recorded metadata set: response model string, system_fingerprint, finish reason, HTTP status) is at ceiling in these two runs. The preregistered scientific criterion then fails, and not marginally: I-3S fails at 0.400 against a 0.90 threshold while the scalar readout of the same instrument repeats to within 0.0067; I-5R fails at 0.78 against 0.99 while each of the recorded metadata fields reports that nothing changed. The engineering context around the table is equally clean: 209/209 offline tests passing at the failing commits, F2 manifest artifacts verified 11/11, zero credential-shaped fields in the audit logs, no batching API, zero retries across the I0-S plan [docs-snapshot@7440c20c] . Appendix C lists the complete recorded field set; "metadata" in this paper always means that set, not an unbounded claim about everything a platform could expose. Two further properties make the pair of failures evidential rather than anecdotal. They come from different instrument generations (a symmetrised two-label logprob readout; a structured full-ranking judge) that were designed two days apart, each repairing the previous generation’s diagnosed defect. And they fail on different axes (repeat stability within a window; exact replay across a 24-hour gate) that share no statistic. 4.2

The failures were executed faithfully

A failed gate is only worth publishing if the failure was not managed away. The frozen protocols left several tempting exits, and the record shows none was taken [docs-snapshot@a021936f] : • thresholds were not lowered, and the primary metric was not switched after the fact (the tempting substitutions are quantified in §6: the same F2 drift reads 95/100 under top-1 agreement and 4436/4500 ≈ 0.986 under pairwise agreement [i0r-round@3be62ed8] , and citing either as the verdict was preregistered out); • all 22 drifted replay records stay in the denominator; nothing was deleted, re-run, or replay-selected, and no post-hoc majority vote was taken; • the two candidates whose final answers failed to parse are retained as failures (318/320 parseable), not imputed; • the fallback observer protocol (verbal confidence) was not activated: a scientific gate failure does not trigger it; • the I-3S failure is broad, not an outlier artifact: 29 of 31 valid rank groups sit below the threshold, and every prefix progress level has a median below 0.90 [docs-snapshot@7440c20c] . 8

M1 · legacy D1 (mapping-inverted repeat)

M2 · I0-S (same-window repeat)

M3 · I0-R F2 (next-day exact replay)

delivered

1814/1816

delivered

3312/3312

delivered

100/100

both labels

99.853%

pairs complete

1496/1496

metadata constant

100/100

0.0

−1.0

-0.632456 fail

1

1.0

gate 0.9

0.5

−0.5

0

repeat-ranking Spearman

within-task rank Spearman

1.0

1

0

1.0

gate 0.9

0.5

0.400000 fail

0.0 −0.5

full-ranking agreement

0

−1.0

0.8

1

gate 0.99

0.78 fail

0.6 0.4 0.2 0.0

top: execution integrity (delivery, validity, metadata constancy) — at ceiling in all three campaigns; bottom: the preregistered verdict against its frozen gate — fail in all three.

Figure 2: Three failure modes, one shape. Top: execution-integrity rates for D1, I0-S, and F2, at or near ceiling (legacy delivery 1,814/1,816; label coverage 99.853%; I0-S and F2 at 100%). Bottom: each campaign’s preregistered verdict against its frozen threshold (dashed), all failing. The contrast between the rows is claim C1. The same discipline extends backwards into the record: every abandoned window, blocked endpoint, schemadefect batch, and rejected design in either campaign is retained and disclosed (§12.1), and none of them enters any denominator in this paper. 4.3

The claim C1. On a shared endpoint, full marks on transport-level and engineering-level compliance carry no implication about measurement-level reliability. "All tests green and all hashes match" is a necessary condition for a preregistered evaluation, and nothing more. Reports must therefore state the two layers separately, and a reproducibility claim that cites only execution-layer evidence (delivery rates, schema validity, byte-identical requests, constant metadata) has asserted nothing about whether the instrument would read the same tomorrow.

The remainder of the paper dissects the gap that C1 exposes: §5 locates the mechanisms (a near-degenerate measurand; a platform that does not promise determinism; gates frozen on never-measured quantities), §6– §7 close the two natural escapes (metric substitution, more sampling), §8 states what the platform’s own metadata can and cannot certify, and §9 turns each mechanism into a design rule.

5

Three failure modes, one root-cause structure (C2)

Section 4 established that the failures are properties of the measurement. This section characterises them. We name the modes M1–M3 in the order they were encountered; each was diagnosed with the audit data of the run that produced it, and each diagnosis then shaped the next instrument generation. 9

5.1

M1: label-semantics mapping bias (legacy D1)

Phenomenon. The legacy single-mapping protocol passed its scalar repeatability criterion and failed its ranking criterion with a negative median: across inverted-mapping repeats, the within-task candidate-ranking Spearman median was -0.632 (frozen threshold ≥ 0.90), while the median absolute probability change was 0.0233, inside its ≤ 0.03 bound [phase1-oldD1@2dc38b05] . A negative median is the signature of something stronger than noise: flipping which label means "correct" tended to invert the ranking. The mundane explanation is excluded. This is not a token-emission defect: all 1,494 successful responses emitted exactly A or B at one answer position with both exact labels in the top-logprobs (rates 1.0), labels balanced (A: 734, B: 760), model string constant [docs-snapshot@a8c3e9bb] . The instrument delivered exactly the readout it was asked for; the readout itself was biased. Mechanism. Define, over an adjacent mapping pair, the symmetrised logit zsym = (z (+) + z (−) )/2 and the mapping-bias diagnostic bmap = (z (+) − z (−) )/2. If the readout carried only a correctness signal, bmap would be zero up to noise. On the 136 legacy inverted pairs it is not: median 0.250 logits, quartiles 0.187 to 0.875, 95th percentile of |bmap | at 1.391 logits: the magnitude of the bias component rivals the dynamic range of the signal. Worse, it is not a constant offset that ranking could ignore: its median is +0.531 on correct candidates and -0.250 on incorrect ones, i.e. the bias interacts with the quantity being measured. Consistently, rankings computed from forward arms agree with rankings computed from reverse arms at a Spearman median of only 0.211 [docs-snapshot@33dda04b] . A single-mapping readout therefore confounds label-direction preference with correctness; whichever mapping a protocol happens to fix becomes part of its measurand. The successor protocol (D1-S) cancels the first-order antisymmetric component by construction, with the caveat, carried through §11, that symmetrisation does not guarantee the vanishing of higher-order interactions between label tokens and prompt content. The bias itself persists undiminished in the fresh window: over the 1,496 I0-S pairs, bmap has median 0.3125, quartiles -0.1875 to 0.7500, and 95th percentile of |bmap | at 1.3125 logits (Figure 6; recomputed from the pair data at figure-build time, [phase1-i0s-fresh@bbed8c1b] ). What symmetrisation changed is that the bias no longer enters the readout, not that the observer stopped having it. 5.2

M2: repeat-ranking instability on a near-degenerate measurand (I0-S)

This is the paper’s central mechanism. The symmetrised instrument produced what looks like a paradox: scalar repeatability of median |∆rsym | = 0.00671 (4.5 times better than its frozen bound) and a repeat-ranking Spearman median of 0.400 against a 0.90 gate, with 29 of 31 valid groups below threshold. The denominator is stated exactly: of the 32 repeated rank groups, one is constant on repeat and has no defined Spearman; the frozen validity rule excludes it, never imputes it, and ties within valid groups receive average ranks [docs-snapshot@e975a840] . The resolution is in two distributions measured by the frozen failure audit [docs-snapshot@57f3389b] : What the instrument moves. Repeat deltas: quartiles 0.001–0.018, 90th/95th percentiles 0.052/0.080, maximum 0.167. Call this the instrument’s noise floor: roughly 10−3 –10−2 in probability. What there is to measure. Within a task, the four candidates’ readouts span a median range of 0.0114; and the median minimum gap between adjacent candidates is 3.04 × 10−10 , with the 75th percentile still at 0.0005. Twelve of 32 groups contain exact ties on the original readout, 7 of 32 on the repeat; only 62 of 128 candidates keep even their within-pair arm order across repeats. A ranking gate asks the instrument to resolve gaps, and here the typical gap sits four to seven orders of magnitude below the noise floor. Under repeat, candidates separated by 10−10 reorder with probability essentially 12 each; "scalar stable, ranking collapsed" is not a contradiction but arithmetic. The progressstratified medians confirm that the instrument detects order exactly where order exists to detect: Spearman medians by visible-prefix level are 0.000 (25% visible), 0.769 (50%), 0.500 (75%), 0.000 (100%). At one quarter of the trace, candidates have not yet diverged; at completion, the final answer is visible to the observer, readouts saturate, and same-correctness candidates become indistinguishable again; the ranking information lives mid-trace. 10

Upstream cause. The degeneracy was manufactured by the task design, not discovered in nature: candidate accuracy was 0.781, and only 16.25% of tasks had candidates of mixed correctness [docs-snapshot@7b8f55c0] . In roughly 84% of groups there was nothing to rank: all four candidates share a correctness value toward which their probability paths converge. The gate, however, was frozen at 0.90 before any of these intervals had been measured. The design rule this yields (§9.3, §9.6) is the detection-limit discipline of analytical chemistry: a ranking gate is only meaningful relative to the measured gap distribution of its measurand, and the gap distribution must be piloted before the gate is frozen. One caveat is owed here and repaid prospectively: both distributions come from the same observer, so on the legacy data alone this section characterises observed score separations. The controlled-error battery of §10.3 closes that loop from outside, with construction-known errors: the readout separates wrong from right at every magnitude tested, but by error type, not error magnitude. The instrument discriminates; it does not meter, and a readout that never orders by magnitude cannot certify order among near-degenerate alternatives, whatever its scalar repeatability. 5.3

M3: byte-identical inputs, drifting outputs (I0-R)

Phenomenon. The second campaign’s judge instrument, replaying 100 frozen requests after a mandatory 24-hour gate with byte-identical bodies (SHA-256-verified 100/100), reproduced the exact ranking 78 times out of 100. The 22 drifted permutations spread across 16 of 25 tasks (not one failing request) while every provider-visible metadata field was constant (§4.1). The drift is local: the Kendall-distance spectrum over replays is 78 exact, 11 at distance 1, and 11 spread over distances 2–11; top-1 agreement is 95/100 and pairwise order agreement 4436/4500 ≈ 0.986 [i0r-round@3be62ed8] . Mostly, these are single adjacent transpositions: exactly what near-degenerate subsets (M2) plus small numerical perturbations produce in a full-permutation readout. What the drift is not. We originally named this mode "overnight drift", implying attribution to backend change across the 24-hour gate. Our prospective supplement A-S1 (§10.1) falsifies that reading: replaying the same battery five times within each of two days and five times across them gives a same-day fullagreement median of 0.805 and a cross-day median of 0.800 (difference 0.005, inside the preregistered equivalence band), with the historical 0.78 sitting inside the same-day distribution itself [index@cf4ea148 appendix A.3] [repo:docs/reports/a_s1_samedayvar_r1.md] . The correct statement of M3 is: on this platform class, exact input identity does not purchase exact output identity even within an afternoon; the 24 hours added nothing. The known mechanism, batch-size-dependent kernel scheduling under shared load (§2.2), is consistent with everything we observe, and A-S2 (§10.2) shows the same floor on four providers. 5.4

The shared structure Root cause 1 (measurand). The score separations to be resolved are near zero in the readout’s own units, and the readout is in any case not ordered by error magnitude (M2, §10.3). Root cause 2 (platform). The tested shared endpoints do not promise bitwise determinism at temperature 0 and expose no auditable snapshot identity; a nonzero noise floor persisted across every window we measured, and nothing recordable at L0 could attribute or remove it (M3, §8). Root cause 3 (process). Both gates were frozen on quantities that had never been pilot-measured (a repeat-ranking statistic, I-3S, and an exact replayagreement rate, I-5R), so neither threshold was reachable by any instrument of the class being audited (§9.2).

M1 is a biased instrument, M2 a degenerate measurand, M3 an unstable platform: three distinct defects that produce the same observable in ranking readouts, because a permutation is an amplifier: an infinitesimal scalar perturbation becomes a discrete, gate-visible rank event whenever it crosses one of many near-zero gaps. This is why §9.4 recommends continuous readouts with equivalence bands over exact-match permutation gates, and why §6 shows that no metric substitution repairs the underlying degeneracy. 11

median separation 3.04e-10 (over all 32, ties included)

(a) rank groups (32 units)

one marker per rank group (n=20) exact ties, separation = 0 (n=12; off scale)

noise median 0.006710

(b) repeat pairs count (136 units)

30

panel (a) median sits ≈7 orders of magnitude below the noise median

20

repeat delta |Δr_sym| per matched pair (n=136)

10

0 10

−13

10

−11

10

−9

10

−7

10

−5

10

−3

10

−1

readout units (probability scale, log); panels share the axis, not a denominator Figure 3: The detection-limit picture behind M2, per sampling unit. (a) The 32 repeated groups’ minimum score separations, the 12 exact-tie groups as a zero-gap category a log axis cannot show. (b) The 136 repeat-pair deltas, the noise floor. Panels share the axis, not a denominator; the medians sit seven orders of magnitude apart.

6

The same drift under three metrics (C3)

A drifting instrument does not report one number; it reports as many numbers as there are ways to score agreement. This section quantifies how far apart those readings sit on the same 100 replays, because the gap between them is exactly the room available for motivated metric choice. 6.1

One batch of drift, three readings

From the F2 replay window (§5.3), scoring the identical 22 drifted permutations three ways [i0r-round@3be62ed8] : Metric

Value

Full-ranking agreement (exact permutation match) Top-1 agreement Pairwise order agreement

78/100 = 0.780000

6.2

Frozen gate 0.99

95/100 = 0.950000 4436/4500 = 0.985778

— —

Reading if taken as the verdict fail, decisively "nearly stable" "almost perfectly stable"

Why this is not a matter of metric taste

The three numbers are not three experiments; they are one experiment under three aggregations, and their spread is structural. Full-ranking agreement decays combinatorially in the candidate count: a single adjacent transposition anywhere in a 10-alias ranking voids the entire record. The q 45 scaling (45 candidate pairs at per-pair stability q, against q for pairwise agreement) is an illustrative independence bound, not an estimate: pair disturbances in a transitive ranking are neither independent nor identically distributed, so the 12

exponent overstates the decay rate; the qualitative point, that record-level agreement is the fastest-decaying aggregation of the three, does not depend on independence. The observed spectrum (0.780 / 0.950 / 0.986, with a Kendall-distance spectrum concentrated at distance 1; §5.3) is what near-degenerate subsets under a small noise floor produce: overwhelming pairwise stability, frequent record-level violations. Two consequences follow. First, the frozen full-ranking gate was the strictest possible reading of the drift: the campaign’s No-Go stands on the metric least favourable to survival, which is the direction preregistration is supposed to bind. Second, and symmetrically: had the metric been chosen after the data, two of the three readings would have supported a "nearly stable" narrative on the very replays that falsified stability at the preregistered granularity. Our protocols prohibited that substitution in advance, and this paper cites top-1 and pairwise values only as sensitivity characterisation, never as a passing verdict [docs-snapshot@a021936f] . 6.3

Rules this yields (input to §9) 1. Freeze the metric at the granularity of the downstream claim. If the conclusion consumes only the top candidate, do not gate on the full permutation; if it consumes pairwise preferences, gate on pairwise agreement with an equivalence band. A gate stricter than the claim invites unnecessary No-Gos; a gate looser than the claim invites unsupported conclusions. Both mismatches were live risks here. 2. Discrete exact-match gates are structurally fragile on shared endpoints. Any readout that maps a continuous perturbation through an argmax or a permutation will convert a platform’s noise floor (§5.3, §10) into gate-visible events at a rate the gate designer never chose. 3. A historical footnote that generalises: the original I-5 definition contained a continuous branch (mean absolute probability distance ≤ 0.02); the redesigned campaign’s structured-judge readout left only the exact-equality branch applicable [ref:project-review §3.3] . Instrument redesigns silently narrow which branches of a frozen gate remain usable; re-check the gate against the new readout before freezing, not after.

7

More sampling cannot repair it (C4)

The most natural response to an unstable readout is to average it. The original design anticipated this: its D2 protocol replaces the single-shot probability readout with M independent binary samples per measurement point, estimated with Jeffreys smoothing r̂ = (m + 12 )/(M + 1). This section shows why that route was closed: not by cost, but by arithmetic. 7.1

Setup

The frozen failure audit ran a deterministic simulation (500 replicates, seed 20260730) of the legacy D2 estimator over the I0-S repeat-task groups, using the frozen D1-S readouts as probability proxies (declared as proxies, not ground truth) and applying the frozen ranking gate (within-task Spearman median ≥ 0.90) to each simulated repeat [docs-snapshot@57f3389b] . Sampling noise falls as M −1/2 ; if instability were a noise problem, some M would pass the gate. 7.2

Result

M

Observer calls

SE at p = 0.5

Gate median

Gate q95

8 16 32 64 100 200 500

11,968 23,936 47,872 95,744 149,600 299,200 748,000

0.157 0.118 0.086 0.062 0.050 0.035 0.022

0.333 0.304 0.280 0.333 0.400 0.447 0.566

0.642 0.577 0.577 0.632 0.632 0.685 0.800

13

Pass rate 0/500 0/500 0/500 0/500 0/500 0/500 0/500

7.3

Why the pass rate is pinned at zero

Between M = 8 and M = 500 the sampling standard error improves sevenfold, the achieved Spearman median climbs from 0.33 to 0.57, and the gate pass rate does not move off zero, with even the 95th percentile of the gate statistic (0.80) still below the frozen 0.90 at three-quarters of a million observer calls. (The calls column is M × 1,496 executed pairs; a pass rate of 0/500 bounds the per-grid-point pass probability below 0.006 at 95% confidence, under the proxy model.) The reason is §5.2’s decomposition: averaging shrinks the denominator of a resolution problem (noise), but the numerator (the measured score separation, median 3×10−10 in readout units, with exact ties in a third of the groups) is a property of the sample the instrument sees. No sample size resolves a separation of zero: for the exactly tied groups, no estimator of any variance orders the pair better than a coin flip, which closes the route in principle for that third of the sample. The cost column makes the practical point redundant: the first row alone exceeds the entire I0-S campaign’s call count by a factor of 3.6, and the design was rejected before cost even entered the argument. 7.4

Boundary of the claim

This is an exploratory measurement-error and request-scale diagnostic, not an executed experiment: it neither authorises the D2 protocol nor predicts the behaviour of redesigned measurands (e.g. §9.6’s controlled-error candidates, whose gaps are constructed to be resolvable; A-S3 executed exactly that redesign, §10.3). The claim is bounded by the tested grid: for the sample this study had, no tested sampling scale (seven grid points up to M = 500, 500 simulated replicates each, frozen readouts as proxies) rescued the frozen ranking gate; and for the exactly tied third of the groups, no finite sampling scale can, because their separation is zero. Extrapolation beyond the grid is an inference from the mechanism, not a measured result. A failed stability gate still cannot in general be answered with "run it bigger": the gap distribution has to be measured first, which is the point of the pilot-then-freeze rule in §9.2.

8

What the platform’s own metadata can and cannot certify (C5)

Sections 4–7 established that the instrument drifts. This section asks the audit-layer question of whether a consumer of these APIs could have known. The answer defines the boundary of every reproducibility claim made over a shared endpoint, and it is quantifiable, because our audit chain recorded the platforms’ exposed metadata verbatim (the field set of Appendix C; claims in this section are about that set on the tested providers). 8.1

The observed facts

Constancy of everything visible, drift of the output. In the F2 replay, all provider-visible metadata was constant across both windows (response model string 100/100, system_fingerprint 100/100, both windows null, finish reason 100/100) while 22 of 100 byte-identical requests returned a different ranking (§4.1, §5.3). A metadata-constancy check of the kind audit tooling can automate would have certified this replay as clean. A fingerprint being present does not repair this. The cross-provider battery (§10.2) measured the fingerprint field itself, and it fails in three distinct ways [index@cf4ea148 appendix B.3] : Failure form

Provider (this battery)

Measurement

Absent: field never populated

mistral, qwen (also qwen in A-S1: null in 1,000/1,000) openai: 19 distinct values across one day’s battery; 8 distinct values within a single 37-call window of the aborted r2 run deepseek: one constant, information-rich value (build, quantisation, date)

no grouping computable

Present but churning: many values inside one window

Present, stable, and still insufficient

14

same-fingerprint pairs agree no better than different-fingerprint pairs (0.854 vs 0.884): the field does not predict stability full-agreement rate under a constant fingerprint: 0.879, far from the 0.99 a replay gate requires

The strongest of the three is the last: even a stable, semantically meaningful deployment identifier does not certify replay stability, because the nondeterminism lives below the deployment level (batch-shape-dependent kernels, §2.2). Recording the fingerprint is necessary for audit and insufficient for reproducibility; both halves of that sentence are now measured rather than argued. 8.2

The audit proposition On a shared endpoint, a model name is not a snapshot identity, and none of the metadata fields exposed by the tested providers upgrades it into one. Any reproducibility claim whose provenance is "same model name" (or even "same model name and same fingerprint") has, as its actual scope, the visible API behaviour of that endpoint during that observation window. Nothing further.

8.3

A three-level auditability ladder

We propose reporting the snapshot-identity level of any LLM evaluation as mandatory metadata, alongside temperature and seed: Level

Snapshot identity

What can be claimed

Grounding

L0

model name only

single-window visible behaviour

L1

provider-pinned snapshot/deployment ID

weights fixed; observable behaviour still not byte-stable

L2

self-hosted weights hash + batch-invariant deterministic kernels + pinned hardware class and framework version

replay stability far above the shared-endpoint floor while the server is quiet; exactness must be verified, not assumed

this paper’s campaigns (both were L0) vendor documentation itself: model weights are fixed for a given ID while serving infrastructure changes around them, producing "minor differences in observable behavior" (Anthropic, 2026); empirically, the deepseek row above is an L1-like case failing a replay gate directly measured on one stack only (the cross-track arm below: one model, one load pattern, driver version unrecorded); the general row is a recommendation grounded in (Thinking Machines Lab, 2025; vLLM project, 2026)

The ladder’s leverage lies in its middle level: L1 is where most careful practitioners believe they are safe, and it is not sufficient for replay-exact claims, by the platform’s own documentation and by our measurement. L2, measured. The ladder’s top level was tested with the same 100-probe battery on a self-hosted arm: an open-weights judge (weights SHA-256 recorded) on a batch-invariant serving stack (vLLM 0.11.0, batchinvariant mode enabled, pinned image digest, RTX 4090-class GPU; one identity field, the driver version, was unavailable and is disclosed as missing). The arm ran on the white-box track’s infrastructure and is cited by frozen artifact hash only, never pooled with any shared-endpoint number (registration in the citation ledger [repo:docs/citations.md] ). Quiet (serial, five passes): full-ranking agreement 0.9872 over pass pairs with well-formed replies (0.928 under the frozen criterion, which counts the judge’s occasional malformed rankings as disagreement), far above the 0.740–0.879 shared-endpoint range. Under batch pressure (eight concurrent background streams): 0.8925 clean (0.830 frozen), back inside that range; per-pair disagreement rose from 1.28% to 10.75%, a factor of 8.4. Two deliberately narrow readings follow. First, the mechanism account’s prediction of exact replay (§9.1) did not hold as stated: batch-invariant kernels were necessary but not sufficient on this stack, so L2’s claim is earned by verification, not by flag. Second, batching pressure alone suffices to reproduce shared-endpoint-magnitude instability on one model and one stack; whether it is the cause of the shared-endpoint floor remains unidentifiable from these data, and this paper does not claim it. 15

8.4

Four further boundary forms, found by trying to instrument the audit

Preparing the cross-provider battery required verifying, per provider, that each audit-relevant request field actually functions. That verification (three capability probes and one throughput probe, 270 calls in total) surfaced audit boundaries of its own kind [index@bee9bc61 appendix B.2–B.3] [repo:docs/reports/a_s2_capability.md, a_s2_capability_r2.md, a_s2_capability_r3.md, a_s2_throughput.md] : 1. Documentation is not an audit source. Documented parameter handling contradicted measured behaviour four times: fields documented as silently ignored were rejected (HTTP 400/422 on seed and on a response_format type the vendor’s compatibility guide lists as ignored); a documented hard cap was silently ignored (a completion-token limit exceeded 39-fold by a reasoning-mode default); a field documented as populated returned null. The direction varied; the presence did not. Probe per provider, per model, per field. 2. A platform can forbid determinism at the protocol level. One provider rejects temperature=0 outright (only 0.6 accepted): an audit boundary no client-side discipline crosses. 3. Rate-limit observability is bimodal. One provider publishes its quota precisely enough to predict the first throttled call; two return no rate-limit headers at all, and observed 429s carried no retry-after. Load, and therefore batch shape (the very variable the nondeterminism mechanism runs on), is invisible on half the platforms. 4. Aliases move under audit. Dated snapshot names resolve stably in the model response field; undated aliases are retireable pointers (one provider retired its rolling aliases mid-2026; another’s binding is explicitly unversioned). The audit trail records the motion only if the response-side model string is logged per call, a mandatory audit field (§9.7). 8.5

Implication for reporting

The reporting consequences of this section are checklist items 1–4 (Appendix D): snapshot-identity level, response-side model distribution, fingerprint status measured over the run, and probed parameter handling. Absent these, a diverging replication cannot even be classified as instrument drift versus platform drift; with them, the divergence becomes attributable, which is all an audit can promise on L0/L1 infrastructure.

9

Design rules for preregistered LLM evaluation (C6)

Nothing here is advice in the abstract: each rule names the failure it prevents, every failure named was purchased and audited in §§4–8, and Appendix A.11 maps each rule to its evidence. 9.1

Lock the snapshot before trusting any stability gate (← C5, M3)

An I-5-class gate on an L0 endpoint tests a proposition no one controls. Prefer self-hosted open weights, then provider-pinned snapshots (L1, which fix the weights but not the observable behaviour; §8.1), then shared endpoints only with gates over aggregated readouts (§9.5). An earlier version called L2 replay-exactness a property "by construction"; the measured arm corrected it (§8.3). Lock the snapshot, then measure the served path’s stability under representative load, rather than inferring it from the serving stack’s guarantees. 9.2

No gate freezes on a never-measured quantity (← C1, C4)

Both campaign-ending gates were frozen on quantities no one had piloted, and neither threshold was reachable by any instrument of the audited class (§4.1, §10.1): they tested nothing, at full ceremony and real cost. The repair is a fixed pipeline: pilot the noise floor and gap distribution, simulate the gate’s operating characteristic over healthy- and degraded-instrument models (healthy ≥ 0.9, degraded ≤ 0.1), freeze only if discrimination is adequate. This project possessed exactly the simulator required (§7) and used it post mortem; the pilot that finally exposed I-5R’s unreachability cost CNY 3 against 52,988 audited attempts. 16

9.3

Condition ranking gates on informative pairs (← M2)

A ranking gate’s denominator must contain only pairs the instrument could in principle resolve: pilotmeasured gap above a resolution bound δ set from the noise floor (0.0067 median, 0.080 p95; §5.2), ties and near-degenerate pairs excluded by preregistered rule before any data is seen. "Mostly below detection limit" is reported as a property of the sample (84% of groups here), not instrument failure. Where possible, check the bound from outside the loop with a constructed-error battery stratified by both error magnitude and error mechanism: 340 calls showed separation governed by error type, not magnitude (§10.3), so a single pooled detection-limit curve can be composition-confounded and is not sufficient on its own. 9.4

Prefer continuous readouts with equivalence bands (← C3)

Exact-match gates over discrete structures amplify the noise floor combinatorially (pairwise 0.986 becomes record-level 0.780; §6.2). Gate on continuous aggregates against a preregistered equivalence band, or decompose rankings into explicitly aggregated per-candidate or pairwise scores, so one token flip moves one comparison, not the whole record. The frozen metric must sit at the granularity of the claim it licenses (§6.3). This is not a licence to swap a strict gate for a friendlier statistic. When the downstream use genuinely consumes a full ranking, the operational prescription is: keep the full-ranking deliverable but make the aggregate, not the single call, the measured object. Concretely, (i) freeze k, the aggregator (Borda or median rank with a deterministic tie rule), and the gate before execution; (ii) set the equivalence band from the piloted floor (for example, aggregated disagreement at or below the pilot’s 95th percentile); (iii) decide by the TOST convention, the whole interval inside the band (§2). Sizing on this study’s own measured spectrum says the price is modest: simulating Borda-of-k readouts under the A-S1 same-day Kendall spectrum (calibrated so k = 1 reproduces the measured pair agreement of 0.7985), two independent aggregated batteries agree exactly at rate 0.94 for k = 3, 0.98 for k = 5, and 0.999 for k = 9, with within-one-transposition agreement at 0.999 by k = 5 [repo:docs/reports/aggregation_prescription.json] . A full-ranking gate that was unreachable per call (§5.3) becomes passable at five to nine calls per measurement point, which is the difference between abandoning the granularity and retaining it at a known, preregistered cost. The simulation shares §7’s evidentiary tier: it sizes a design and authorises nothing (§11.4). 9.5

Put aggregation inside the measurement definition (← M3, §10.1)

On L0/L1 infrastructure a single call is one draw from the platform’s nondeterminism distribution. Fix k, the aggregator, and the gate before execution, and keep every draw; the forbidden variants (replay-untilagreement, post-hoc majority votes, dropping discordant draws) condition on the verdict. A-S1 bounds what aggregation buys: the floor is instantaneous (0.805 versus 0.800, replicated), and scheduling remedies offer nothing that can be planned in advance: where a day did differ (r2’s day 2), nothing observable marked it beforehand (§10.1). 9.6

Make the measurand exist (← M2, upstream)

No instrument rule repairs quantities degenerate by construction. Manufacture candidate sets to be resolvable: controlled error injection, difficulty near accuracy 0.5, mixed-correctness coverage above a preregistered floor. The values this study ran with (0.781, 0.163; §5.2) should have been disqualifying at design time; the §9.2 pilot would have caught them. 9.7

Report against a fixed checklist (Appendix D)

The recurring failure behind M1–M3 is unstated instrument status. Appendix D gives the checklist this study’s evidence supports, phrased so each item is reportable from data an audit chain already collects (§3.5): adopting it costs logging, not methodology. 17

9.8

Fail closed: a guard that cannot parse must refuse (← §12.1)

Both of this study’s aborted windows trace to guards that defaulted to allow on unrecognised input (a wireshape parameter, a scheduling check; Appendix A.10). Unparseable means refused, and any parameter that load-bears for the science is verified at the wire or response level (§8.4), never assumed from SDK acceptance. We disclose both incidents because a methods paper whose tooling failed open twice, caught it, and re-froze rather than patching silently is the checklist argument in miniature.

10

Prospective supplements: decomposing, generalising, and calibrating the floor

The legacy campaigns left three attributions open: whether the replay disagreement of §5.3 was a cross-day effect or an instantaneous one; whether it was a property of one endpoint or of the platform class; and whether the readout can resolve true differences of known size at all, a question the legacy data could pose only in the instrument’s own units (§5.2). All three were answered with new, preregistered, separately archived supplements, never pooled with the historical data, which enters below only as a qualitative reference line. The main-text claims were frozen before any supplement ran, and none moved a threshold: the supplements sharpen attribution and scope, the role prospective replication should play in a measurement paper. 10.1

A-S1: same-day/cross-day variance decomposition

Design (preregistration frozen at commit ffbeb95 before execution; descriptive, no pass/fail gate). The 100 frozen judge request bodies of the F2 window (§5.3) were replayed as a fixed probe battery: five windows per day with a ≥ 2 h spacing rule, on two days, 10 × 100 = 1,000 calls against the same endpoint and dated model snapshot as the historical window. Interpretation bands were frozen in advance: same-day/cross-day median difference |d| < 0.03 reads as a nondeterminism floor, d ≥ 0.10 as a systematic cross-day component. Result. Coverage was complete (1,000/1,000 valid readouts, zero transport failures, system_fingerprint null in 1,000 of 1,000) and the decomposition is unambiguous [index@cf4ea148 appendix A.3] [repo:docs/reports/a_s1_samedayvar_r1.md] : Quantity

Value

median full-agreement, 20 same-day window pairs median full-agreement, 25 cross-day window pairs difference d top-1 agreement medians (same-day / cross-day) pairwise agreement medians

0.805 0.800 0.005 → nondeterminism floor 0.950 / 0.960 0.9859 / 0.9860

The historical 0.78 sits inside the same-day distribution (range 0.73–0.83). Twenty-four hours contribute nothing measurable: what §5.3’s window observed was the endpoint’s instantaneous nondeterminism, reachable twice in one afternoon. The metric spectrum reproduces at the same three magnitudes as the frozen window (0.780/0.950/0.986 historically; 0.805/0.950/0.986 here) without any pooling: two independent measurements of one floor. Dependence-aware uncertainty. The 45 contrasts share ten windows, so the frozen-band reading is rechecked with the window as sampling unit [repo:docs/reports/dependence_reanalysis.json] . A cluster bootstrap over windows (10,000 replicates; five per day, resampled with replacement; self-pairs dropped) puts the same-day median at 95% interval [0.77, 0.83], the cross-day median at [0.76, 0.82], and d at [-0.03, 0.04]; 87% of replicates fall inside the band. The exact reference is sharper. Over all 252 balanced relabelings of the ten windows into two pseudo-days, |d| at least as large as observed arises in 98% of splits, and no relabeling produces |d| > 0.025: the decomposition is a property of the window collection, not of the day labels. Consequences for the argument. §5.3 was renamed by this result, and §9.5 (prospective aggregation) gains its quantitative basis: the floor is per-call, so k-fold aggregation attacks it. Two days cannot falsify the existence of a stabler day somewhere; they do show that cross-day variation was indistinguishable from the same-day floor whenever we looked (§11.4). 18

Replication, and the day that stood out (r2). A second preregistered round (frozen at commit 29b5962) ran the same frozen battery over five days spanning 2026-08-14 to 08-26: 2,500 of 2,500 valid readouts, fingerprint null throughout [index@cf4ea148 appendix D.3] [repo:docs/reports/a_s1_samedayvar_r2.md] . Sameday median 0.8150 (50 pairs) versus cross-day 0.8100 (250 pairs), d = 0.0050: the frozen replication rule fired. The rounds remain separate units, never pooled; "seven days of coverage" describes scope, not a sample. With 25 windows the dependence-aware intervals tighten: d at [-0.01, 0.03], entirely inside the band (99.8% of replicates), and no relabeling among 10,000 seeded balanced ones produces |d| > 0.02 [repo:docs/reports/dependence_reanalysis.json] . The round also asked what r1 could not: whether the days themselves are alike. They are not. Per-day same-day medians run 0.8250 / 0.7800 / 0.8000 / 0.8200 / 0.8250, a spread of 0.0450 against the 0.03 band, and the frozen reading fired: a day stood out (day 2, the least stable). Both findings stand because they are different quantities: the cross-day median never leaves the floor, while the per-day floor itself wanders beyond the band. "Measure on a good day" is therefore not even well-posed prospectively: nothing recorded distinguishes day 2 in advance, and its pre-flight canary passed all four assertions. Every waiting-related statement in this paper stays bounded to the days sampled. Disclosure. The first A-S1 attempt (v1) is retained as a failed instrument window (100 calls, 100 responses, zero valid readouts: an SDK-shape parameter never entered the wire body; §9.8, §12.1), permanently excluded and archived separately. In r2, the first day-4 attempt (2026-08-20) is likewise retained, voided, in its own archive: two executor processes concurrently wrote one run directory, window 2 received 285 requests with 21 probes carrying conflicting readings, and day 4 re-ran four days later under a new run group and a runbook guard against concurrent executors (§12.1). One governance deviation ships with it: the round’s authorisations were pre-signed for seven candidate dates (the researcher was travelling), trading the daily human checkpoint for the frozen runbook’s machine preflights; one date expired unused. One limitation is inherited by construction: byte-equivalence of the replayed bodies with the F2 originals is asserted by construction, not verified against archived wire bytes, because the legacy archive records SDK-parameter shape (§11.5). 10.2

A-S2: the floor is a platform-class property

Design (preregistration frozen at cdc012f; descriptive; per-provider analyses never merged). Four providers of mutually distinct families received the same 100-probe battery (deepseek-v4-flash, mistral-small-2603, gpt-4.1 returning dated snapshot gpt-4.1-2025-04-14 in every response, and qwen3.7-plus-2026-05-26 as the A-S1 anchor) through byte-equivalent requests differing only in model and base URL: five windows × two days × four providers = 4,000 calls, same-day spacing 2.02–2.04 h, cross-day gap 24.14 h. Three frozen readings: a platform-class test (max per-provider same-day median < 0.95), a per-provider decomposition (bands as in A-S1), and a fingerprint-predictiveness test. Result.

Coverage

4,000/4,000 :

responses,

zero

transport

failures

[index@bee9bc61 appendix B.3]

[repo:docs/reports/a_s2_xprovider_r3.md]

Provider

valid rate

deepseek mistral openai qwen

0.990 1.000 1.000 1.000

median same-day agreement

d (same - cross)

decomposition

0.879 0.740 0.850 0.775

0.000 0.000 0.000 -0.005

nondeterminism floor nondeterminism floor nondeterminism floor nondeterminism floor

All four providers entered the platform-class test (valid rate ≥ 0.95 precondition), and its frozen reading fired: max = 0.879 < 0.95 → platform-class property, a verdict name whose scope is exactly what was measured: these four providers, this 100-probe structured-ranking battery, these observation windows — valuable evidence about shared serving infrastructure, not a census of it. The window-level bootstrap and exact relabeling of §10.1, applied per provider, give d intervals within ±0.03 for every provider (in-band fractions 0.91–0.95; all 252-split relabeling probabilities 1.0; no relabeling exceeds |d| = 0.03) [repo:docs/reports/dependence_reanalysis.json] . No provider approaches replay-exactness; every provider’s crossday behaviour equals its same-day behaviour; the ordering readout’s stability differs by provider (0.74–0.88) 19

but its character does not. The fingerprint field completes the picture in three failure forms, absent / churning / stable-but-insufficient, measured in §8.1. Reading discipline. These are stability comparisons, not quality comparisons: the four models sit at different capability tiers by design choice, and no cross-provider number above compares judging accuracy (§11.4). The qwen column is an independent re-measurement of A-S1’s endpoint in new windows: 0.775 against A-S1’s 0.805, read side by side and never pooled. One verbatim-reproducible failure is disclosed rather than patched: a single probe on deepseek returned a bare JSON array instead of the required object in all ten windows, byte-identical every time, scored as invalid per the frozen rule, ten out of ten, on both days. Excluded windows. Two aborted Day-1 attempts (238 and 618 calls) are retained for disclosure and enter no denominator: the first exposed a rate-limit quota that a frozen premise had assumed away, the second a fail-open scheduling guard (§9.8); both triggered full re-freezing rather than in-place amendment (§12.1). 10.3

A-S3: a controlled-error battery puts resolution outside the loop

Everything above measures the observer against its own readouts: §5.2’s gap distribution and noise floor come from the same instrument, so "the gaps were too small to resolve" is, on the legacy data alone, an operational diagnosis rather than an independently identified fact. A-S3 constructs the measurand instead. Design (preregistration frozen at c145f0c; single provider, single session; not a stability measurement). The 25 task stems of the I0-R battery were reused (re-derived in exact rational arithmetic, 25 of 25), but every candidate was rebuilt: at each of 85 (task × visible-prefix) variants, one correct candidate plus nine slip candidates from a five-type catalogue of human-plausible errors, injected at rotated step positions, so the relative error ε of every slip is known by construction before any call. All ten candidates share identical wording; arithmetic is the only signal, and the plan builder refuses to execute on a battery-hash mismatch. Four passes over the 85 variants gave 340 calls, each returning one full ranking; a judgment is whether the correct candidate strictly outranks a given slip candidate (85 × 9 × 4 = 3,060 judgments; chance 0.5) [index@cf4ea148 appendix C.3] [repo:docs/reports/a_s3_detection.md] . Result. Coverage 340/340 valid readouts, zero transport failures, system_fingerprint null in 340 of 340 (L0): ε bin −3

below 10 (outside the main verdict) [10−3 , 10−2 ) [10−2 , 10−1 ) [10−1 , 100 ) [100 , ∞)

judgments

separation

96 556 932 1,068 408

0.7604 0.7914 0.8069 0.9860 0.9020

Wilson 95% [0.6661, 0.8347] [0.7556, 0.8231] [0.7803, 0.8309] [0.9770, 0.9915] [0.8692, 0.9272]

Every group is resolved (Wilson lower bound above 0.5), so the frozen main verdict is the fourth of the preregistration’s four named outcomes: the battery did not bracket the detection limit; it certifies only that the limit lies below 10−3 relative error for constructed slips. The within-type contrast governs. Error size and slip type are correlated by construction, so the preregistration ordered a same-type contrast and declared it decisive over the pooled table. It reverses the naive reading: in four of five slip types, separation does not rise with ε, and the largest-error cell of one type (off-by-one at ε ≥ 1, n = 36) sits at 0.333, below chance. The pooled rise across bins is mostly composition (which slip types populate which bin), not error magnitude. What the battery certifies is a discriminator sensitive to error type and surface form, not a meter that orders candidates by the size of their error. Two further disclosures: replay disagreement across the four passes was 19 of 765 pairs (0.0248), minutes apart and therefore not comparable to any same-day agreement number in §10.1–§10.2; and the battery’s scope is fixed by its construction, with uniform rewritten wording (a deliberate difference from the I0-R candidates, whose styles vary), a prefix-length parameter that does not inherit the archived progress scale, one provider, one task family. Extrapolating its detection limit to the original I0-R candidates would need a separate argument (§11.1). 20

nondeterminism floor 0.150

same-day (n=20) cross-day (n=25)

I0-R F2 = 0.78 (reference)

0.125

d (within − between)

median 0.800

cross-day

median 0.805

same-day

d ≥ 0.10 cross-day component

0.100 0.075 0.050 d = 0.005

0.025

|d| < 0.03 nondeterminism floor

0.000 −0.025 −0.050

0.74

0.76

0.78

0.80

0.82

0.84

agree_full (complete-case) per window pair complete-case and invalid-as-disagreement coincide: 1000/1000 readings valid | new windows; pooled with neither the F2 window nor the failed v1 round

Figure 4: A-S1, the same-day/cross-day decomposition of full-ranking agreement (§10.1): 20 same-day and 25 cross-day window-pair agreements from ten 100-probe windows, with medians and frozen bands; the historical F2 value (dotted) sits inside the same-day distribution. New windows, pooled with nothing.

platform class property same-day (n=20) cross-day (n=25)

qwen L1, no fp section 7.2: d = -0.0050 nondeterminism floor

0.65

0.70

0.75

0.80

0.85

0.90

0.95 (section 7.1)

section 7.2: d = +0.0000 nondeterminism floor

0.95

agree_full (complete-case) per window pair

max(M) = 0.8788

0.85 0.80 0.75 0.70 qwen

openai L1, fp

0.90

openai

section 7.2: d = +0.0000 nondeterminism floor

< 0.95 platform class

0.95

mistral

mistral L1, no fp

≥ 0.99 provider specific

deepseek

section 7.2: d = +0.0000 nondeterminism floor

median(A_within) per provider

1.00

deepseek L0, fp

section 7.3, agreement among probe pairs sharing one fingerprint: deepseek 0.8790 (n=4455), openai 0.8538 (n=4146); mistral, qwen return no such field. invalid-as-disagreement differs for deepseek 0.8700 (median A_within); complete-case shown | providers are never pooled: max(M) is an extremum over four separate measurements, not a summary of one.

Figure 5: A-S2, cross-provider stability of the same battery (§10.2): per-provider window-pair agreement distributions and medians, never merged; none approaches the 0.99 a replay gate would need. Stability comparison only; capability tiers differ by design, and nothing here compares judging accuracy.

21

11

Threats to validity and the boundary of claims

11.1

External validity

The mechanism evidence (M1, M2) comes from one observer family (qwen3.7-plus) on one shared endpoint, over one task family (synthetic, deterministic rational arithmetic with exact canonical answers), one prompt template, and one language. A-S2 extends the stability characterisation to four providers across three jurisdictions, and A-S1 gives the same-day noise model for the primary endpoint; neither extends the M1/M2 mechanism measurements (mapping bias, degeneracy decomposition) beyond their original instrument, and we claim nothing broader for them. The A-S3 detection-limit numbers are similarly bounded: they hold for constructed slip candidates with uniform rewritten wording on one provider in one session, and do not transfer to the stylistically heterogeneous I0-R candidates without a separate argument; its four-pass replay disagreement (0.0248, minutes apart) is not a same-day stability measurement and is never compared with AS1/A-S2 numbers. The L2 control point has now run (§8.3): one open-weights model on one batch-invariant serving stack, under two load conditions, executed on the white-box track’s infrastructure and cited by frozen artifact hash. Its boundaries are inherited by every statement that uses it: a single model, stack, GPU class, and concurrency setting; one identity field (driver version) unrecorded on that stack, disclosed; and its design supports sufficiency (batching pressure can produce shared-endpoint-magnitude instability) but not attribution (that the shared-endpoint floor is batching), which this paper accordingly never claims. 11.2

The measurand was ours

The near-degeneracy at the heart of M2 was manufactured by our task design (candidate accuracy 0.781, mixed coverage 0.163; §5.2), not discovered as a law of LLM-as-judge practice. Deployed judge pipelines with well-separated candidates need not exhibit ranking collapse of this severity. The claim we defend is invariant to that: a ranking gate is only meaningful relative to the measured gap distribution of its measurand, and a study that has not measured its gap distribution does not know which regime it is in. We were in the degenerate regime and did not know until the failure audit; that ignorance, not the degeneracy itself, is the transferable hazard. 11.3

What this paper does not claim

None of the results here bear on the internal states, representations, or semantic dynamics of the observed models; every quantity is an operationalisation of external measurement over visible records (§3.1). This paper is not a test of any theory of model internals; the project’s white-box track addresses those questions separately, under its own preregistration, and no data flows between the two. Nor is any result a service-quality assessment of a named provider: the behaviours documented are consistent with the public engineering properties of shared inference services (dynamic batching without batch-invariant kernels), our provider exclusions were protocol- or account-level (§8.4), and the capability tiers of the A-S2 models were deliberately not equalised (§10.2). Finally, this paper does not conclude that LLM-as-judge evaluation is unusable; it concludes that judge instruments have measurable noise floors, gap distributions, and audit boundaries, and that gates written without measuring them test nothing. 11.4

Statistical limitations

The I0-S ranking verdict rests on 31 valid groups of 32 (one group’s repeat ranking is constant, its Spearman undefined, reported as undefined and not imputed to 0 or 1). The D2 simulation (§7) uses frozen D1-S readouts as probability proxies, declared as proxies; its conclusion is about the frozen gate on this measurand, not about redesigned measurands. The legacy 136-pair diagnostic behind M1’s bmap quantiles is exploratory (preregistration-ineligible) and is used only for mechanism characterisation, never as gate evidence. The §9.4 aggregation sizing simulates on the measured A-S1 same-day spectrum of one endpoint and one battery, under an adjacent-transposition surrogate calibrated at k = 1; its k values size a design on that floor and transfer to other endpoints only after their own pilots. The supplements’ medians summarise window-pair sets that share windows and are not independent, so §10 reports window-level cluster bootstrap intervals 22

and balanced day-relabeling references alongside the frozen bands [repo:docs/reports/dependence_reanalysis.json] . The same-day/cross-day arms sample two days (r1) and five days (r2), never pooled; cross-day statements are bounded by those designs and never extrapolated to "any future day", a boundary r2 made concrete by measuring a day whose own floor stood out (§10.1). The A-S2 judge shares a model family with the generator of the historical candidates (gpt-4.1 judging gpt-4o-mini outputs), a self-preference confound we disclose; it can shift which permutation a judge prefers, not whether the same request reproduces the same permutation, which is the only quantity A-S2 reads.

11.5

Preregistration deviations, disclosed

Two, both bounded. In the second legacy campaign, the calibration-stage freeze postdates two developmentstage results; the record marks it a preregistration-timing deviation, and this paper accordingly never describes I0-R as fully pre-frozen (the F2 replay protocol itself, and its 0.99 gate, were frozen before the replay window opened). In A-S1, byte-equivalence between the probe battery and the historical F2 wire bodies is asserted by construction rather than verified against archived bytes, because the legacy archive records requests in SDK-parameter shape (§10.1); every stability statistic in §10 is internal to the new windows and unaffected. The supplements’ own aborted windows and their causes are disclosed in §12.1 and were handled by re-freezing, not amendment.

12

Disclosure, data, and reproducibility

Every number in this paper resolves to a file inside a frozen, hash-manifested archive; every archive is listed below with its package SHA-256; and every run that was excluded from analysis is disclosed here with the reason and the disposition. The index that governs all of it is itself version-pinned [index@cf4ea148] . Evidence packages. Legacy campaigns (sealed 2026-08-03):

Package

SHA-256

phase1-oldD1.zip

859ba16e971c3ca2f87e7db c75106d00340852801acc7c df1a6d97714c68ae56 e41bb822aebb310e38f2618 df20b0b942d160342036578 5e69100f01bc9c8e44 d0cff5cdcd4d3005338f34d 9a077f8a2e552311a2f6367 62c922460f4b65040d 87179236958ec882f091a57 db1ceafcfd07b44d202889e 3f485c67549114e24c

phase1-i0s-fresh.zip

i0r-round.zip

docs-snapshot.zip

Members

Bytes

83

2,989,738

failure mode M1; legacy ledgers; excluded windows

41

2,215,094

458

66,243,938

204

1,112,797

failure mode M2; I-3S primary evidence; D2 simulation failure mode M3; I-5R primary evidence; metric spectrum preregistration texts, gate definitions, closeout reports

Prospective supplements (this study’s new windows, sealed 2026-08-05/-26): 23

Role in this paper

Package

SHA-256

a-s1-r1.zip

196ab2e72cd10c72324ed269163 feb39dd95a5ca2c9da11e644b01 e1e7670d3a 4caac1c8e270b82ab47dd92a17e 513d17609a5ec2e8634332781a5 6f27a93def ebde3c776727e774444528ed189 6c021347066c1a40d371bab8ca4 3ee6d81d88 f5f4a64f3e8d0531a8daf528bd3 a942869003dc0d34bbd92fa1561 9195e64115 07ae32a50776a1e19b5bf7ed1ce ebc334bf94d4c5981eab78c5e17 037445e365 a333e15fe5e3b5712f40a964860 f4bad155d58d498fce3de09580e 82e91d647f 682b9b73e138c1268b68f233651 fcba10c43109bc3df81d64a51e5 74d6c5348a 35a5450d33f74a1511dbc0c67c1 040372fff0d6d149addd7f24cd7 74028b80ac

a-s1-v1-failed.zip

a-s2-probes.zip

a-s2-abandoned.zip

a-s2-r3.zip

a-s3.zip

a-s1-r2.zip

a-s1-r2-voided.zip

12.1

Members

Role

39

A-S1 primary evidence (§10.1)

19

A-S1 v1 failed instrument window (disclosure only)

50

capability & throughput probes (instrument calibration)

35

A-S2 v1 (never run) and r2 aborted window (disclosure only) A-S2 primary evidence (§10.2)

30

7

A-S3 controlled-error battery, primary evidence (§10.3)

70

A-S1 r2 five-day replication, primary evidence (§10.1)

13

A-S1 r2 voided day-4 first attempt (disclosure only)

Excluded data (retained for disclosure; never in any denominator)

Nothing was deleted, edited, or repaired anywhere in either campaign or in the supplements; "excluded" always means retained, disclosed, and kept out of every denominator, with costs left on the ledger. Eleven exclusion events span the study, from Round 0’s duplicate candidates through the supplements’ three aborted or voided windows; the full register (event, disposition, standing obligation) is Appendix A.10, sourced from [index@bee9bc61] §4 and appendices A–D and the per-run ledger [repo:docs/costs.md] .

12.2

Request-attempt ledger

These figures exist so that no reader mistakes audit volume for sample size; no scope was ever pooled into another, and none of the totals counts successful or billable calls. The legacy workspace logged 4,088 attempts in the D1 group and 3,314 in the fresh I0-S group (phase-1 total 7,402), plus 45,586 in the I0-R campaign: 52,988 attempts, of which 52,574 received a provider response, 413 failed in transport, and one carries neither event. That residual category exists by construction: the cap counter reserves before each network attempt (§3.5), so an attempt interrupted between reservation and outcome is counted without an event record, and the arithmetic closes at 52,574 + 413 + 1 = 52,988 [index@bee9bc61 §8] . The supplements add 9,467 attempts: A-S1 with its failed v1 window (1,104), the A-S2 probes, aborted windows, and battery (5,126), A-S3 (340), and A-S1 r2 with its voided day (2,897). The four primary batteries completed 1,000/1,000, 4,000/4,000, 340/340, and 2,500/2,500 with zero transport failures (§10); per-window detail ships in the supplement’s ledgers.

12.3

Reproduction package and access tiers

The supplement is a purpose-built public-layer package, not a copy of the archives. Two tiers, decided before writing began: 24

Tier

Contents

Public

every manifest; all derived data; reports; sealed configuration snapshots; cost ledgers; preregistration texts (frozen commits); figure-rebuild code and tests; for A-S2, provider response headers sanitised at capture time by a credential denylist and secret-shape heuristic raw/api_events.jsonl (verbatim provider responses) for every run; one operator-side launcher script that embeds a local account path (frozen bytes are never edited, so it ships hash-only whole)

Restricted

The public package includes the SHA-256 of every restricted-layer file, so the integrity chain is verifiable endto-end without access to raw responses. The complete run-directory index (110 runs with archive, manifest anchor, and standing classification) ships in the supplement as the store’s own searchable index; the packages table above is its compact summary. Every figure in this paper rebuilds from the public layer alone with one command and zero network calls; the package asserts its own byte size and re-verifies its own hashes at build time. Five further figure views (progress-stratified stability, the metric-sensitivity bars, the sampling-scale panels, the drift-locality panels, and the audit-ladder diagram) rebuild with the same command and ship in the supplement; each duplicates numbers or structure kept in the main text. Raw provider responses are held as sensitive research data, available on request, subject to a provider terms-of-service review at publication time. 12.4

A provenance gap, honestly registered

Nineteen phase-1 runs predate the repository’s git history and reference configuration files outside their run directories by absolute path; those files changed later (28 items). In-run evidence shows zero mismatches, but the exact bytes those runs executed under cannot be reproduced, so every conclusion drawn from them is capped at exploratory-diagnostic status. The two runs sealed inside their run directory with exact hash agreement (the I0-S canary and full run) are the only phase-1 runs cited as primary evidence [index@bee9bc61 §6] . 12.5

Cost ledger

Reported so that the scale of evidence is not mistaken for scale of spend, and because cost asymmetry is itself a finding: the entire F2 refutation window cost CNY 0.296479056 [i0r-round@e8420c2f] . The legacy campaigns closed at CNY 30.868827240 (the sealed ledger through Stage 3-E4 [docs-snapshot@c01bb1c6] plus that F2 window, the only later legacy spend); the I0-S full run alone cost 1.742772 [docs-snapshot@7b8f55c0] . The supplements added ≈ 37.73: A-S1 r1 ≈ 2.973, A-S2 probes and aborted windows ≈ 4.63 (sunk, disclosed), A-S2 r3 ≈ 20.623, A-S3 ≈ 0.932, and A-S1 r2 ≈ 7.43 plus its voided day-4 attempt ≈ 1.145 [repo:docs/costs.md] . Provider billing statements remain authoritative. Every live window was authorised in writing before execution, against a per-line budget envelope, with a hard request cap wired into the executor; the itemised per-window ledger and the authorisation ledger ship in the public layer.

13

Conclusion

Two preregistered audits of black-box LLM observers on shared endpoints terminated at their instrument gates, with execution records at ceiling and scientific verdicts far below threshold. The dissection is compact. Transport perfection implies nothing about measurement reliability (C1). The failures decompose into a biased readout, near-degenerate score separations, and an instability that byte-identical inputs could not cross, amplified by a permutation readout into one observable (C2). Metric substitution did not repair them (C3), nor did scale on the tested sampling grid (C4). The platform’s own metadata cannot certify what reproducibility claims need, even when the fingerprint field is populated and stable (C5). Each mechanism converts into an enforceable design rule, summarised as a reporting checklist whose every item traces to measured evidence in this paper (C6). The supplements sharpen attribution within the tested configurations: the historical replay disagreement sits inside its endpoint’s same-day nondeterminism floor, replicated over a second five-day round in which one day’s own floor stood out (A-S1); the floor appears at comparable magnitude on all four providers measured, unrepaired by any exposed metadata (A-S2); and the readout separates constructed errors by type, not magnitude (A-S3). A cross-track L2 arm closes the ladder from above, on one stack and one load pattern only (§8.3): self-hosted batch-invariant serving beats every shared 25

endpoint while quiet, loses exactness anyway, and falls back into the shared-endpoint range under concurrent load. Whether the shared-endpoint floor originates in batching, kernel scheduling, or deployment rotation is still not identifiable from outside: batching pressure is demonstrated sufficient for instability of that magnitude, and no measurement here selects it as the cause. The proposition we intend to be quoted, scoped to what was measured (the tested providers, their exposed metadata fields, this ranking protocol, these observation windows), is this: On a shared endpoint, a model name is not a frozen instrument. A preregistered evaluation must measure its instrument (noise floor, gap distribution, metadata semantics) before freezing any gate on it, because a gate frozen on an unmeasured quantity tests nothing at full ceremony. The constructive path exists at every layer and is priced in this paper: snapshot locking with verification under load (§9.1); operating-characteristic calibration before any freeze, at roughly 2% of this study’s call volume (§9.2); detection-limit conditioning and continuous readouts where ranking is the deliverable (§9.3– 9.4); prospective aggregation as the unit of measurement on shared infrastructure (§9.5). None of it requires new access, only the discipline of treating an evaluation instrument as metrology treats any instrument: as the first object of study. These conclusions are statements about black-box observation of language models through shared serving infrastructure, under this paper’s operationalisation; they assert nothing about the internal dynamics of the observed models, for which a separate white-box track with its own preregistration exists. The supplements A-S1/A-S2/A-S3 are complete and reported in full (§10); the checklist (Appendix D) and the evidence packages (§12) are released for reuse.

References Anthropic. Model IDs and versioning. Claude Platform documentation, https://platform.claude.com/ docs/en/about-claude/models/model-ids-and-versions, 2026. Vendor documentation. Berk Atil et al. Non-determinism of “deterministic” LLM settings. arXiv preprint arXiv:2408.04667, 2024. Automotive Industry Action Group (AIAG). Measurement Systems Analysis (MSA) Reference Manual, 4th edition, 2010. Industry standard handbook. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. arXiv:2406.18403. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782, 2024. Lingjiao Chen, Matei Zaharia, and James Zou. How is ChatGPT’s behavior changing over time? Harvard Data Science Review, 6(2), 2024. arXiv:2307.09009. Mohd Sameen Chishti, Damilare Peter Oyinloye, and Jingyue Li. Test before you deploy: Governing updates in the LLM supply chain. In 2nd International Workshop on Large Language Model Supply Chain Analysis (LLMSC, co-located with FSE), 2026. arXiv:2604.27789; workshop paper. Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, and Bugeun Kim. Diagnosing the reliability of LLM-as-a-judge via item response theory. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. arXiv:2602.00521. 26

Lloyd A. Currie. Detection and quantification limits: origins and historical overview. Analytica Chimica Acta, 391(2):127–134, 1999. Timo Freiesleben and Sebastian Zezulka. The benchmarking epistemology: Construct validity for evaluating machine learning models. arXiv preprint arXiv:2510.23191, 2025. Jiawei Gu et al. A survey on LLM-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. Rajarshi Haldar and Julia Hockenmaier. Rating roulette: Self-inconsistency in LLM-as-a-judge frameworks. In Findings of the Association for Computational Linguistics: EMNLP, 2025. arXiv:2510.27106. International Organization for Standardization. ISO 2859-1:1999. Sampling procedures for inspection by attributes – Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection, 1999. IUPAC. Limit of detection (L03540). IUPAC Compendium of Chemical Terminology (the “Gold Book”), https://goldbook.iupac.org/terms/view/L03540. Standard terminology entry. Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2021. arXiv:1912.05511. Sayash Kapoor et al. REFORMS: Consensus-based recommendations for machine-learning-based science. Science Advances, 10(18), 2024. doi: 10.1126/sciadv.adk3452. Daniël Lakens. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4):355–362, 2017. Haitao Li et al. LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024. Ian Magnusson, Noah A. Smith, and Jesse Dodge. Reproducibility in NLP: What have we learned from the checklist? In Findings of the Association for Computational Linguistics: ACL, 2023. arXiv:2306.09562. Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024. Jean-Philippe Monteuuis, Cong Chen, and Jonathan Petit. The great pretender: A stochasticity problem in LLM jailbreak. arXiv preprint arXiv:2605.14418, 2026. Justin D. Norman, Michael U. Rivera, and D. Alex Hughes. Reliability without validity: A systematic, large-scale evaluation of LLM-as-a-judge models across agreement, consistency, and bias. arXiv preprint arXiv:2606.19544, 2026. Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. An empirical study of the non-determinism of ChatGPT in code generation. ACM Transactions on Software Engineering and Methodology, 2025. doi: 10.1145/3697010. arXiv:2308.02828. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). Journal of Machine Learning Research, 22(164): 1–20, 2021. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.11324. 27

Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of AACL-IJCNLP, 2025. arXiv:2406.07791. Anders Søgaard, Daniel Hershcovich, and Miryam de Lhoneux. A two-sided discussion of preregistration of NLP research. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2023. arXiv:2302.10086. Thinking Machines Lab. Defeating nondeterminism in LLM inference. Engineering blog, https:// thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/, 2025. Technical report; not peer reviewed. Emiel van Miltenburg, Chris van der Lee, and Emiel Krahmer. Preregistering NLP research. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021. arXiv:2103.06944. vLLM project. vLLM documentation: Batch invariance; reproducibility. latest/features/batch_invariance/, 2026. Software documentation.

https://docs.vllm.ai/en/

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2305.17926. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLMas-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. arXiv:2306.05685.

A

Gate definitions, frozen anchors, and the verdict map

Every confirmatory statement in the main text traces to a gate or reading rule that was frozen before the data it judges existed. This appendix states each one in full: statistic, threshold, denominator, invalidity and tie handling, freeze anchor, and outcome. The verbatim frozen texts (gate documents, preregistrations, sealed configuration snapshots) are members of the archives listed in §12 and ship in the supplement; each entry below names its anchor so the correspondence is checkable file by file. A.1

Legacy D1 stability gate (first campaign)

Statistic. Median within-task rank Spearman between repeat readouts under the single-mapping protocol (B.1). Threshold. ≥ 0.90, frozen before execution. Denominator. All repeated task groups with defined Spearman; missing labels invalidate the affected arm. Outcome. -0.632456 at near-ceiling delivery (1,814/1,816; coverage 99.853%) [phase1-oldD1@2dc38b05] ; diagnosed in §5.1. A.2

Gate I-2R: candidate distinctness (second campaign, Stage 2)

Rule. Within every task, the presented candidates must be pairwise distinct as rendered strings; a round in which any task presents duplicate candidates fails whole. Outcome. Round 0 failed (40 of 80 tasks carried pairwise-duplicate candidates); all 160 generations were excluded, and Round 1 reran with fresh identities and tasks (§12.1). Nothing from Round 0 enters any denominator. A.3

Gate I-3S: instrument stability (I0-S full run)

Five subcriteria, all frozen in the sealed gate document [phase1-i0s-fresh@ac06bb97] : 28

(b) bias interacts with the measurand (legacy window only)

I0-S window (n=1496 pairs) — phase1-i0s-fresh

100

median 0.3125 Q25/Q75 -0.1875/0.7500 P95|b| 1.3125

0.75 0.50

50

0 legacy diagnostic (n=136 pairs) — docs-snapshot; exploratory only

pairs

10

median 0.2500 Q25/Q75 -0.1875/0.8750 P95|b| 1.3906

legacy b_map median (logits)

pairs

+0.531

0.25 0.00 -0.250

−0.25 −0.50

5 −0.75

fwd vs rev ranking ρ = 0.211 single-mapping repeat ρ = -0.632456

0 −2

−1

0

1

2

correct candidates

mapping-bias diagnostic b_map (logits)

incorrect candidates

Figure 6: The mapping-bias diagnostic bmap (half the difference of the two arms’ logits; zero for an unbiased readout) in the legacy and fresh I0-S windows, faceted, never pooled. Right: the bias interacts with candidate correctness (legacy window; exploratory stratified view, no interaction test claimed).

#

Subcriterion

Frozen threshold

Measured

Verdict

1 2 3

Single-arm exact two-label coverage Pair completeness Arm label mass, median and 1st percentile Repeat scalar stability, median |∆rsym | Repeat ranking stability, within-task Spearman median

≥ 0.995 ≥ 0.990 ≥ 0.95 / ≥ 0.80

1.000 1.000 0.999475 / 0.997172

pass pass pass

≤ 0.03

0.006710

pass

≥ 0.90

0.400

fail

4 5

Denominators. 1,496 executed pairs; the repeat subset is the frozen 10% re-measurement (136 pairs); subcriterion 5 runs over 32 repeated rank groups, 31 with defined Spearman (one constant on repeat, excluded by the frozen validity rule, never imputed; 29 of 31 below threshold). Handling. Missing labels invalidate arm and pair; no reallocation, imputation, or metric substitution; ties receive average ranks (sealed implementation in the supplement). A.4

Gate I-5R: next-day exact replay (I0-R, Stage 3-F2)

Design. After a mandatory ≥ 24 h gate, the 100 preregistered judge requests are replayed with byte-identical bodies (SHA-256 verified 100/100) against the same model name. Statistic. Full-ranking agreement (exact permutation match; "structured-field agreement" in the frozen artifacts). Threshold. ≥ 0.99. Denominator. All 100 replays, canaries kept. Outcome. 78/100 = 0.78 → fail [i0r-round@e8420c2f] . Prospectively frozen inside a campaign not described as fully preregistered (§11.5). 29

A.5

A-S1 frozen interpretation bands (descriptive; no pass/fail)

Statistic. d = median(Awithin ) − median(Abetween ), where A is full-ranking agreement between window pairs (20 same-day, 25 cross-day pairs from ten windows). Frozen bands. |d| < 0.03 reads as a nondeterminism floor; d ≥ 0.10 as a systematic cross-day component; intermediate values are reported without a name. Outcome. 0.805 - 0.800 = 0.005 → nondeterminism floor (§10.1). Preregistration frozen at commit ffbeb95; run manifests 069fd364... (Day 1), 47a78e6a... (Day 2). Replication round (r2). Frozen at 29b5962 with two further reading rules: a replication reading (same bands, five days, separate unit, no pooling) and a per-day spread reading (per-day same-day medians; spread beyond the 0.03 band names the least stable day). Outcomes: d = 0.0050 → replicated; spread 0.0450 → a day stood out (day 2); five day-manifests 38a73362.../6800952c.../ba032eaf.../26c25814.../35dbe1a2... (§10.1). A.6

A-S2 three frozen readings (descriptive; per provider, never merged)

Preregistration frozen at commit cdc012f; run manifests dabb7183... / 2a37b642.... (i) Platformclass test: given valid-readout rate ≥ 0.95 on every provider, read maxp median(Apwithin ) against 0.95; outcome 0.8788 → fired. (ii) Per-provider decomposition: A-S1’s bands per provider; four floors (d = 0.000/0.000/0.000/-0.005). (iii) Fingerprint predictiveness at probe-level pairing; outcomes in §8.1. A.7

A-S3 reading rules (controlled-error battery)

Preregistration frozen at commit c145f0c; plan 487e4e30..., battery 8f41e2fd..., manifest 2dd41dd5.... (i) Per-bin resolution: a bin is resolved when the 95% Wilson lower bound of its separation exceeds chance 0.5. (ii) Main verdict, one of four frozen outcomes over the four ε bins at or above 10−3 ; realised outcome: limit_below_smallest_bin_tested. (iii) The within-type contrast governs interpretation over the pooled table (frozen §14.3). (iv) The four-pass replay disagreement and the below-10−3 group are reported but enter no verdict. Outcomes in §10.3. A.8

The verdict map

Every headline verdict, its freeze anchor, and where it is reported: Verdict

Frozen anchor

Execution anchor

Reported

Legacy D1 gate fail (-0.632)

sealed gate doc 2dc38b05...

§5.1

I-3S fail (0.400 vs 0.90) I-5R fail (0.78 vs 0.99) A-S1 nondeterminism floor (d = 0.005) A-S2 platform-class + fingerprint readings A-S3 limit not bracketed; type-governed separation L2 arm: quiet 0.9872 / loaded 0.8925 (clean subset); batch invariance necessary, not sufficient

sealed gate doc ac06bb97... frozen replay plan (§3.4) prereg @ ffbeb95

legacy run directory (§12, phase1-oldD1) run manifest 207cd6eb... run manifest 22c05608... manifests 069fd364..., 47a78e6a... manifests dabb7183..., 2a37b642... manifest 2dd41dd5...

A.9

prereg @ cdc012f prereg @ c145f0c cross-line prereg @ 33a5f25... (white-box track)

run manifest 5d34c129...; report hash in the citation ledger

§4.1, §5.2 §4.1, §5.3 §10.1 §10.2, §8.1 §10.3 §8.3

Frozen wire configuration by campaign

The I0-S full run executed under the configuration below, quoted from the authorised snapshot sealed inside the run directory and hash-pinned by its manifest [phase1-i0s-fresh@75ea5677] : 30

Item

Value

Task seed / design

20260729; 80 tasks × 4 candidates = 320; visible steps 4–8; prefix grid J = 4 gpt-4o-mini-2024-07-18 (family openai-gpt-4o), temperature 0.8, strict structured outputs qwen3.7-plus-2026-05-26 (family qwen3.7-plus), shared OpenAI-compatible Chat Completions endpoint (DashScope compatible mode), temperature 0.0, max_completion_tokens 1, top_logprobs 5, thinking disabled Protocol d1_symmetric; mappings {A7→+, B7→−} and {B7→+, A7→−}; clip ε = 10−4 ; repeat fraction 0.10; no imputation; no candidate reuse 320 generator + 2,720 main observer arms (1,360 pairs) + 272 repeat arms (136 pairs); 3,312 fixed calls, hard cap 3,632, concurrency 1 Hard limit CNY 9.99; per-run authorisation row binding models, counts, cap, and budget

Generator Observer

Measurement

Request plan

Budget

The judge-protocol campaigns share one frozen wire body shape, quoted from the sealed request plans (members of the archives in §12): temperature 0.0, max_completion_tokens 256, response_format {"type": "json_object"}, seed 2026080302, thinking disabled via the provider’s documented switch. Provider deviations, all frozen before execution and disclosed: mistral rejects seed with HTTP 422, so that field is dropped for that provider only, and its rate-limit headers impose a measured 1.5 s send interval; deepseek disables thinking through extra_body. For the legacy archives the recorded request hash pins the SDK-parameter shape rather than wire bytes, a limitation stated wherever byte-equivalence is claimed (§11.5).

A.10

The excluded-data register

Nothing was deleted, edited, or repaired anywhere in the study (§12.1); every exclusion event, its disposition, and its standing obligation: 31

Event

Disposition

Standing obligation

Stage 2 Round 0: 40 of 80 tasks produced pairwise-duplicate candidates (Gate I-2R fail)

all 160 generations excluded; R1 reran with fresh identities and tasks; an exclusion registry ships with the package all 160 calibration candidates voided; E2-R1 changed key order only retained as raw failure; shared endpoint adopted as an independent replacement window retained as raw failure records

never merged with R1; cost retained

voided before live; re-frozen; both manifests retained exploratory only

first version never used for execution never grounds for declaring I-3S passed recorded as a rejected design

Stage 3-E2: wire key-order serialisation defect Stage 3-B0: 13 calls, all HTTP 403 (workspace endpoint) Phase-1 transport-blocked window: 408 attempts, 0 responses; observer all-403 window: 1,817 attempts Stage 3-F1 first freeze: missing local parser/alias-map metadata Legacy D1-S diagnostic (136 pairs) Rejected candidate-reuse route (2 dry runs) A-S1 v1 Day-1 window: 100 calls, 100 responses, 0 valid readouts (an SDK-shape parameter never entered the wire body, so thinking mode stayed enabled and every response hit the token limit) A-S2 r2 Day-1: aborted after 238 attempts (provider rate-limit quota exposed two frozen premises as false) A-S2 r3 first Day-1: aborted after 618 attempts (a scheduling guard parsed plan IDs by fixed position, never matched A-S2’s shape, and silently allowed a 10.6-minute window gap against a frozen ≥ 2 h rule) A-S1 r2 first Day-4 (385 attempts): two executor processes concurrently wrote one run directory; window 2 received 285 requests, and 21 probes carry conflicting readings

fresh restart adopted (legacy provenance unrecoverable, §12.4) permanently excluded; r1 re-froze with the wire defect fixed

permanently excluded; defect disclosed; CNY 0.141878400 retained never cited as capability evidence

not research diagnostics

disclosed as an instrument-failure exhibit; the archived request hash pins SDK-parameter shape, not wire bytes

run left unsealed; never enters any denominator; one in-flight plan ID permanently retired

disclosed; redesign went through full re-freezing, not amendment

run left unsealed; excluded; one in-flight plan ID permanently retired; the guard was rewritten to refuse what it cannot parse

disclosed; the frozen schedule itself was correct and unchanged

run left unsealed; voided whole and archived separately (a-s1-r2-voided.zip); day 4 re-ran four days later under a new run group, after a runbook guard against concurrent executors

permanently excluded; cost retained; the round’s pre-signed seven-date authorisations (researcher travelling; one date expired unused) are disclosed with it

32

Rule ↔ evidence map (§9)

A.11 Rule

Prevented failure

Evidence

9.1 snapshot locking

replay gates on uncontrolled propositions unreachable gates at full ceremony detection-limit violations in denominators combinatorial amplification of the noise floor single-draw measurements on nondeterministic infrastructure degenerate-by-design quantities unauditable instrument status silent default-allow in evaluation tooling

§5.3, §8.1–8.3, §10.1–10.2

9.2 pre-freeze OC calibration 9.3 informative-pair conditioning 9.4 continuous readouts 9.5 prospective aggregation 9.6 measurand construction 9.7 checklist reporting 9.8 fail-closed guards

A.12

§4.1, §5.2, §7, §10.1 §5.2, §10.3 §6 §5.3, §10.1 §5.2 §3.5, §8, §12 §12.1, A.10

Terminology map

Name

What it is

Home

D1, D1-S

first campaign’s readouts: single-mapping two-label logprob; symmetrised successor first campaign’s full symmetrised run (3,312 calls) second campaign: structured full-ranking judge I0-R’s next-day byte-identical replay window (100 replays) the two decisive instrument gates failure modes: mapping bias; near-degenerate separations; replay drift the paper’s claims supplements: same/cross-day; four providers; constructed errors snapshot-identity levels of the auditability ladder mapping-bias diagnostic; symmetrised readout fraction of replays returning the exact reference permutation

§3.2, §5.1

I0-S I0-R F2 I-3S, I-5R M1–M3 C1–C6 A-S1/S2/S3 L0–L2 bmap , zsym full-ranking agreement

B

Readout protocol derivation

B.1

Two-label logprob readouts (legacy D1 and D1-S)

§3.3–§3.4 §3.2, §5.3 §5.3 §3.4, A.3–A.4 §5 §4–§9 §10 §8.3 §3.2, App. B §3.4, §6

The legacy readout instructs the observer to answer with exactly one of two labels and reads both labels’ log-probabilities at the single answer-token position: r = eℓ+ /(eℓ+ + eℓ− ), exact (no sampling); a missing label invalidates the arm. Its gate outcome (A.1) showed the readout confounds label-direction preference with the correctness signal (§5.1), motivating the symmetrised successor: each measurement point is asked twice with the label-to-meaning mapping reversed, each arm’s probability is clipped at ε = 10−4 and taken through the logit z = logit(clip(r, ε)), giving zsym = (z (+) + z (−) )/2,

bmap = (z (+) − z (−) )/2.

An unbiased readout has bmap = 0 up to noise; its measured distribution is the M1 evidence (§5.1). Arm order was balanced by frozen seed (748 forward-first, 748 reverse-first); a missing label invalidates arm and pair, with no imputation. The frozen 10% repeat subset grounds Gate I-3S subcriteria 4–5 (A.3). 33

B.2

Structured full-ranking judge (I0-R) and the A-S3 judgment

The judge returns one JSON object whose ranking field must be an exact permutation of the ten presented aliases; parsing is all-or-nothing, and an invalid readout is retained as a failure, never retried into validity. The readout is the permutation; derived statistics are full-ranking agreement (the frozen decision metric), top-1 agreement, and pairwise agreement (1 minus normalised Kendall distance) (§6). The protocol was selected on frozen development data before the confirmatory stages (§11.5). A-S3 reuses this wire contract on constructed candidates: one judgment is the event that the correct candidate strictly outranks a given slip candidate, and a bin’s separation is the fraction of its judgments that hold, with a 95% Wilson interval against chance 0.5; bin membership is fixed by the constructed relative error, known before any call (§10.3).

C

Audit-chain field reference

Every network attempt appends one pre-call and one post-call record to an append-only JSONL event log inside its run directory. The fields, all recorded verbatim at call time:

Field

Content

Checklist item

request body canonical request SHA-256 wire request SHA-256

the exact JSON sent, stored whole hash of the key-sorted serialisation hash of the order-preserving serialisation sent on the wire unmodified provider bytes, with SHA-256 UTC, before and after the call copied verbatim copied verbatim, including null prompt/completion token counts as returned copied verbatim as received attempts consumed by this logical request credential-shaped values dropped at capture (§12.3)

16 16 16

raw response timestamps response model system_fingerprint usage finish reason HTTP status retry count response headers

16 16 1, 2 3 24 16 16 16 16

Rules around the log, each load-bearing for a claim in the main text: reserve-before-network cap (a counter reserves before each attempt and counts retries and transport failures, so a run cannot exceed its frozen cap; checklist 17); strict-prefix resume (an interrupted run resumes only if the existing log is a byte-exact strict prefix of the frozen plan; checklist 17); fail-closed guards (unparseable input is refused; the one historical fail-open guard is disclosed in A.10 and was rewritten; §9.8, checklist 18); sealing (on completion a manifest records every artifact’s SHA-256, the directory becomes immutable, and verification tooling re-hashes every sealed run; checklist 19); two-layer reporting (execution integrity stated separately from scientific verdicts; C1, checklist 19). The implementation ships in the supplement with per-file hashes; the field list can be checked against any public-tier run and the restricted-tier hashes (§12.3).

D

Reporting checklist for preregistered LLM evaluations

Each item is a report-this field, not a best practice; an evaluation that cannot fill an item should say "unmeasured", which is itself a report. Trace: the section whose evidence justifies the item. 34

# 1 2 3 4

5 6

7 8 9 10 11 12 13 14 15

16 17 18 19 20 21 22 23 24

Report

Trace

D.1 Instrument identity Snapshot-identity level: L0 (model name) / L1 (pinned snapshot or dedicated deployment) / L2 (self-hosted; weights hash, framework, hardware class, determinism settings) Requested model string and the distribution of response-side model strings over the run Fingerprint field status measured over the run: absent / churning (distinct-value count) / stable; never taken from documentation For every load-bearing request parameter: the probed behaviour on this provider and model (accepted / rejected / silently ignored) D.2 Pre-freeze instrument measurement Noise floor of the readout: repeat-delta distribution from a frozen pilot battery, per aggregation level if §9.5 applies Gap distribution of the measurand: within-group ranges, minimum-gap quantiles, tie rates; where possible a constructed-error pilot stratified by both error magnitude and error mechanism/surface form (a single pooled detection-limit curve can be composition-confounded) Fraction of the intended denominator below the declared resolution bound δ, and the preregistered exclusion rule For platform-stability gates: the same-window agreement floor measured on this endpoint D.3 Gate design Operating characteristic of every confirmatory gate: pass rates under healthy- and degraded-instrument models, with seed and code The granularity match: which downstream claim consumes the gated statistic, and why metric granularity equals claim granularity For discrete exact-match gates (discouraged): the implied per-comparison stability after combinatorial decay Freeze timestamps for every gate and threshold, and any deviation D.4 Measurement definition The unit of measurement: single call, or prospective aggregate (k, aggregator, tie rule) fixed before execution Handling of invalid readouts: retained-as-failure vs excluded, fixed in advance; no imputation The prohibition list in force (no replay-selection, no post-hoc metric substitution, no threshold movement) D.5 Execution audit Append-only event log with the Appendix C field set (canonical and wire-order request hashes, raw response hash, UTC timestamps, verbatim metadata, retry counts) Hard request cap (reserve-before-network) and its relation to the frozen plan; resume rule (strict-prefix only) Guard behaviour on unparseable input: fail-closed, with evidence the guards were exercised Execution-integrity summary separated from scientific verdicts D.6 Disclosure and reproduction Excluded-data register: every excluded run with reason, disposition, standing obligation; costs retained Request-attempt ledger by scope, with the non-pooling declaration (audit volume ̸= sample size) Reproduction package tiering: public contents; restricted layer listed by hash Provenance status of every cited run: sealed in-run vs external-pointer (capped at exploratory) Cost ledger with the authoritative-source declaration, including excluded and aborted runs

§8.3 §8.4(4) §8.1 §8.4(1–2)

§5.2 §5.2, §10.3

§9.3 §10.1 §9.2 §6.3 §6.2 §11.5 §9.5 §4.2 §4.2, §6.2

§3.5 §3.3, §3.5 §9.8 §4.3 §12.1, A.10 §12.2 §12.3 §12.4 §12.5

Adoption note. Items 1–4 and 16–19 are collectible by instrumentation alone; 5–11 need a pilot battery (roughly 2% of this study’s call volume; §9.2); 12–15 and 20–24 are writing discipline. Nothing requires access beyond what the evaluated API already returns.

35

Related documents

• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints#927061
• Characterization-Guided GPU Fault Resilience in NVIDIA MPS#229468
• Behemoth: Benchmarking unlearning in LLMs using fully synthetic data#208831
• Constraints on the Intranight Optical Variability of Intermediate-Mass Black Hole Candidates#822575
• Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking#321913