Stable Within, Unidentified Across: Semantic Identification of Benchmark Effects and Rankings

S TABLE W ITHIN , U NIDENTIFIED ACROSS : S EMANTIC I DENTIFICATION OF B ENCHMARK E FFECTS AND R ANKINGS Xi Qin Wuhan University

arXiv:2608.19269v1 [cs.SE] 18 Aug 2026

A BSTRACT Evaluation conclusions depend on evaluator-controlled semantics: legal references, scoreability, and aggregation. We call an artifact-defined endpoint Q(s) evaluation-semantically identified when it is invariant over a declared family. A frozen 217-row analysis appears stable within its restricted contract family. In TraceElephant, Fasym yields precise task-disjoint adoption effects from −40.76 to −43.03 pp, whereas both jointly frozen contrast-comparable treatments in Fcc yield exactly zero with [0, 0] intervals. Their union Faudit therefore does not identify T , even though Fcc identifies T = 0. A factorial audit attributes 100% of the disagreement to one-sided terminal deletion, with zero nonterminal contribution. Rankings remain specification-dependent across the symmetric treatments. This is a localized non-implication witness and reusable family-indexed audit, not a prevalence estimate, exhaustive semantic taxonomy, or independent replication.

1

I NTRODUCTION : ROBUSTNESS I S N OT I DENTIFICATION

An evaluation maps fixed artifacts and predictions through evaluator-controlled semantics to legal references, scoreable rows, row outcomes, and aggregate endpoints. Robustness checks commonly vary seeds, thresholds, resampled rows, or a declared set of contracts and then summarize the result as robust. Yet every analyzed cell can share the same unexamined semantic operation. In a process-level diagnostic benchmark, for example, a trace may support an earliest anomaly, a causal event set, a submission event, a terminal state, several legal alternatives, or no scoreable localization. The resulting effect or ranking is therefore a property of the artifacts, predictions, and evaluation semantics jointly. Qin and Tong (2026) study whether an observable trajectory determines the scored diagnostic object and whether heterogeneous realizations agree under a fixed contract. Here, we address a distinct downstream question: whether effects and rankings remain invariant across evaluator semantics while holding the candidate and prediction substrate fixed. The 217-trajectory substrate permits these questions to be separated experimentally: the former concerns identification of the scored object, whereas the present analysis concerns identification of conclusions induced by an evaluator. A particularly close contemporary domain-external audit makes the shared empirical pattern explicit. Abdulaziz and Bondarev (2026) hold weakly supervised video-anomaly scores fixed and recompute AUC over global, per-category, and within-video frame-pair populations. They report rank reversals and, within each evaluated backbone family, no pooled-AUC resolution at the sub-point margins used for recent state-of-the-art claims. Predictions need not change for an evaluator-controlled support rule to change the scientific conclusion. Our target is the formal inference license behind that pattern: family-indexed identification for effects and rankings, finite specification variation separated from sampling uncertainty, outcome-independent audit admissibility, and a diagnostic disagreement localized to an exact row-level semantic operation. We audit one narrow inference license—the permission to promote a result from what the analyzed specifications establish to a broader scientific interpretation: within-family stability

?

=⇒

cross-semantic identification. 1

(1)

The answer is family-relative. We use evaluation semantic identification to mean invariance of a scientific evaluation conclusion over a declared family of admissible meanings, rather than statistical identification of a fixed estimand from a data-generating distribution. The key question is therefore not whether a result is “robust” without qualification, but over which semantic family it is identified. We make that scope explicit with Fasym , Fcc , and Faudit in Section 2. We establish a localized non-implication witness, not a prevalence estimate or an exhaustive semantic taxonomy. This framing connects partial identification and multiverse analysis, which make assumption-indexed conclusions explicit (Tamer, 2010; Steegen et al., 2016; Simonsohn et al., 2020), to measurement and construct-validity work, which asks whether an operationalization represents the intended object (Jacobs & Wallach, 2021; Raji et al., 2021; Bean et al., 2025). Our audit target is the evaluator-controlled map from fixed artifacts and predictions to legal references, eligibility, row scores, denominators or weights, and aggregate endpoints. The preserved evidence chain assigns distinct roles to the restricted-family premise, the V1 prediction-free diagnostic event, the V2 conditional result, its correction, the symmetric falsification controls, and ranking non-recovery. Our contributions are: • a specification-indexed formulation of evaluation semantic identification, with three explicit families that distinguish within-family identification from cross-family extrapolation; • a finite identified-set criterion and audit-admissibility discipline for outcome-independently justified semantic alternatives; • the empirical separation of a negative, precise T over Fasym , IT (Fcc ) = {0}, and |IT (Faudit )| > 1, together with ranking non-identification inside the symmetric family; and • a fully localized real-data witness—from a prediction-free NO _ GO through jointly frozen controls and factorial deletion—that turns an abstract identification failure into a rowauditable mechanism.

2

E VALUATION S EMANTIC I DENTIFICATION : W HAT M UST B E S HOWN

2.1

E VALUATOR - SEMANTIC MAP

Let xi be a fixed artifact and pim method m’s frozen prediction. An evaluation specification s controls a semantic map Ms , a row-scoring rule ℓs , and an aggregation rule As :

Ms (xi ) = Hi (s), ei (s) , Yim (s) = ℓs pim , Hi (s) ,
(2) Q(s) = As {Yim (s) : ei (s) = 1} . Here Hi (s) is the legal reference family and ei (s) records endpoint eligibility; As may also encode denominators, task weights, and aggregation support. As one concrete aggregation-support example, global, per-category, and within-video AUC read the same frame scores over different comparison populations (Abdulaziz & Bondarev, 2026); in Eq. (2), that frame-pair population is a coordinate of As . Equivalent-reference rules, partial credit, abstention or invalid disposition, empty-family treatment, weighting, and special-object comparability can likewise be coordinates of s. The identification question therefore arises whenever fixed artifacts and predictions pass through evaluator-controlled semantics before an endpoint is reported. 2.2

I NSTANTIATED DIAGNOSTIC ENDPOINTS

Our audit instantiates Eq. (2) with row → observed execution trace → contract-defined legal eventhypothesis family → frozen prediction → binary row score. Each row has a pre-existing candidate union and seven frozen prediction records. Here s = (C, P, S) indexes the declared audit family: object contract C filters candidates into legal hypotheses, disposition policy P determines how empty legal or predicted families enter the score, and terminal treatment S determines comparability of structurally special candidates. We focus on PageRank versus Earliest because it is the only pair that strictly reverses across the eight original contracts; the complete registry is in Appendix C. Write the policy-specific denominator as I(s). Then X 1 µ bm (s) = Yim (s), R(s) = rankm∈M µ bm (s). (3) |I(s)| i∈I(s)

2

For task-aware PageRank p and Earliest Anomaly e, define the pair margin ∆(s) = µ bp (s) − µ be (s). For a common frozen background b ∈ B and adoption state a ∈ {0, 1}, let sb,a = (Cb,a , Pb , Sb ), where Cb,0 and Cb,1 differ only in whether the adoption restriction is active. Define T (b) = P ∆(sb,1 ) − ∆(sb,0 ) and T = |B|−1 b∈B T (b). We keep T and R separate: a contrast can vanish while score levels and rankings still change. 2.3

T HREE DECLARED SEMANTIC FAMILIES

Let C = {Cb,a : b ∈ B, a ∈ {0, 1}} be the eight frozen contract cells, P the three disposition policies, Sasym the original one-sided terminal treatment, and Skeep , Sexclude the two contrast-comparable treatments. We declare Fasym = C × P × {Sasym }, Fcc = C × P × {Skeep , Sexclude }, (4) Faudit = Fasym ∪ Fcc . For the task-disjoint target effect, T is negative and precise throughout Fasym : the three policy values range from −43.03 to −40.76 pp, and every task-cluster interval excludes zero. In Fcc , T (b) = 0 for every background, policy, and symmetric treatment, so the effect’s finite identified set is the singleton IT (Fcc ) = {0}. The union contains both the negative asymmetric effects and zero, hence |IT (Faudit )| > 1. This is the audited extrapolation failure: identification inside Fasym or Fcc does not imply identification over Faudit . It does not deny that T is point identified at zero inside Fcc . By contrast, |IR (Fcc )| > 1: rankings remain non-identified even between the symmetric treatments. 2.4

D ISAGREEMENT- WITNESS PRINCIPLE

For any deterministic artifact-defined endpoint Q(s), define its finite empirical identified set over a declared family F as IQ (F) = {Q(s) : s ∈ F}. (5) Point identification requires |IQ (F)| = 1. If two audit-admissible specifications s1 , s2 ∈ F satisfy Q(s1 ) ̸= Q(s2 ), then |IQ (F)| ≥ 2. Moreover, for any literal expansion F ′ ⊇ F, IQ (F) ⊆ IQ (F ′ ); expansion cannot restore point identification. Thus one admissible disagreement witness establishes non-identification, and the remaining burden is to justify its admissibility. The same logic clarifies the inference license from a restricted family. For a bounded scalar endpoint, let DQ (G) = sups,s′ ∈G |Q(s) − Q(s′ )|. If F0 ⊊ F, then even DQ (F0 ) = 0 does not by itself bound DQ (F) more tightly than the endpoint’s a priori range. A tighter bound requires a coverage, smoothness, or structural-invariance assumption linking F0 to the added semantics. Arbitrarily narrow sampling intervals conditional on specifications in F0 cannot supply that missing link. A finite specification interval summarizes variation over declared definitions; a paired or task-cluster bootstrap interval summarizes sampling variation for a fixed estimand and resampling design. Under a superpopulation reading, the latter treats observed tasks as exchangeable draws from a target task population while keeping the semantic specification fixed. The two axes answer different questions and are reported separately. 2.5

AUDIT ADMISSIBILITY

A specification is audit-admissible relative to a stated endpoint and inference license when it is both procedurally admissible and substantively live under an outcome-independent domain rationale. Procedural admissibility requires that: (i) the semantic degree of freedom can be described without using the alternative’s outcomes; (ii) data, methods, predictions, and resampling remain fixed; (iii) the change targets the exposed semantic layer; (iv) alternatives are jointly frozen before either outcome is computed or inspected; (v) every declared alternative is reported; and (vi) claims remain relative to the declared family. Substantive liveness remains a domain argument. Here it is supplied by restoring terminal comparability, the only semantic layer changed by the two controls. For the transport interpretation under audit, we use a contrast-comparability condition. If the contrast changes the adoption restriction while holding treatment of structurally special terminal candidates fixed, then a terminal candidate z whose identity and NOT _ APPLICABLE status are unchanged must 3

Benchmark ranking is indexed by specification: R(C, P, S)

Object contract C

Disposition policy P

Terminal semantics S

Downstream endpoints

Which diagnostic objects are legal?

What scores when the legal family is empty?

Is terminal NA retained or excluded symmetrically?

Aggregate effect Absolute scores Ranking / identified set

Within-family bootstrap precision does not integrate over an omitted semantic layer.

Figure 1: Evaluation specification has three declared layers. The object contract C defines legal diagnostic objects; disposition policy P defines how empty legal families enter the score; and terminal treatment S defines whether terminal NOT _ APPLICABLE candidates remain comparable. Each layer can affect effects, score levels, rankings, and finite empirical identified sets. The layers define the audited family, not an exhaustive ontology. satisfy EligS (z | b, a=0) = EligS (z | b, a=1), unless adoption-dependent terminal eligibility is explicitly part of the estimand. Retaining z on both sides and excluding it on both sides are distinct construct interpretations satisfying this condition; a domain definition can reject either by respectively excluding or requiring terminal NOT _ APPLICABLE objects. The original asymmetric V2 remains a valid conditional joint (C, S) specification, but it cannot by itself license transport with terminal comparability fixed. The terminal-neutral study is classified exactly as a post-hoc motivated, prospectively frozen robustness control: the earlier audit supplied its motivation, and the two direct symmetric controls were frozen together and both reported before either result was inspected. Figure 1 summarizes the three coordinates of this instantiated family.

3

I NSTANTIATING THE AUDIT: O NE T RACE , T HREE S EMANTIC T REATMENTS

The general map in Eq. (2) can vary through many semantic coordinates. Our disagreement witness instantiates one: whether a structurally special terminal candidate remains comparable across an adoption contrast. Table 1 follows one fixed schematic trace across three treatments. Only terminal comparability changes, exposing the one-sided operation directly.

Table 1: One fixed schematic trace under three terminal treatments. Original V2 changes terminal comparability on one side; the two jointly frozen controls retain the terminal candidate on both sides or exclude it from both. No treatment is asserted universally correct.

4

A real trace worked end to end. Frozen task-disjoint trace 004ac8f1... supplies a preserved row-level counterpart to Table 1. Under background R0T0, PageRank predicts terminal event e0047 and Earliest predicts nonterminal event e0013. In original V2, both are legal on the inclusive side, giving scores (1, 1) and pair margin 0; the adopted-only filter then deletes e0047 but retains e0013, giving (0, 1) and margin −1. This single-row adoption effect is therefore −1 under all three policies. Under NA _ KEEP _ BOTH, both sides score (1, 1); under NA _ EXCLUDE _ BOTH, both score (0, 1). Each symmetric row effect is exactly 0. The predictions, nonterminal legal objects, and row remain fixed; only the terminal candidate’s cross-side comparability changes. The same decomposition applies at larger scale. The primary study freezes 217 rows, seven prediction rules, and eight object-contract cells. The empty-family control declares three policies: baseline empty/empty credit, conservative empty-no-credit, and localization conditional on a nonempty legal family. TraceElephant adds the asymmetric treatment and the two direct symmetric treatments above. We therefore write both the target effect and the ranking as functions of (C, P, S) rather than as context-free properties of a method and dataset.

4

E STABLISHING THE P REMISE : A R ESTRICTED FAMILY L OOKS ROBUST

4.1

E IGHT FROZEN CONTRACTS

Within the original contract family, the seven-method ranking is close to stable. Eight cells produce two tied weak orders; 20 of 21 method pairs have a robust strict relation; and the only changing pair is a strict reversal. Cross-cell Kendall τb ranges from 0.9048 to 1. Mini ranks first and Terminal Event Only second in every cell. PageRank and Earliest occupy ranks three and four, exchanging order with the adoption clause: PageRank leads in adoption-inclusive cells, while Earliest leads in adopted-only cells. The PageRank-minus-Earliest point specification interval is [−3.23, +8.76] pp. The average adoptedminus-inclusive effect is −7.72 pp, with a 10,000-draw stratified paired linear-percentile 95% interval [−12.10, −3.92] pp (seed 20260816). The complete observed eight-cell sign pattern appears in 24.1% of bootstrap draws. The simultaneous specification-plus-sampling envelope is [−13.36, +18.89] pp. The pooled effect is heterogeneous: AgentRx contributes −2.40 pp with interval [−5.82, 0] pp and no within-corpus reversal, whereas TELBench contributes −10.42 pp with interval [−17.01, −4.86] pp. Their interaction is +8.02 pp with interval [+1.47, +14.95] pp; the complete reversal is concentrated in TELBench.

4.2

E MPTY- FAMILY POLICY AND ROW- LEVEL MECHANISM

Changing only the empty-family endpoint expands the finite empirical identified set. Baseline empty/empty credit reproduces the crossing and yields T = −7.72 pp. Conservative empty-no-credit retains a negative average effect, −3.46 pp with interval [−5.99, −1.15] pp, but PageRank leads in every contract cell. Conditional nonempty localization yields −0.89 pp with interval [−4.96, +2.87] pp and PageRank remains clearly ahead in levels. Across all 24 contract-policy analyses there are five weak orders, 15 of 21 robust pair relations, and six strict-reversal pairs. Summary ranges and policy-level effects are in Appendix D.1; complete cell-level values are in the ancillary reproducibility archive. The row audit reconstructs the aggregate shift exactly: −67/868 = −7.7189 pp. Family collapse accounts for −61 of the −67 unit margin change, or 91.0% in magnitude. Contracts can delete PageRank’s predicted legal hypothesis while turning Earliest’s empty prediction into baseline unit credit. An independent shard rebuild matches all 868 row-background records; complete transitions and corpus counts are in Appendix D.2. 5

Table 2: The preserved TraceElephant audit chain. Each stage has one evidential role; V1 did not produce scores, and V2’s target-effect stability must not be described as global ranking stability. Stage

Preserved result

Scientific role

V1 prediction-free gate

NO _ GO; required unchanged-family support 10, observed 0; scoring stopped Asymmetric terminal-NA effects −40.76 to −43.03 pp; all primary intervals exclude zero 8 weak orders, 12 strict-reversal pairs, Kendall τb down to −0.0501 Both symmetric treatments yield exactly 0 in every stratum/policy; primary intervals [0, 0]

Exposes ontology/gate incompatibility before outcomes Establishes a precise conditional effect in the restricted asymmetric family

Transparent V2 V2 reporting addendum Terminal-neutral control

Corrects the false global-rankingstability interpretation Tests the exposed asymmetry without result-based selection

5

C ONSTRUCTING AN AUDIT-A DMISSIBLE W ITNESS

5.1

T HE PREDICTION - FREE DIAGNOSTIC GATE (V1)

TraceElephant is a third execution/trajectory population with partial task overlap. Its V1 adapter study stopped before predictions, reference scoring, margins, crossings, or rankings. The structural gate required support from an unchanged-family stratum E=, but every terminal candidate was fixed to NOT _ APPLICABLE while the adopted-only filter retained only ADOPTED. Consequently, E= had zero task support against a frozen minimum of ten and was logically impossible under that ontology. This prediction-free NO _ GO is diagnostic evidence: it exposed the semantic incompatibility before outcome inspection. V1’s denominator discrepancy and two native-label anomalies remain preserved and unrepaired.

5.2

T HE CONDITIONAL ASYMMETRIC ESTIMATE (V2)

V2 transparently replaced the infeasible structural gate before any V2 prediction or score was inspected, while preserving V1 and its failure. Its declared populations are 138 tasks/176 traces task-disjoint, 34/41 exact-prompt, 35/42 task-ID sensitivity, and 173/218 projectable full descriptive. Under the original asymmetric terminal treatment, task-disjoint effects are −43.03 pp under baseline, −41.30 pp under conservative no-credit, and −40.76 pp under conditional localization. Their task-cluster 95% intervals are [−50.82, −35.14] pp, [−48.91, −33.70] pp, and [−48.61, −32.98] pp. These are precise conditional effects throughout Fasym . The original V2 report’s claim of “global stability” is invalid. Its version-preserving correction addendum reports eight weak orders, 12 strict-reversal pairs, and Kendall τb down to −0.0501 across 24 analyses. V2 supports stable direction and precision of the target adoption effect inside the restricted asymmetric family.

5.3

F OR THE TRANSPORT READING , THE AUDIT REQUIRES TWO CONTRAST- COMPARABLE CONTROLS

The prediction-free diagnostic gate (V1) and conditional asymmetric estimate (V2) isolate one asymmetry: terminal comparability changes on only the adopted-only side. There are two direct terminal-neutral, contrast-comparable controls. One retains the terminal candidate on both sides; the other excludes it on both sides. Both were jointly frozen, run, and reported while holding rows, predictions, methods, policies, and bootstrap design fixed. This is the bridge from the diagnostic event to the falsification control. 6

Adopted-minus-inclusive change in PageRank-minus-Earliest margin (percentage points)

A stable effect under one-sided terminal deletion vanishes under both contrast-comparable terminal treatments 0.00

0

0.00

0.00

−10 −20 −30

-43.03 pp

-40.76 pp

-41.30 pp

−40 Contrast-comparable: retain terminal on both sides Contrast-comparable: exclude terminal on both sides Original V2: one-sided terminal deletion

−50 Baseline empty/empty credit

Conservative empty no credit

Conditional nonempty localization

Figure 2: Task-disjoint PageRank-minus-Earliest adoption effects under the original one-sided terminal deletion and two contrast-comparable terminal treatments. Error bars are frozen 95% taskcluster percentile intervals; both contrast-comparable treatments are exactly zero with [0, 0] intervals.

6

L OCALIZING THE W ITNESS : FALSIFICATION , M ECHANISM , AND N ON -R ECOVERY

6.1

T WO SYMMETRIC CONTROLS FALSIFY AGGREGATE - EFFECT INVARIANCE

The terminal-neutral control evaluates both contrast-comparable treatments—symmetric retain-both and exclude-both—across all four strata and three policies with no result-based selection. Every PageRank-minus-Earliest adoption effect is exactly 0. For the 138-task task-disjoint stratum, every 10,000-draw task-cluster percentile interval is [0, 0], reflecting a structural identity of the frozen row contrast under each declared treatment. The contrast with Fasym exposes the omitted terminalsemantic axis. The direct task-disjoint contrasts between terminal-neutral and original V2 are +43.03 pp for baseline and +41.30 pp for conservative, with intervals [+35.14, +50.82] pp and [+33.70, +48.91] pp. For conditional localization, original V2’s −40.76 pp uses each cell’s own nonempty-family denominator, whereas the direct control-minus-V2 contrast first restricts both semantics to paired taskwise common support (138 tasks) and is +41.21 pp with interval [+32.98, +48.61] pp. Semantic-dependent eligibility makes these quantities non-algebraic negatives; their 0.46 pp difference is denominator/support standardization, not a residual effect under the symmetric treatments, whose target contrast remains exactly zero. 6.2

FACTORIAL DELETION LOCALIZES THIS WITNESS

A symmetric four-corner decomposition attributes 100% of the original V2 effect to deleting the terminal candidate on only one side. The nonterminal contribution is exactly zero in every background and policy on declared common support. The factorial audit therefore identifies the precise semantic operation that generates this disagreement witness. Exact-prompt differences of roughly −51 to −54 pp under the asymmetric convention likewise collapse to small values that generally include zero under terminal-neutral treatments. Adding the single nonexact prompt-drift task mapping does not change the conclusion; Appendix D.3 separates raw mapping support from native-label-projectable analysis support. 7

6.3

R EMOVING THIS CONTRAST DOES NOT RECOVER IDENTIFICATION

The two contrast-comparable symmetric treatments give T (b) = 0 for every declared background but disagree on absolute scores, rankings, reversal pairs, and finite empirical identified sets. Retaining versus excluding the terminal candidate changes relative performance even when the adoption contrast is structurally zero. Thus T is point identified inside Fcc while the remaining score and ranking conclusions are not. For the weak ranking R(s) to be identified over F, sign(b µm (s) − µ bm′ (s)), including the tie state, must be constant over s ∈ F for every method pair (m, m′ ). The scalar condition T (b) = 0 imposes none of these full pairwise constraints. Effect identification therefore does not imply score-level or ranking identification. The defensible central statement is: Identification is family-relative: IT (Fcc ) = {0}, yet |IT (Faudit )| > 1, and |IR (Fcc )| > 1. One-sided terminal deletion is the fully localized witness that separates these conclusions, not the definition of the general problem.

7

I DENTIFICATION R ESULT ACROSS THE T HREE FAMILIES

Established. Restricted-family precision does not license cross-semantic identification. Fasym supports a precise negative T , both jointly frozen contrast-comparable treatments give IT (Fcc ) = {0}, and their union gives |IT (Faudit )| > 1. One-sided terminal deletion generates the asymmetric effect, while rankings remain specification-dependent even across Fcc . Scope. The original V2 estimate remains valid for its joint specification. The audit does not select a universal terminal ontology or estimate the prevalence of semantic non-identification.

8

R EUSABLE O UTPUT: A N I DENTIFICATION AUDIT R ECORD

The reusable output is a family-indexed record with a compact, inspectable interface. Table 3: Minimal evaluation-semantic identification audit record. Audit-record field

Required content

Fixed substrate Semantic map Endpoint Admissibility Results Uncertainty Evidence roles

Artifacts, predictions, methods, and resampling design Legal references, eligibility, scoreability, denominators, and weights Q(s), declared family F, and inference license Outcome-independent rationale and freeze/access receipt Every jointly frozen alternative Specification variation and sampling variation reported separately Premise, diagnostic event, conditional result, correction, falsification, and nonrecovery

The record supports claims only relative to its declared family.

9

L IMITATIONS

Four limits define the evidential scope. First, one localized disagreement witness establishes a non-implication, while its prevalence and the behavior of unenumerated ontologies, partial-credit rules, task weights, and decision criteria remain open. Second, the 217 rows span two heterogeneous corpora rather than two replications of the complete reversal; TraceElephant has partial task overlap, with 34 exact-matched truth-projectable tasks and a 35-task task-ID sensitivity analysis. Third, V1 cannot identify its frozen E= stratum, and two native-label anomalies plus a census denominatorcheck defect remain disclosed. Fourth, bootstrap intervals characterize the frozen resampling design, while finite specification intervals characterize the declared semantic family. The results concern evaluator identification rather than causal correctness or downstream utility. 8

10

R ELATED W ORK AND P OSITIONING

Relationship to a shared evaluation substrate. Qin and Tong (2026) examine whether an observable trajectory determines the scored diagnostic object and whether heterogeneous operational realizations agree under a fixed contract. The present study instead asks whether effects and rankings remain invariant when fixed artifacts and predictions are evaluated under different audit-admissible semantics. We use the same frozen 217-trajectory candidate and prediction substrate while introducing specification-indexed effect and ranking analyses, empty-family controls, and a separate TraceElephant witness with two symmetric falsification controls. Thus, the two analyses differ in their primary endpoints, experimental contrasts, and inferential targets. A direct contemporary empirical neighbor: frame-level AUC. Abdulaziz and Bondarev (2026) hold fixed WSVAD frame scores and recompute AUC under global, per-category, and within-video pair supports on UCF-Crime. They report rank reversals, unresolved within-backbone pooled subpoint margins under a paired video bootstrap, and shared scene sensitivity across examined models. Their study directly shows that evaluator-controlled aggregation support can alter model comparisons without changing predictions, and it reinforces the need to keep readout variation distinct from sampling precision. The inferential targets differ. They ask whether pooled frame-level AUC measures within-video anomaly localization and resolves model gaps; we ask whether an artifactdefined effect or ranking is invariant over a declared semantic family. We add a finite identified-set criterion, monotonicity under literal family expansion, outcome-independent audit-admissibility conditions, and exact mechanism localization. Conversely, their multi-backbone, 16-model scenefactor audit covers a pairing-population coordinate that our diagnostic witness does not enumerate. The results are complementary; neither transfers the other’s empirical findings. Identification, multiverses, and measurement. Partial identification typically holds a target estimand fixed while indexing conclusions by assumptions (Tamer, 2010). Multiverse, specificationcurve, and many-analyst studies expose variation over defensible analytic choices and can include alternative operationalizations (Steegen et al., 2016; Simonsohn et al., 2020; Silberzahn et al., 2018). Measurement work distinguishes theoretical constructs from operationalizations and emphasizes construct validity (Jacobs & Wallach, 2021); benchmark critiques similarly ask whether broad scores measure their stated targets (Raji et al., 2021; Bean et al., 2025). We combine these concerns at one evaluator-controlled map: from a fixed trace to legal reference objects and scoreable emptiness. Our narrower addition is to declare that map as the audited variable, report endpoint-specific finite empirical identified sets alongside sampling uncertainty, and require outcome-independent semantic alternatives that are jointly frozen and all reported. Metrics, leaderboards, and benchmark design. Performance measures embed measurement choices and can support ill-advised evaluation practices without an explicit theory of what is measured (Flach, 2019). Leaderboards can also be poor proxies for practitioner utility (Ethayarajh & Jurafsky, 2020), while benchmark reliability depends on dataset design, annotation, scale, and bias rather than adversarial difficulty alone (Bowman & Dahl, 2021). Machine-learning comparisons further vary with pipeline randomness (Bouthillier et al., 2021), task selection (Dehghani et al., 2021), and underspecified predictors (D’Amour et al., 2022). Here the distinctive variable is the evaluator’s semantic map from a fixed trace to legal objects and scoreable emptiness. Abstention and structured agent evaluation. Selective prediction treats abstention through risk– coverage trade-offs (El-Yaniv & Wiener, 2010; Geifman & El-Yaniv, 2017); an evaluator-declared empty legal family is different from method abstention. HELM, CheckList, and Dynabench broaden metrics, tests, or evolving data (Liang et al., 2022; Ribeiro et al., 2020; Kiela et al., 2021), while AgentBench, τ -bench, and ToolSandbox expose interactive environments, final states, repeated trials, or trajectory milestones (Liu et al., 2024; Yao et al., 2024; Lu et al., 2024). We instead hold trajectories fixed and vary the semantic map from traces to legal objects and scoreable emptiness.

9

R EFERENCES Sara Abdulaziz and Egor Bondarev. Auditing frame-level AUC in weakly supervised video anomaly detection: Granularity, resolution, and scene bias, 2026. URL https://arxiv.org/abs/ 2608.11985v2. Version 2, accepted at the ECCV 2026 Empirical Theory Workshop. Andrew M. Bean et al. Measuring what matters: Construct validity in large language model benchmarks. In Advances in Neural Information Processing Systems 38, pp. 19868–19949, 2025. doi: 10.52202/085713-0590. Xavier Bouthillier et al. Accounting for variance in machine learning benchmarks, 2021. Samuel R. Bowman and George Dahl. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843–4855, 2021. doi: 10.18653/v1/2021.naacl-main.385. Alexander D’Amour et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226):1–61, 2022. URL https://jmlr.o rg/papers/v23/20-1335.html. Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery, 2021. Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53):1605–1641, 2010. URL https://jmlr.org/papers/ v11/el-yaniv10a.html. Kawin Ethayarajh and Dan Jurafsky. Utility is in the eye of the user: A critique of NLP leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 4846–4853, 2020. doi: 10.18653/v1/2020.emnlp-main.393. Peter Flach. Performance evaluation in machine learning: The good, the bad, the ugly, and the way forward. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1):9808–9814, 2019. doi: 10.1609/aaai.v33i01.33019808. Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks, 2017. Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 375–385, 2021. doi: 10.1145/3442 188.3445901. Douwe Kiela et al. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110–4124, 2021. doi: 10.18653/v1/2021.naacl-main.324. Percy Liang et al. Holistic evaluation of language models, 2022. Xiao Liu et al. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. Jiarui Lu et al. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities, 2024. Xi Qin and Jizhou Tong. Observability is not specification: Specification sensitivity and implementation stability in agent diagnosis. Zenodo, August 2026. URL https://doi.org/10.5281/ zenodo.21983110. Version v1, preprint. Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark, 2021. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902–4912, 2020. doi: 10.18653/v1/2020.acl -main.442. 10

Raphael Silberzahn et al. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 2018. doi: 10.1177/2515245917747646. Uri Simonsohn, Joseph P. Simmons, and Leif D. Nelson. Specification curve analysis. Nature Human Behaviour, 4:1208–1214, 2020. doi: 10.1038/s41562-020-0912-z. Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 2016. doi: 10.1177/174569 1616658637. Elie Tamer. Partial identification in econometrics. Annual Review of Economics, 2010. doi: 10.1146/annurev.economics.050708.143401. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024.

A

R EPRODUCIBILITY S TATEMENT

Every quantitative claim is bound to a frozen source through the ancillary reproducibility archive at anc/eval uation_specification_identification_reproduction_R2_20260818.zip. The archive reports expected and actual SHA-256 values for all 17 bound sources and exposes complete cell-level results, freeze/run receipts, independent rebuilds, and version-preserving corrections. Its three entry points are: • Manifest: MANIFEST.json. • Verifier: verify_ancillary.py. • Complete-results index and claim ledger: evidence_ledger.md.

B

E THICS AND T RANSPARENCY S TATEMENT

The main risk is interpretive: a benchmark convention could be mistaken for a method property. The audit record preserves failed gates, anomalies, post-run corrections, and symmetric falsification controls. Applicable follow-ups are described verbatim as post-hoc motivated, prospectively frozen robustness control.

C

AUDITED S ETTING AND F ROZEN M ETHOD R EGISTRY

The primary study holds 217 observed trajectories, their pre-existing candidate unions, and seven prediction rules fixed while the contract changes. A candidate contributes a legal event hypothesis only when it is localizable and every object in it satisfies the active contract predicates. Objects in the same conjunction group form one hypothesis; different groups form alternatives. The primary binary row endpoint is one when at least one frozen predicted hypothesis exactly matches a legal hypothesis. Under the baseline empty-family rule it is also one when both the legal and predicted families are empty; the later disposition policies change only how such cases enter the endpoint or denominator. The registry contains Mini plus six deterministic comparators: AgentDebugX core heuristic, task-aware PageRank, task-aware minimal hitting set, BM25 task-event top, earliest explicit anomaly, and terminal event only. The six comparators contribute 1,302 frozen prediction rows and made zero LLM calls; Mini contributes 217 frozen rows. Predictions are not rerun or adapted across contracts. The reader-facing descriptions in the main text follow the frozen registry and prediction reason codes: PAGERANK_MAXIMUM, EARLIEST_EXPLICIT_ANOMALY, and TERMINAL_EVENT; they document the provenance of the frozen predictions.

11

D

S UPPLEMENTARY Q UANTITATIVE D ETAILS

D.1

C ONTRACT- POLICY RESULTS

Table 4: Contract and empty-family policy alter both the PageRank-minus-Earliest margin range and the average adopted-minus-inclusive effect. Point ranges are finite eight-contract intervals. Parenthesized intervals are paired 95% linear-percentile intervals: the unchanged baseline carries forward the primary seed-20260816 bootstrap, while the two added-policy rows use the jointly resampled seed-20260817 control. Policy

Cell-margin interval

Average adoption effect (95% CI)

Cellwise crossing

Baseline empty/empty credit Conservative empty no credit Conditional nonempty localization

[−3.23, +8.76] pp [+20.74, +25.35] pp [+42.28, +52.00] pp

−7.72 pp [−12.10, −3.92] −3.46 pp [−5.99, −1.15] −0.89 pp [−4.96, +2.87]

Yes No No

D.2

ROW- SEMANTIC LOCALIZATION

The row audit reconstructs 868 row-background units: 217 rows under four frozen non-adoption backgrounds. Fifty-seven are transition drivers and 53 change the pair margin. Their shifts sum to −67 unit outcomes, exactly recomposing −67/868 = −7.7189 pp. PageRank changes from correct to incorrect 36 times, always after deletion of its legal hypothesis. Earliest changes from incorrect to correct 37 times, always when an empty prediction receives credit after family collapse. Fifty-one of 57 driver units collapse; they contribute −61 of the −67 margin shift, or 91.0% in magnitude. AgentRx contributes −7 over 292 row-background units and TELBench −60 over 576. An independent script rebuilds all 868 records from accepted shards and matches the primary outputs exactly.

D.3

T RACE E LEPHANT CENSUS AND SECONDARY STRATA

V1 contains 174 observable tasks/220 traces; the independent recount finds 173 projectable tasks/218 traces. The frozen census expected 172 projectable tasks under its own any-projectable-execution definition. Exact-prompt and task-ID denominators are 34 tasks/41 traces and 35 tasks/42 traces. Two native-label anomalies remain unrepaired. V2’s declared populations are 138 tasks/176 traces task-disjoint, 34/41 exact-prompt, 35/42 task-ID sensitivity, and 173/218 full descriptive. Under original asymmetric V2, exact-prompt TraceElephant-minus-existing217 differences are approximately −51.47 pp, −51.47 pp, and −54.23 pp across the three policies, with paired intervals excluding zero. Under terminal-neutral treatments, the corresponding 34-task effects are +3.68 pp, +1.47 pp, and 0, with intervals [0, +10.29] pp, approximately [0, +4.41] pp, and [−2.75, +3.61] pp. The task-ID sensitivity manifest adds one nonexact prompt-drift task mapping: raw mapped support rises from 35 tasks/43 traces to 36 tasks/44 traces, while fixed native-label exclusions leave 34 tasks/41 traces and 35 tasks/42 traces projectable. At 35-task support, the effects are +3.57 pp, +1.43 pp, and 0, without changing the qualitative conclusion.

D.4

R ANKING DETAILS UNDER SYMMETRIC TERMINAL TREATMENTS

For rankings, define the finite empirical winner set W(F) = {m : min rankm (s) = 1}, s∈F

(6)

the methods that attain competition rank 1 under at least one declared specification. This is a possible-winner set, not the identified set of complete ranking vectors IR (F) = {R(s) : s ∈ F}. For the task-disjoint population, NA _ KEEP _ BOTH gives PageRank-minus-Earliest finite intervals [43.84, 55.43] pp, three weak orders, one strict-reversal pair, and Kendall τb ∈ [0.5534, 1]. Its five-method winner set contains Mini, PageRank, Minimal Hitting Set, BM25, and Terminal Event Only. NA _ EXCLUDE _ BOTH gives policy-specific finite intervals from [−0.72, 13.41] pp to [2.54, 14.66] pp, five weak orders, six strictreversal pairs, Kendall τb ∈ [0.4286, 1], and a winner set containing only Mini. Even the latter singleton winner set does not identify the full ranking: five weak orders remain. Thus R(s) is not identified within either symmetric treatment or across Fcc .

D.5

S AMPLING DESIGN AND INTERPRETATION

The 217-row analyses use paired resampling that carries relevant cell comparisons together. TraceElephant uses 10,000 task-cluster draws with equal task weight; multiple eligible trajectories are averaged within task before

12

aggregation, and matched comparisons carry both TraceElephant and existing217 sides within each sampled task. A superpopulation interpretation requires treating the observed tasks as exchangeable draws from a target population for which the task-standardized estimator is relevant. Under that interpretation, the interval describes repeated sampling of tasks while keeping systems, trajectories-within-task, predictions, and semantic specification fixed. It does not cover new evaluator semantics, new systems, new source ecologies, or ontology uncertainty. If the observed tasks are instead treated as the fixed audit corpus, the same bootstrap output is a resampling-stability summary rather than a population-coverage guarantee. A specification interval is not a confidence interval, and a bootstrap sign-pattern frequency is not a posterior probability. The 24.1% original complete-pattern frequency describes the frozen resampling procedure rather than a population-ordering probability. Likewise, terminal-neutral [0, 0] intervals reflect a structurally invariant row contrast under the two declared symmetric treatments, not a universal zero over unenumerated semantics.

E

V ERSION T IMELINE

Stage

Version-preserving event

V1 gate V2

Prediction-free NO _ GO; scoring stopped and the failed gate was preserved Replacement design frozen before V2 outcomes; conditional asymmetric effect reported Original files unchanged; global-ranking statement replaced by the 24-analysis ranking audit Retain-both and exclude-both jointly frozen, both reported, and factorial deletion executed Report completed without rerunning scientific scoring or bootstrap arrays

V2 correction Symmetric control Reporting addendum

Full receipts, payload hashes, failure evidence, and the machine-readable claim ledger remain in the ancillary reproducibility archive.

13

Related documents

• Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics#1049008
• Who's Keeping Score? Interactive Steering of LLM-Powered Scoring with Attune#612387
• Who's Keeping Score? Interactive Steering of LLM-Powered Scoring with Attune#610910
• Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation#447985
• Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference#1141528