Selection-Aware Stress Testing for Interactive Agents

Selection-Aware Stress Testing for Interactive Agents

arXiv:2608.30916v1 [cs.LG] 31 Aug 2026

Yang Xu∗ Purdue University xu1720@purdue.edu Haixiang Sun Purdue University sun1321@purdue.edu

Chenang Li∗ University of California, Irvine chenangl@uci.edu Zhou Li University of California, Irvine zhou.li@uci.edu

Jiefu Zhang Purdue University zhan4018@purdue.edu Vaneet Aggarwal Purdue University vaneet@purdue.edu

Abstract Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selected from the same data. We introduce Selection-Aware Semantic Stress Testing (SASST), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks. The protocol checks support and stability, uses joint bounds for all planned claims, and can return no claim. We prove conditional asymptotic validity under stated cluster assumptions. A forty-cluster audit finds Gaussian undercoverage and conservative Bonferroni t bounds. In one 480-episode τ -bench study, a 3.75 point discovery gain vanished on confirmation. A second-model study likewise confirmed neither a workflow benefit nor a stable stress rule.

1

Introduction

Interactive-agent evaluations increasingly inform workflow selection [1, 13]. A team may compare workflows that interleave reasoning and acting, add reflection or search-based planning, or compile retrieved skills [11, 17, 22, 24], then select the strongest on a heterogeneous benchmark. However, that winner may lose its advantage on a coherent mixture of subset tasks. If confirmed, such a reversal could support reconsidering the workflow choice for that task regime. Teams may therefore search many task types for where the selected workflow loses its overall advantage. Because the subgroup is chosen after inspecting these same outcomes, an extreme reversal may reflect sampling noise rather than a pattern that persists on new tasks, making independent confirmation necessary before changing the deployment decision. In existing works, [20] protects workflow comparison, [4, 7, 18, 21] find coherent error groups, [2] protects fixed group families, and [14] confirms frozen descriptors for a fixed model. These methods do not by themselves protect the full process of choosing an average winner and then searching for task types where its advantage weakens or reverses. We propose Selection-Aware Semantic Stress Testing (SASST). It keeps repeated runs of each task together and divides tasks into two sets: a discovery set for choosing a reweighting rule and a confirmation set for evaluating that frozen rule on tasks not used to choose it. Discovery-set workflow scores are used to choose among candidate rules based only on pre-execution attributes, such as goal length, required tools, or permission flags. Before any results from the confirmation set are inspected, the chosen rule is fixed. It then assigns weights to those tasks, and the same workflows are compared again. SASST therefore protects two data-dependent choices: the workflow winner and the rule that most weakens its advantage. It rejects rules supported by too few tasks or unstable under small changes to the discovery data, and may return no claim. Contributions. SASST (i) asks whether a workflow that wins on average still wins when a coherent task type receives more emphasis, with all workflows evaluated on the same task mix; (ii) learns ∗ Equal contribution.

Preprint.

from discovery results a reusable rule based on task attributes known before execution, rejecting rules that rely on too few tasks or exceed a declared shift; (iii) freezes the rule and retests the same comparison on separate confirmation tasks, with uncertainty bounds for all planned claims; and (iv) returns confirmed, not confirmed, failed discovery, failed feasibility, or abstained, distinguishing supported, inconclusive, infeasible, and unstable findings. Theory, controlled studies, and agent studies evaluate the procedure.

2

Method

Tasks and the fixed comparison. Let W ∼ P denote one benchmark task before execution. It contains the goal, initial state, available tools, permissions, and other fixed task information. Scores produced by running workflows under different simulator seeds are observed outcomes, not being a part of W . All outcomes from the same task remain together as one task-level cluster. Before choosing a task-weighting rule, we fix the comparison plan denoted as A for both the discovery and the confirmation. Given the observed workflow scores and a set of task weights, A returns one performance estimate for each workflow. It also specifies how repeated scores are combined and, when applicable, how workflows are selected or weighted. Let θ(Q; A) be these estimates when Q provides the task weights, and let c specify the comparison target of interest. We define Tc (Q; A) = c⊤ θ(Q; A). For example, when c aims at comparing Plan + Verifier with ReAct [22], Tc (Q; A) is their difference in weighted average success under the same task weights. Throughout the setting, SASST aims at changing task emphasis, not the workflows or the comparison plan. The stress target defined next asks whether this same comparison weakens when coherent task types receive more weight. Stress target. A rule g uses only pre-execution task features to assign a bounded nonnegative score sg (W ). Larger scores give a task more emphasis. With P sg = EP [sg (W )], define ωg (W ) =

sg (W ) , P sg

dP g = ωg dP,

∆(g) = Tc (P g ; A).

(1)

Here P g is the benchmark distribution reweighted by g, and ∆(g) is the workflow contrast under that emphasis. It covers represented task types and allowed shifts, not absent tasks or arbitrary deployment traffic. We next explain how g is chosen on discovery data and evaluated independently. b D. Discovery and confirmation. On discovery clusters D, the equal-weight comparison gives ∆ The contribution ψbi approximates how changing task i’s weight changes the P final comparison, including any workflow-selection step inside A. With wi (g; D) = sg (Wi )/ ℓ∈D sg (Wℓ ), rank candidate rules by  X 1 loc b b ψbi . (2) ∆D (g) = ∆D + wi (g; D) − |D| i∈D

Lower values predict a weaker contrast. Discovery outcomes choose among candidate functions, but every candidate uses only pre-execution features. The selected function is frozen before confirmation. Equation (2) ranks rules only during discovery. Confirmation later recomputes the exact contrast. Density and effective sample size (ESS) checks prevent a few tasks from dominating. A radius limits the distance between the original and reweighted distributions of fixed task features. We use a finite-witness Kernel Max-Sliced Wasserstein (KMS) approximation [19], with maximum mean discrepancy (MMD) and linear max-sliced alternatives [8]. Resampling the discovery clusters checks rule stability. On confirmation clusters I, SASST applies the frozen rule, repeats the support checks, recomputes the exact paired comparison, and forms a joint uncertainty bound over all planned claims. Each planned claim fixes its contrast, direction, candidate rule family, allowed radius, thresholds, and inference rule. Algorithm 1 combines these choices, support checks, and confirmation into one decision. 2

Algorithm 1 SASST discovery and confirmation. Input: Fixed comparison plan A, independent task clusters, pre-execution features, planned claims J0 . 1: Split whole clusters into discovery D and confirmation I. Freeze workflows, candidate rule families, thresholds, and the inference rule. 2: On D, select for each claim the feasible rule minimizing Equation (2). If none is feasible, record failed discovery. 3: Freeze the rule and estimate its stability using only D. Apply it unchanged to I. If support fails, record failed feasibility. 4: For every support-feasible rule, compute the exact weighted comparison on I and its joint uncertainty bound. 5: Record confirmed or not confirmed when stability passes. Otherwise report the estimate diagnostically and record abstained. Output: A complete status and diagnostic report for every planned claim.

The algorithm above defines the operational report. The theorem below states when its confirmation bounds are jointly valid. Theorem 1 (Validity after discovery). Condition on the discovery data FD . For a finite predeclared family, suppose the confirmation cluster vectors Hk = (Hk,m : m ∈ J0 ) are conditionally indeb m = Fm (N −1 PN Hk,m ), µm = E(H1,m | FD ), and pendent and identically distributed. Let ∆ k=1 ∆m = Fm (µm ), where Hk,m contains every weighted numerator and denominator, every selector input, and both sides of the paired contrast. Assume bounded density ratios, denominators bounded away from zero, continuously differentiable Fm near µm , finite cluster second moments, a finitedimensional Lindeberg condition, conditional covariance converging to Σ, and positive limiting √ b m − ∆m : m ∈ J0 ) is asymptotically Gaussian, and variances for reported coordinates. Then N (∆ the cluster Gaussian multiplier consistently approximates its joint law. Pointwise intervals and signed simultaneous bands have asymptotic conditional coverage, including for a coordinate highlighted after inspecting the protected family. In plain language, once discovery fixes the rules, confirmation supports joint uncertainty statements for all planned claims. Appendix B gives the proof [3]. The theorem assumes identically distributed confirmation clusters, so the stratified designs in Studies F and G require an extension not supplied here. With hard workflow selection, the protected objects are the candidate contrasts, not the argmax. Section 3 audits the forty-cluster regime.

3

Validation

We test whether confirmation preserves coverage after rule discovery, what fails with forty clusters, and whether a supported signal remains detectable. For M ≥ 1000, pointwise coverage is 94.75– 95.30% and six-slot simultaneous coverage is 94.95–95.50%. At M = 1000, same-data worst-tercile coverage is 91.6%, versus 93.6% with discovery and confirmation separated (p = 0.032). At N = 40, an initial audit found 92.2–92.8% Gaussian coverage; a later fresh-seed diagnostic found 90.55–93.50% Gaussian and 96.40–97.55% Bonferroni-t coverage across four null settings. With 200 clusters and adequate stressed ESS, a planted-fragility control selects the target rule in 92.1% of trials, confirms it conditionally in 70.1%, detects it end-to-end in 64.6%, and has 1.7–1.8% null familywise error. Table 1: Inferential checks and the small-cluster safeguard. Setting

Result

Reading

Pointwise, M ≥ 1000 94.75–95.30% near 95% Six-slot simultaneous 94.95 / 95.50% near 95% Ten-slot Gaussian, N = 40 92.2–92.8% undercoverage Ten-slot Bonferroni t, N = 40 96.4–97.55% conservative Positive control, N = 200 64.6% end-to-end signal found

Additional audits test the rule-search safeguards: no geometry dominates, a declared radius can exclude a planted rule, support checks reject concentrated rules, and a detectable real-feature effect is reported as abstained when its description is unstable. 3

4

Agent studies

We apply the full protocol to two models. ReAct [22] is the baseline tool-use loop; Plan + Verifier adds planning and model-based verification; Confirmation Gate requests approval before declared irreversible actions. For each model, two slots test average success against ReAct, six test the same comparisons under three allowed task shifts, and two compare the enhanced workflows. Each study uses the same 80 τ -bench tasks [23], two simulator seeds, and three workflows (480 episodes). Task ID is the independent cluster, whole tasks are split 40/40 for discovery and confirmation, and rules below the predeclared 0.60 stability threshold are abstained. Table 2: Across-model results. Confirmation is the discovery-selected workflow minus ReAct; counts are confirmed / not confirmed / abstained / failed feasibility. Model

Discovery result

Qwen3-8B Plan+Verifier (+3.75 pp) Qwen2.5-7B Confirmation Gate selected

Confirmation Final counts 0.0 pp +2.5 pp

0/2/8/0 0/2/6/2

The Qwen3 discovery advantage disappeared on confirmation. Qwen2.5 had lower overall success, selected a different workflow, and produced different stress rules. Maximum rule stability was only 0.37 and 0.27, respectively, so neither model yielded a reusable stress description or confirmed workflow benefit. These studies do not establish equivalence or safety. Forty confirmation clusters and stressed ESS near 20 provide little power for effects of practical size, many tasks fail under every workflow, and the Qwen3 kernel bandwidth made most task embeddings nearly orthogonal. The second study reuses the same task pool, so it is a model replication rather than an independent benchmark replication.

5

Decision interpretation and scope Table 3: Decision status returned for every predeclared slot. Status

Meaning

Confirmed

Support and stability pass, and the joint uncertainty bound supports the planned claim. Not confirmed Checks pass, but the bound does not support the claim; this is not equivalence. Failed discovery No candidate rule satisfies the discovery constraints. Failed feasibility The selected task weighting lacks confirmation support or violates a predeclared check. Abstained The effect can be estimated, but the rule is too unstable to reuse.

The positive control shows that the protocol can confirm a planted fragility when the rule is eligible and confirmation has adequate support. The Qwen3 study shows the opposite case: a +3.75 pp discovery gain and an apparently weak segment could have motivated a verifier, but the average gain vanishes on confirmation and the segment description is unstable. The protected report supports neither claim and records what is missing rather than searching again on the same confirmation-task results. Reported evidence. For every planned claim, SASST reports the frozen rule, estimated comparison, uncertainty bound, support, stability, geometry diagnostics, and final status. These fields explain a missing claim: low support calls for more independent task clusters; instability calls for better features or rule geometry; and a non-confirmed bound calls for more power or a redesigned intervention. Limitations. The theorem requires conditionally i.i.d. confirmation clusters, bounded task weights, stable denominators, honest discovery, and smooth comparison functions. The Bonferroni t safeguard is supported by four audited null settings rather than a universal finite-sample theorem. Empirically, the studies use one τ -bench task pool, low-success open models, the same model as actor and user simulator, and forty confirmation tasks; the second model is not an independent benchmark replication. The next protected study needs fresh task clusters, score-blind pilot calibration of geometry and 4

radius, an outcome aligned with the intended policy or state transition, an independent checker where relevant, and a power plan based on stressed ESS; more seeds on the same tasks do not replace independent clusters. Operational scope. SASST supports decisions only for the benchmark and allowed task shifts; it is not a deployment-security certificate. It distinguishes supported, underpowered, out-of-scope, and unstable claims before a workflow or control is released.

5

References [1] Anthropic. Demystifying evals for ai agents. Anthropic Engineering, 2026. URL https:// www.anthropic.com/engineering/demystifying-evals-for-ai-agents. Published January 9, 2026. [2] John J Cherian and Emmanuel J Candès. Statistical inference for fairness auditing. Journal of machine learning research, 25(149):1–49, 2024. [3] Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. The Annals of Statistics, pages 2786–2819, 2013. [4] Greg d’Eon, Jason d’Eon, James R Wright, and Kevin Leyton-Brown. The spotlight: A general method for discovering systematic errors in deep learning models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1962–1981, 2022. [5] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11642–11662. PMLR, 2024. [6] John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378–1406, 2021. [7] Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher LeeMesser, Jared Dunnmon, James Zou, and Christopher Re. Domino: Discovering systematic errors with cross-modal embeddings. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=FPCMqjI0jXN. [8] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The journal of machine learning research, 13:723–773, 2012. [9] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pages 52989–53046, 2024. [10] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, 2025. [11] Xiangcheng Meng, Shu Wang, and Yixiang Fang. Skillrae: Agent skill-based context compilation for retrieval-augmented execution. arXiv preprint arXiv:2605.10114, 2026. [12] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In International Conference on Learning Representations, volume 2024, pages 9025–9049, 2024. [13] National Institute of Standards and Technology. Ai agent standards initiative. NIST, 2026. URL https://www.nist.gov/artificial-intelligence/ ai-agent-standards-initiative. Announced February 17, 2026; accessed August 2026. [14] Vyzantinos Repantis, Ameya Gawde, and Harshvardhan Singh. Decoy-calibrated failure audits for language models. arXiv preprint arXiv:2606.09046, 2026. [15] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryxGuJrFvS. [16] Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. Advances in Neural Information Processing Systems, 37:4540–4574, 2024. 6

[17] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. [18] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems, 33:19339–19352, 2020. [19] Jie Wang, March Boedihardjo, and Yao Xie. Statistical and computational guarantees of kernel max-sliced Wasserstein distances. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 62373– 62400. PMLR, 2025. [20] Yang Xu, Jiefu Zhang, Haixiang Sun, Zihan Zhou, Tianyu Cao, and Vaneet Aggarwal. Towards reliable llm evaluation: Correcting the winner’s curse in adaptive benchmarking. arXiv preprint arXiv:2605.05973, 2026. [21] Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look at subpopulation shift. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 39584–39622. PMLR, 2023. [22] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. [23] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In 13th International Conference on Learning Representations, ICLR 2025, 13th International Conference on Learning Representations, ICLR 2025, pages 74824–74876. International Conference on Learning Representations, ICLR, 2025. [24] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview. net/forum?id=njwv9BsGHF.

7

A

Related work and industry context

Agent benchmarks now test tool use, persistent state, web work, and general task completion. Examples include τ -bench, ToolSandbox, WorkArena, AgentBench, GAIA, and TaskBench [5, 9, 10, 12, 16, 23]. The benchmarks make workflow evaluation concrete, but they do not provide after discovery confidence bounds for a task mixture selected after workflow search. Several methods discover weak groups or protect group comparisons. GEORGE, Domino, Spotlight, and SubpopBench study hidden groups, systematic errors, and subpopulation shift [4, 7, 18, 21]. Fairness auditing gives simultaneous inference over rich group families [2], while Janus uses decoys and held out replication for a frozen descriptor list and a fixed model [14]. Distributionally robust optimization trains models for worst group or nearby distribution performance [6, 15]. SASST instead audits a frozen adaptive report. It discovers a common task reweighting, recomputes the complete paired report on independent clusters, and allows non-confirmation or abstention. KMS and MMD provide two choices of geometry for comparing feature distributions [8, 19]. The geometry limits the task mixtures considered by discovery. Coverage comes from the frozen rule cluster inference, which follows finite dimensional multiplier bootstrap arguments [3]. Industry guidance also treats agent evaluation as a release and governance process. Anthropic discusses the difficulty of evaluating agents that use tools and change state, Microsoft provides repeated evaluation for business agents, and NIST is developing agent security evaluations and standards [1, 13].

B

Proofs and technical details

B.1

Normalized weighted-ratio expansion

Lemma 1 (Weighted-ratio expansion). Let ag = P sg > 0, P g h = P (sg h)/ag , and Png h = Pn (sg h)/Pn sg . Whenever Pn sg > 0, ag Png h − P g h = (Pn − P ) {ωg (h − P g h)} , ωg = sg /ag . (3) Pn sg If P {ωg2 (h − P g h)2 } < ∞ and Pn sg →p ag , then √

1 n(Png h − P g h) = √

n X

n i=1

ωg (Wi ){h(Wi ) − P g h} + op (1).

(4)

The same statement holds coordinatewise for a fixed-dimensional vector h. Proof. Put the empirical and population tilted means over the common denominator Pn sg : Png h − P g h =

Pn {sg (h − P g h)} ag = Pn {ωg (h − P g h)}. Pn sg Pn sg

Because P {ωg (h − P g h)} = 0, this is exactly (3). The centered empirical average is Op (n−1/2 ) under the stated second-moment condition, while ag /(Pn sg ) = 1 + op (1). Slutsky’s theorem gives (4). The vector statement follows coordinatewise because the dimension is fixed. B.2

Cluster linearization and plug-in contributions

Lemma 2 (Weighted-report cluster linearization). Conditional on FD , suppose the conditions of Theorem 1 hold. Let µm = E(H1,m | FD ) and Φk,m = ∇Fm (µm )⊤ (Hk,m − µm ). Uniformly over the finite family J0 , N X √ b m − ∆m ) = N −1/2 N (∆ Φk,m + op (1), (5) k=1

and the canonical plug-in contributions satisfy max N

m∈J0

−1

N X

p

b k,m − Φk,m )2 − (Φ → 0.

k=1

8

(6)

Proof. Every population weighted scoring and held out denominator is at least a fixed b0 > 0. The cluster law of large numbers and finiteness of J0 imply that, with probability tending to one, all empirical denominators are at least b0 /2. Hence every Fm is evaluated inside a neighborhood where its gradient is bounded and uniformly continuous. Finite second moments and the cluster Lindeberg condition give µ bm − µm = Op (N −1/2 ). Taylor’s theorem with integral remainder yields Fm (b µm ) − Fm (µm ) = ∇Fm (µm )⊤ (b µm − µm ) + RN,m , |RN,m | ≤ ∥b µm − µm ∥ sup ∥∇Fm (µm + t(b µm − µm )) − ∇Fm (µm )∥. 0≤t≤1

The first factor is Op (N −1/2 ) and the second is op (1), uniformly over the finite family. The Taylor expansion proves (5). The coordinates of Hk,m include the weighted scoring and held out numerators and denominators, along with the selector and aggregation inputs. Differentiating q(S)⊤ T gives d{q(S)⊤ T } = q(S)⊤ dT + {Dq(S)dS}⊤ T, so Φk,m contains both held out evaluation and splitwise selection effects. Held out evaluation and splitwise selection are the two channels through which an evaluation unit changes an adaptive report. For the plug-in result, write b k,m − Φk,m = {∇Fm (b Φ µm ) − ∇Fm (µm )}⊤ (Hk,m − µm ) − ∇Fm (b µm )⊤ (b µm − µm ). The average squared first term is op (1) by gradient continuity, Cauchy Schwarz, and bounded empirical second moments. The second term is common across k and is op (1) because the gradient is bounded and µ bm − µm = op (1). Taking the maximum over the finite family proves (6). B.3

Proof of Theorem 1

Proof. Condition throughout on FD . The discovered rules and their random targets are then fixed. Let N X Φk = (Φk,m : m ∈ J0 )⊤ , SN = N −1/2 Φk . k=1

By Lemma 2, the centered estimator vector equals SN + op (1). For every fixed t ∈ R|J0 | , the P scalar triangular array N −1/2 k t⊤ Φk satisfies the Lindeberg Feller conditions and has variance converging to t⊤ Σt. The scalar CLT and Cramér–Wold device therefore give d

SN − → N|J0 | (0, Σ). Slutsky’s theorem proves the asserted joint limit for the estimator. It remains to establish the bootstrap approximation. Let b = N −1 Σ

N X

b Φ b k − Φ) b ⊤. b k − Φ)( (Φ

k=1

b k − Φk . Lemma 2 gives N −1 P ∥Dk ∥2 = op (1). Cauchy Schwarz then shows that Let Dk = Φ k P ⊤ every cross term N −1 k Dk Φ⊤ k is op (1), and the Dk Dk terms are also op (1). The cluster law of large numbers and covariance assumption imply that the sample covariance of the true Φk converges to Σ. Hence p b− Σ → Σ. Conditional on the inference data, the multiplier vector ∗

G =N

−1/2

N X

b k − Φ), b ζk ( Φ

k=1

9

iid

ζk ∼ N (0, 1),

b Continuity of centered Gaussian laws in their is exactly Gaussian with mean zero and covariance Σ. covariance matrices therefore yields conditional weak convergence to N (0, Σ) in bounded Lipschitz distance. Coordinatewise bootstrap quantiles give pointwise coverage. 2 For signed simultaneous claims, fix ηm ∈ {−1, +1} and let σm = Σmm . On the reported nondegen2 erate coordinates assume minm σm > 0. The map η m xm x 7−→ max m∈J0 σm is continuous, and its finite dimensional nondegenerate Gaussian distribution is continuous. Conditional bootstrap quantiles of the corresponding studentized maximum are therefore consistent. The resulting lower bands Bm for ηm ∆m satisfy

P(Bm ≤ ηm ∆m for all m ∈ J0 | FD ) ≥ 1 − α − op (1). On this event the inequality holds for every coordinate, so it also holds for any coordinate selected after inspecting the estimates and bands. The simultaneous event proves the pointwise, simultaneous, and selected report conclusions.

C

Expanded operational algorithm

The following details make Algorithm 1 fully executable. The algorithm is common to static LLM evaluations and agentic workflows; only the evaluation-unit representation, cluster definition, score tensor, and allowed feature map change. C.1

Frozen inputs and reporting slots

The inputs are: independent clusters C1 , . . . , CN ; the completed reporting layer A; a score blind feature map ϕ; and a finite family J0 . A slot m ∈ J0 fixes the contrast cm , one sided sign ηm ∈ {−1, +1}, feature/kernel protocol, candidate family Gm , KMS radius ρm , density cap κm , minimum effective sample size nmin,m , denominator thresholds, stability threshold, optimization tolerance, and deterministic tie rule. A candidate must be a reusable object g that maps any allowed feature vector to a nonnegative score sg (W ); a transductive vector of discovery weights is not an admissible output. Split whole clusters into D and I. All encoders, standardization statistics, random Fourier features, KMS witness directions, dictionaries, split schedules, and baseline systems are frozen according to the slot before confirmatory outcomes are read. C.2

Discovery on D

b D,c For each contrast needed by a slot, run the unweighted fixed reporting layer on D and obtain ∆ b and first-order contributions ψc,i . For every g ∈ Gm , compute ( )−1 X sg (Wi ) 2 dg = wi (g; D) = P , ESS wi (g; D) , κ bg = |D| max wi (g; D). i ℓ∈D sg (Wℓ ) i∈D

Let PbD be uniform on the discovery features and let PbDg use the weights above. With a frozen nonlinear feature map z and finite witness set BL , the implemented radius is   [ 1,L (PbD , Pbg ) = max W1 (β ⊤ z)# PbD , (β ⊤ z)# Pbg . KMS D D β∈BL

A candidate is discovery-feasible when its density, ESS, KMS, source/template, and other predeclared score blind constraints pass. Among feasible candidates, select " # X b D,c + gbm ∈ arg min ∆ wi (g; D) − |D|−1 ψbc ,i , g∈Gm

m

m

i∈D

up to the declared optimization tolerance and fixed tie rule. If no candidate is feasible, record failed discovery. Serialize every fitted threshold, basis, center, bandwidth, preprocessing statistic, and diagnostic needed to apply gbm to new units. 10

Discovery stability is evaluated by resampling discovery clusters, rerunning the same slot, and comparing the resulting weight functions on a frozen reference feature set. Discrete rules may use exact agreement; continuous rules use a predeclared functional metric such as weight correlation and top-weighted-set overlap. Stability never uses inference outcomes. C.3

Frozen-rule confirmation on I

Apply sgbm to inference features. Recompute the score-blind density ratio, ESS, source/template caps, and every scoring and held-out denominator. If any predeclared inference feasibility check fails, record failed feasibility and stop that slot. If support passes but the discovery description fails its predeclared stability criterion, set an abstention flag and continue so that the frozen-rule estimate and bound can be reported diagnostically. For every support-feasible slot, let Sr and Er be the fixed scoring and held-out subsets in repeated split r. For system j and retained artifact a, compute the exact ratios P P sg (Wi )Zj (Wi , a) i∈Sr sg (Wi )Zj (Wi , a) g g r b b P Sr,j,a = , Tr,j,a = i∈EP . i∈Sr sg (Wi ) i∈Er sg (Wi ) Apply the frozen selector and aggregation without modification: X b m = c⊤ θbg . qbg = πj (Sbg ), Yb g = (b q g )⊤ Tbg , θbg = λr Yb g , ∆ r,j

r,j

r,j

r,j

r,j

j

r,j

m

r

Both sides of a paired comparison are included in this same contrast. Construct the cluster tensor Hk,m from all weighted numerators and denominators, selector inputs, aggregation coordinates, and systems in the contrast, and compute b k,m = ∇Fm (b Φ µm )⊤ (Hk,m − µ bm ). The exact derivative contains both the held out evaluation term and the splitwise selection term displayed in Lemma 2. C.4

Inference rule and small cluster safeguard

The Gaussian multiplier maximum is the default asymptotic procedure. The inference rule must be frozen before confirmation outcomes are read. When a score-blind pilot or an independent simulation places the study in a small-cluster regime that has not passed the multiplier calibration check, the protocol may instead use an independently audited conservative family. Our audited safeguard is the one-sided Bonferroni t band σ bm Bonf b m − t1−α/|J |, N −1 √ Bm = ηm ∆ . 0 N Theorem 1 does not imply this finite-sample safeguard. It is an operating rule supported by the small-cluster simulations in Appendix D. The Gaussian analysis was the original Study F analysis; Bonferroni t was a later conservative sensitivity, and both give the same status for every Study F slot. Future small-cluster studies should freeze the safeguarded rule before confirmation outcomes are read. C.5

Simultaneous reporting and statuses

Use one multiplier ζk per inference cluster and share it across all slots. For predeclared signs ηm , b k,m . A fragility claim uses ηm = −1, apply the multiplier maximum to the signed contributions ηm Φ so a lower band for −∆m is an upper bound for ∆m . Report the complete attempted family, including nonreported slots: Status

Meaning

confirmed not confirmed

Feasibility and stability pass, and the predeclared signed band supports the stated claim. Feasibility and stability pass, but the signed band does not support the predeclared effect threshold. No candidate in the slot satisfies the discovery constraints. The frozen rule lacks inference support, violates a cap, or has an unstable scoring or held out denominator. The frozen rule effect may be estimable, but its semantic or workflow description fails the declared replicability or auditability criterion.

failed discovery failed feasibility abstained

11

The output row contains the frozen rule and description, ordinary and stressed contrasts, signed bound, ESS, maximum density ratio, minimum split denominator, stability, geometry diagnostics, runtime, and audit/control recommendation. Any external retrieval, generation, human curation, or mitigation experiment is a separate generalization study and is not part of the benchmark relative coverage certificate.

D

Corrected experiments and audits

Artifact policy. Only corrected and versioned artifacts from the independent code and artifact audit support the results below. Superseded first pass analyses are excluded from every paper table and claim. The original full paper plan reserved Study E for a larger static evaluation. The workshop paper does not claim a completed Study E.

D.1

Study A and A4: calibration after discovery

Study A uses eight Gaussian score blind features, five artifacts with different stress responses, a fixed paired baseline, nineteen bounded candidate rules, and a 50/50 discovery and confirmation split. Each configuration uses 2,000 trials and 2,000 multiplier draws. Table 4: Study A pointwise coverage in percent. Fixed means a fixed oracle rule, and learned means a rule chosen on discovery data. M

Fixed, R = 5

Learned, R = 5

Fixed, R = 20

Learned, R = 20

500 1000 2000 4000

95.00 94.75 95.10 95.30

93.85 95.10 95.30 95.15

94.60 95.10 94.95 95.75

94.05 94.75 95.05 95.10

Table 5: Matched same data comparison and Study A4 signed family results. Setting

Split procedure

Same data procedure

Protected family result

M = 500 M = 1000 M = 2000

94.4% coverage 93.6% coverage 94.4% coverage

91.3% coverage 91.6% coverage 93.3% coverage

not run 94.95% familywise, 0.85% false fragility 95.50% familywise, 0.55% false fragility

At M = 1000, the paired split minus same data coverage gap is 2.0 percentage points, with one sided lower bound 0.31 percentage points and McNemar p = 0.032. Fixed rule coverage ranges from 94.60% to 95.75%. For M ≥ 1000, learned rule coverage ranges from 94.75% to 95.30%, with negligible bias and estimated to empirical standard error ratios from 0.991 to 1.018.

D.2

Small cluster audit and safeguard

Study F has only N = 40 inference clusters, placing it in a regime where the Gaussian multiplier bootstrap may undercover. This subsection quantifies the gap and identifies a conservative alternative. The corrected small sample audit uses the exact Study F estimator and the complete ten slot family. Table 6: Small cluster familywise coverage across four audited null settings (FSV-remedy, post-hoc fresh-seed diagnostic). Values are ranges over the settings. Inference rule Gaussian multiplier maximum HC1 variance inflation Common t scaling Bonferroni one sided t

Familywise coverage

Reading

90.55 to 93.50% 91.95 to 93.85% 93.75 to 95.20% 96.40 to 97.55%

below the declared threshold not adequate not uniformly adequate conservative in all four settings

12

Table 7: Power for a negative ten point stressed effect under density-feasible designs (FSV-remedy, fresh seeds). Clusters

Stressed ESS

Gaussian power

Bonferroni t power

Reading

40 80 200 200

20 20 40 100

10.95% 9.70% 18.90% 48.10%

3.90% 4.35% 14.45% 40.25%

low support more clusters do not help at fixed ESS moderate support support improves power

Gaussian familywise coverage falls 1.5–4.5 points below the 95% target at N = 40. The nominal 200-cluster/ESS-20 design is excluded because its maximum density ratio is 10.26, exceeding κ = 5. The theorem is asymptotic, while the Bonferroni rule is a tested operating safeguard rather than a theorem consequence. D.3

Corrected positive control

The small cluster audit shows the method is conservative; this subsection shows it is not vacuous by demonstrating that SASST can confirm a real fragility when one exists at sufficient scale. The positive control is a same-seed diagnostic (not fresh validation) using frozen τ -bench features, the Study F estimator, independent confirmation clusters, a planted fragility in the claim direction, and Bonferroni t inference. The conditional confirmation rate is 0.701 with MC95 interval [0.671, 0.730], barely clearing its 0.700 gate. Table 8: Post-audit same-seed positive-control diagnostic.

D.4

Gate

Value

Required value

Result

Null familywise error, N = 40 Null familywise error, N = 200 Target rule discovery, N = 200 Conditional confirmation, N = 200 End to end detection, N = 200

0.017 0.018 0.921 0.701 0.646

at most 0.075 at most 0.075 at least 0.600 at least 0.700 at least 0.500

pass pass pass pass pass

Study B and the geometry audit

Study B compares KMS, MMD, linear max-sliced, no radius, and a restricted linear registry. A corrected Wasserstein routine changes 2.125% of KMS selections and 0.625% of max-sliced selections, but it does not change the qualitative result. Table 9: Corrected Study B held-out AUC after KMS replay (mean over 200 trials). Bold marks the best method per mechanism. Mechanism

KMS

MMD

Max-sliced

No-radius

Linear-only

Linear XOR Circular Policy by tool

0.602 0.639 0.695 0.698

0.631 0.747 0.716 0.679

0.623 0.610 0.696 0.707

0.597 0.649 0.558 0.700

0.594 0.631 0.500 0.576

MMD is strongest on three of four mechanisms; max-sliced wins on policy by tool. No geometry wins in every mechanism. Study B supports nonlinear rule recovery for some mechanisms; it does not show that KMS is universally best or provide a confirmed natural fragility result. A separate geometry audit uses frozen τ -bench features and a planted short goal mechanism. Table 10: Geometry audit under one frozen absolute radius budget (fixed registry axis, mean over six kernel seeds). Geometry

Accepted rules

Correct rule recovery

AUC

53.0 50.7 50.7 52.5 71

0.000 0.000 0.000 0.000 0.610

0.579 0.526 0.522 0.527 0.814

Standardized features Normalized plus structured features PCA plus structured features Structured features only All geometries without radius constraint

13

The target rule is outside the common 75% radius budget for every geometry. Median unit bandwidth RBF similarity in the standardized high dimensional space is about 4 × 10−177 . The radius and bandwidth must therefore be checked on score blind pilot data. D.5

Study C: feasibility and abstention

Study C verifies that the support gates (density, ESS, split denominators, stability) correctly block infeasible or unstable rules across five synthetic settings. Table 11: Study C operating checks over 500 trials per setting. Setting

Abstention or failure

Nonabstention

Recorded check

100.0% 100.0% 100.0% 95.8% 0.0%

0.0% 0.0% 0.0% 4.2% 100.0%

ESS and density density, while ESS passes split support, while global checks pass stability none

Severe concentration Density only Split support only Near tied rules Stable positive rule

The 4.2% near tie nonabstention rate is not a false fragility rate. Study A4 measures false fragility separately. D.6

Study D: partly synthetic real feature study

Study D uses 1,001 MMLU-Pro Law items, Qwen3-8B score tensors, 384 dimensional text features, and an injected degradation that flips 70% of correct answers within a 40-item jurisprudence subgroup. Table 12: Study D result and explanation diagnostics. Quantity

Value

Weighted confirmation contrast Pointwise 95% interval Unweighted confirmation contrast Stress amplification Recovery AUC Top weighted precision and Jaccard Top weighted recall Inference ESS Functional stability, 100 resamples Formal status

−0.0554 [−0.0886, −0.0221] −0.0294 −0.0260 0.974 0.444 1.000 405.7 0.18 abstained

The corrected radius is 0.003052. The frozen-rule effect is detectable, but functional stability is 0.18, so the formal status remains abstained and the description is not reusable. The high AUC is a ranking result and does not imply exact subgroup recovery. D.7

Controller mechanical audit

Before interpreting Study F and G workflow differences, the routing code must be verified: does ReAct pass everything through, does the verifier block on rejection, and does the confirmation gate respect consent tokens? A fixed deck sends sixteen state changing calls to the three workflows. Oracle verifier signals isolate controller mechanics from model judgment quality. Table 13: Controller replay. The test measures mechanics rather than semantic verifier quality. Workflow ReAct pass through Planner with oracle verifier Confirmation gate

Invalid block recall

Valid approval

Confirmed executed

Declined blocked

0.0% 100.0% 50.0%

100.0% 100.0% 50.0%

100.0% 50.0% 100.0%

0.0% 50.0% 100.0%

All twenty declared mechanical gates and ten boundary and regression tests pass. The prompt contains the exact serialized mutation and full arguments, and malformed verifier output fails closed. This audit applies to the revised f2_v2 controller; Studies F and G used the historical study_f_v1 controller, whose routing mechanics are consistent but whose confirmation prompt disclosed only the tool name. The audit does not validate informed consent or model based semantic verification. 14

D.8

Studies F and G: interactive agent evaluations

Study F uses Qwen3-8B. Study G uses Qwen2.5-7B-Instruct. Both use the same eighty task IDs, two simulator seeds, three workflows, 40/40 task cluster split, frozen features, three radii, and ten reporting slots. Each study contains 480 episodes. Table 14: Across-model agent summary. Study F full-study rates combine the exact 80-observation discovery and confirmation halves; Study G rates are reported over its full 160 observations per workflow. Quantity Full-study ReAct success Full-study Plan + verifier success Full-study confirmation-gate success Discovery choice Plan + verifier minus ReAct on confirmation Confirmation gate minus ReAct on confirmation Maximum rule stability Confirmed Not-confirmed Abstained Failed-feasibility

Study F, Qwen3-8B

Study G, Qwen2.5-7B-Instruct

18.75% 20.625% 19.375% Plan + verifier 0.000 −0.0250 0.37 0 2 8 0

11.9% 11.2% 14.4% Confirmation gate +0.0125 +0.0250 0.27 0 2 6 2

Table 15: Study F simultaneous ten-slot report under the original Gaussian procedure (critical value 2.4632). Bonferroni t (critical 2.7424) widens bands by factor 1.113 and leaves every status unchanged. Slot

Estimate

SE

Simultaneous bound

ESS

Density

Stability

Uniform: Plan + verifier − ReAct Uniform: confirmation gate − ReAct r1 = 0.01017: Plan + verifier r1 = 0.01017: confirmation gate r2 = 0.01799: Plan + verifier r2 = 0.01799: confirmation gate r3 = 0.04003: Plan + verifier r3 = 0.04003: confirmation gate Control r2 : Plan − gate Control r2 : gate − Plan

0.0000 −0.0250 −0.0227 −0.0227 +0.0400 0.0000 +0.1298 +0.0178 +0.0400 −0.0400

0.0791 0.0498 0.0782 0.0454 0.0836 0.0490 0.1398 0.0801 0.0574 0.0574

lower −0.195 lower −0.148 upper 0.170 upper 0.089 upper 0.246 upper 0.121 upper 0.474 upper 0.215 lower −0.101 lower −0.181

40.0 40.0 34.6 34.6 31.3 31.3 20.6 20.6 31.3 31.3

1.00 1.00 2.73 2.73 2.40 2.40 2.40 2.40 2.40 2.40

1.00 1.00 0.05 0.05 0.07 0.07 0.37 0.37 0.07 0.07

Status not confirmed not confirmed abstained abstained abstained abstained abstained abstained abstained abstained

Study F’s maximum density ratio is 2.73, well within κ = 5. Study G’s third radius fails feasibility because ESS = 18.7 falls below the predeclared minimum of 20. Table 16: Study G ten-slot report under the preregistered Gaussian simultaneous procedure (critical value 2.347). The Gaussian band is not well calibrated at N = 40 (Section D); Bonferroni t widens all intervals without changing any status. NA means unavailable after feasibility failure. Slot

Estimate

SE

Simultaneous bound

ESS

Density

Stability

Uniform: Plan + verifier − ReAct Uniform: confirmation gate − ReAct Stress r1 : Plan + verifier Stress r1 : confirmation gate Stress r2 : Plan + verifier Stress r2 : confirmation gate Stress r3 : Plan + verifier Stress r3 : confirmation gate Control r2 : Plan − gate Control r2 : gate − Plan

+0.0125 +0.0250 +0.0125 +0.0250 +0.0145 +0.0021 +0.0356 +0.0711 +0.0124 −0.0124

0.0414 0.0393 0.0414 0.0393 0.0435 0.0411 NA NA 0.0384 0.0384

lower −0.085 lower −0.067 upper 0.110 upper 0.117 upper 0.117 upper 0.099 NA NA lower −0.078 lower −0.103

40.0 40.0 40.0 40.0 34.8 34.8 18.7 18.7 34.8 34.8

1.00 1.00 1.00 1.00 1.16 1.16 2.85 2.85 1.16 1.16

1.00 1.00 0.10 0.10 0.27 0.27 0.14 0.14 0.27 0.27

Status not confirmed not confirmed abstained abstained abstained abstained failed feasibility failed feasibility abstained abstained

Study G uses the same tasks and is therefore an across model replication of the reporting decision rather than an independent benchmark replication. It selects a different workflow and different stress rules, but it also produces zero confirmed claims. Table 17: Study F descriptive execution counts. The counts are not causal safety effects. Workflow ReAct Plan + verifier Confirmation gate

Tokens

Tool calls

Failed calls

Irreversible calls

19,932,313 17,535,902 19,728,565

1,091 1,058 1,021

217 157 200

269 176 233

15

Study F used Qwen3-8B for 480 episodes, totaling 57,196,780 model tokens and 18,280 summed episode seconds. Study G used Qwen2.5-7B-Instruct for 480 episodes and required about 2.5 hours of collection at three concurrent workers plus 30 seconds of inference. The recorded execution environment used an NVIDIA RTX 6000 Ada Generation (49,140 MiB); a separate Study G environment manifest was not preserved. Study F sensitivity analyses answer different questions. Treating three parser errors as missing leaves the two confirmation contrasts and all statuses unchanged. A stronger balanced deletion lowers stressed ESS for two slots below the threshold, changing those slots from abstained to failed feasibility while keeping zero confirmed claims. Post-hoc descriptive decomposition. Across all task-seed blocks, 104/160 in Study F and 118/160 in Study G scored zero under all three workflows. These are observed all-workflow-zero blocks, not estimates of a structural floor. In Study G, Plan + verifier reached environment completion in 76/160 episodes versus 118/160 for ReAct. Trace termination codes show that the 84 Plan + verifier non-completions comprise 32 max-step terminations and 52 verifier-rejection terminations, so the completion difference primarily reflects controller-induced stopping. Conditional success among completed episodes was 23.7% for Plan + verifier and 16.1% for ReAct. Because completion is affected by workflow assignment, these completed sets are selected post-treatment populations; the difference should not be interpreted as a causal quality improvement. D.9

Reproducibility and superseded analyses

The paper tables use only corrected artifacts. The arithmetic audit verifies the discovery gain, confirmation contrasts, protected bounds, critical values, and status counts. The claim source matrix maps each paper claim to a corrected artifact and records allowed and forbidden wording. An exclusion audit checks that no paper script reads a superseded first pass result. Table 18: Artifact provenance summary. Analysis

Provenance

Study A, A4, B, C, D Study F, G KMS correction FSV-v2 FSV-remedy FGEO-v2 FPOS-v2 FCTL2

Preregistered simulation Preregistered interactive Corrective overlay (frozen Study F) Independent calibration audit Post-hoc fresh-seed diagnostic Corrective geometry audit Same-seed diagnostic Mechanical audit (f2_v2 controller)

Seeds dedicated dedicated same fresh fresh fresh same deterministic

A prospective natural follow up would require fresh tasks, a valid outcome aligned with the policy claim, a predeclared small sample inference rule, corrected feature geometry, a nonredundant intervention, and a power plan based on stressed effective sample size. We leave that experiment to later work.

E

Limitations and broader impact

The theorem is asymptotic and assumes bounded density ratios, stable denominators, honest discovery, and plausibly independent clusters. The forty-cluster audit shows that the generic Gaussian family can undercover in a small sample, while the independently audited Bonferroni t safeguard is conservative in the tested settings. The safeguard has been audited in four null settings rather than proved for every finite sample. The stress rule search is limited by the declared registry, feature map, radius, and support thresholds. A rule outside the radius is outside the audit scope. High-dimensional kernel features can also become nearly orthogonal when the bandwidth is poorly scaled. The agent studies use one benchmark task pool, low-success open models, the same model as actor and user simulator, and forty confirmation task clusters. The second-model study is a model replication on the same tasks rather than an independent benchmark replication. The intended use is evaluation and release control. A team can use the protocol to distinguish a supported workflow claim from a development-set gain, an unstable segment, or a low-support 16

analysis. Misuse would include treating a benchmark relative result as proof of deployment safety, treating non-confirmation as equivalence, or selecting a new analysis after seeing confirmation outcomes. The paper rejects those interpretations.

17

Related documents

• Selection-Aware Stress Testing for Interactive Agents#818950
• Selection-Aware Stress Testing for Interactive Agents#808958
• WEFT: Scaling Tool-Use Post-Training for General-Purpose Agents#1132088
• What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal#627749
• Efficient Test-Time Adaptation through Human-AI Interaction#927009