International Journal of Information Security manuscript No. (will be inserted by the editor)
Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection
arXiv:2609.04388v1 [cs.CR] 3 Sep 2026
Roberto Fernández-Barrios · Iker Pastor-López · Amaia Pikatza-Huerga · Pablo García Bringas
Received: date / Accepted: date
Abstract Adaptive network intrusion detection systems retrain classifiers after drift alarms, but an alarm detects change; it does not establish that a challenger should replace the deployed incumbent. Promotion is security-relevant because it changes the model responsible for subsequent attack detection, and evaluating it has a methodological problem: promotion conclusions may depend on how the challenger was constructed and on how much evidence supports it. We test that dependence on CICIDS2017, UNSW-NB15 and ToN-IoT with self-contained challenger pipelines, nested candidate-size controls, a common-harness comparison of nine update policies, and a final sensitivity confining every exact feature vector to one evaluation, training or probe role. Incumbent-owned frozen preprocessing amplified apparent promotion harm; with self-contained challenger pipelines the mean full-drift harm did not persist. Raising nominal candidate evidence from 512 to 2,000 samples per class improved promotion under pool-constructed progressive drift by +0.53, +1.67 and +0.38 balanced-accuracy points: positive and statistically resolved in all three benchmarks, but materially benchmark-dependent rather than homogeneous, and driven mainly by fewer false positives. Policy conclusions were partially robust: policy ordering changed with candidate comparability, no policy globally dominated, and earlier compatibility statements for a label-free estimator and a calibrated ensemble narrowed. Validation helped evidencedisadvantaged challengers but added no average benefit at R. Fernández-Barrios (corresponding author) · I. Pastor-López · A. Pikatza-Huerga · P. García Bringas Faculty of Engineering, University of Deusto, Avda. de las Universidades, 24, 48007 Bilbao, Spain E-mail: [email protected]; [email protected]; [email protected]; [email protected] ORCID: R. Fernández-Barrios 0009-0003-5312-2634 · I. Pastor-López 0000-0002-3068-6248 · A. Pikatza-Huerga 0009-0003-9080-6242 · P. García Bringas 0000-0003-3594-9534
parity. Thirteen replays on real, time-ordered traffic showed no net harm from always deploying. Challenger construction and evidence should be controlled, reported and interpreted explicitly when promotion is evaluated. Keywords distribution drift · candidate comparability · risk-aware model updating · adaptive model updating · machine learning · intrusion detection
1 Introduction Machine-learning network intrusion detection systems (NIDS) are trained on historical traffic [62], but the traffic they monitor evolves. New services, shifting user behaviour, reconfigured infrastructure and novel attacks move the input distribution away from the training distribution — concept drift [1, 72, 79]. A frozen classifier loses accuracy, so operational NIDS periodically retrain on recent traffic [1, 20, 22], and the dominant answer to when is drift detection: compare recent traffic with a reference window and retrain when a statistical monitor signals change [29, 56, 68]. A large literature therefore competes to build more sensitive monitors, from univariate change detectors to multivariate two-sample tests [33, 82] and quantum-kernel similarity measures [37, 44, 76], on the implicit premise that a better change detector yields a better retraining policy. This paper separates two questions that the detect-thenretrain loop merges. Drift detection answers has the distribution changed? Promotion asks is this particular challenger sufficiently supported to replace this incumbent? A drift alarm — or a schedule, or a false alarm — proposes a challenger; it does not establish that the challenger beats the incumbent. In a deployed NIDS that promotion is also a security-relevant integrity decision: it replaces the model responsible for all subsequent attack detection, so a poorly supported update can degrade the detector’s future security
2
behaviour even when the drift alarm itself was correct. Between the alarm and the promotion decision sits a layer the loop leaves implicit, and this paper makes it explicit: candidate comparability. Before a promotion outcome can be interpreted, let alone automated, the evaluation must establish that incumbent and challenger are comparable in how they were constructed and in how much evidence supports them. Prior work has framed retraining as a cost–benefit decision rather than an automatic response to detected change [57, 69, 90], and adaptation itself as a choice among mechanisms that includes not adapting [11]; we add the comparability audit that must precede any such decision, and we show experimentally that it can reverse the conclusions drawn without it. Scope of the net-harm findings. Several controlled configurations in this paper show always-deploy updating losing to never adapting. These net-harm observations are configuration-dependent: they were produced by specific candidate-construction and evidence conditions, and we do not claim them as universal properties of adaptive NIDS or of retraining. The study traces the mean harm to two asymmetries between incumbent and challenger — a frozen, incumbent-owned preprocessing representation, and a fourfold nominal training-evidence disadvantage — each isolated by a pre-specified control. Configurations without those asymmetries behave differently: robust learners (random forest, logistic regression, MLP) stay non-negative where the fragile SVC-RBF pipeline goes net-negative, and selfcontained, size-matched challengers show no mean deficit at zero drift and a larger benefit under drift. Every harm figure below should therefore be read as a statement about the configuration that produced it, and none of it was observed on real, chronologically ordered traffic: in thirteen chronological replays always-deploy updating never lost to never adapting (§5.8), which bounds neither the frequency nor the impossibility of such harm in deployment. The evidence has three layers. An initial controlled study exposed the phenomenon under a historical configuration. Pre-specified replications then isolated each comparability factor with fresh seeds, frozen margins and machineevaluable outcome rules, extended by a size-matched control under pool-constructed progressive drift and a commonharness comparison with published and reference baselines. A final pre-specified sensitivity repeated the two central blocks with exact-feature-group-disjoint role assignment, after an audit found identical cleaned feature vectors crossing the source-row-disjoint roles of the earlier design. Throughout, pre-specified and preregistered denote a protocol for an individual stage that was specified and version-controlled before that stage’s confirmatory seeds ran; they do not denote an external registry covering the sequential study as a whole. Building on active-testing and limited-label modelselection research [36, 46, 49, 64, 73], the paper makes three contributions.
R. Fernández-Barrios et al.
C1 (primary — candidate comparability before promotion): promotion conclusions and policy rankings are conditional on challenger construction and evidence. Adaptive updating is decomposed into drift trigger → candidate construction → evidence comparability → optional validation → promotion → future deployment outcome (Fig. 1). Each stage answers a distinct question: an alarm proposes; construction and evidence determine the conditions under which the proposal is compared; validation estimates the sign of the specific proposal; and promotion is a decision a healthy incumbent can win. Experimentally, the two upstream conditions changed not only whether promotion appeared harmful or beneficial but also which update policy appeared preferable: in the common-harness comparison the pre-specified ordering-change rule fired for label-free and ensemble alternatives when the challenger’s nominal evidence changed, and no evaluated policy dominated globally (§5.6, §5.7). Promotion conclusions should therefore be controlled, reported and interpreted explicitly with respect to challenger construction and evidence. C2 (secondary): controlled identification of two comparability asymmetries. Preprocessing ownership and nominal candidate-evidence size materially change promotion outcomes. The mean full-drift harm does not persist once each challenger owns its preprocessing (a role-randomized ownership experiment localizes the mechanism to feature scaling; Online Resource 1, §S2.12); the residual zero-drift mean deficit of 512-per-class challengers disappears at nominal 2,000-per-class parity; and under pool-constructed progressive drift raising nominal evidence retains a positive, statistically resolved effect in all three benchmarks under exact-feature-disjoint roles, although the material benefit is benchmark-dependent rather than homogeneous (§5.2–§5.7). C3 (secondary — conditional validation and contextdependent policies). Point/strict validation materially helps when the challenger’s construction or evidence is asymmetric and provides no detectable average benefit — with a measurable strict-gate cost in one cell — once challengers are self-contained and evidence-matched, at zero drift and under drift alike (§5.5). The exact-feature-disjoint policy sensitivity is partially robust: size-dependent ordering and the absence of a global winner survive, while the original ATC and calibrated-ensemble retention statements narrow (§5.6, §5.7). Supporting instruments and boundaries (tertiary — formal, diagnostic and operational instruments) complete the study: the sequential VBC-SG family is a secondary formal instrument for probe-level false-superiority control under a stated conditional null; a quantum-kernel monitor serves only as a detector-invariance check; thirteen chronological replays delimit external empirical support — none showed net harm from always deploying — without estimating deployment frequency; and an acquisition-yield simulation bounds attack-
Candidate Comparability Before Promotion
label discovery cost (§5.8). These are not universal policies or primary contributions. The paper proceeds: related work (§2), the decision pipeline and gates (§3), experimental design (§4), results (§5), implications and limits (§6–§8). 2 Related work Concept drift and adaptive intrusion detection. Concept drift degrades ML-based NIDS, and a body of work adapts models online or on a schedule to compensate [30, 55, 79]. Recent systems make that loop more operational. CARAVAN uses labeling agents and an accuracy proxy to trigger online retraining of in-network models, then installs the updated weights; its public artifact covers simulation and an FPGA/P4 testbed [87]. SPIDER performs semi-supervised continual learning with limited annotations and memory constraints [5], while NOCTOWL incrementally adjusts an interpretable tree under delayed, sampled labels [65]. SSF finetunes continually on a refreshed memory buffer, selecting the new samples that carry the drifted pattern and discarding outdated ones once a Kolmogorov–Smirnov test signals significant drift [89]. ADAWU-IDS is especially close in motivation: it calibrates response configurations on a heldout chronological validation stream and fixes the selected policy before testing; its selected CICIDS2017 configuration continuously reweights an ensemble and disables hierarchical retraining [86]. These systems contribute labeling, monitoring, continuous adaptation or stream-level response calibration; none performs the same per-proposal incumbent–challenger promotion comparison, so a direct transplant would change the decision unit and information interface rather than provide a decision-equivalent baseline (Online Resource 1, §S11). A 2026 journal-level addition to this line is HOIDS, a concept-drift-aware hybrid online intrusion detection system [24]. We study whether and when a concrete retrained candidate should replace its incumbent, connecting to, but distinct from, work on negative transfer and catastrophic forgetting [47, 58, 88]. Drift detection and two-sample monitors. Drift is commonly detected with change detectors [10, 17, 29, 67], statistical-test ensembles over batch streams [48], or multivariate two-sample tests — MMD, energy distance, Kolmogorov–Smirnov and Jensen–Shannon divergence — used as retraining or alarm triggers [23, 33, 53, 70]; how drift manifests (type, locality, severity) itself varies widely across streams [3]. Our classical baselines are drawn from this family. A complementary family of performance-aware detectors monitors the model’s error signal directly, presupposing continuous labeled feedback; Bayram et al. [14] formalize the distinction between distributional change and model degradation that our degradation–headroom analysis (Online Resource 1, §S1.1) quantifies empirically. Improving distribution-level monitors did not improve the promotion decision in any
3
regime we evaluated, because they are blind to model degradation; the reference DDM and ADWIN implementations are evaluated as retraining triggers in the common-harness comparison (§5.6). Quantum kernels for drift monitoring. Quantumkernel methods embed data via parameterized quantum feature maps [37, 40, 76], with exploratory use in cybersecurity [15, 32, 63]; a quantum-kernel MMD monitor is included here only as a detector-invariance check. Changing the detector family did not change any promotion outcome we evaluated, consistent with evidence that quantum-kernel advantage claims require careful classical baselining and face concentration limits [74, 83]. Drift, rejection and evaluation discipline in ML security. A parallel security literature asks when a deployed classifier should no longer be trusted. Transcend [43] and its revision Transcendent [13] use conformal evaluation to reject individual predictions whose low credibility signals model aging; CADE [85] detects and explains drifting samples via contrastive representations; INSOMNIA [6] updates a NIDS continuously with semi-supervision as drift accrues. These systems abstain on individual samples or adapt continuously; none evaluates whether a specific retrained candidate should replace the incumbent, and none can be instantiated in this decision problem without changing what it decides (§5.6). Sommer and Paxson [81] documented early the striking gap between extensive academic research on ML-based anomaly detection and its rare operational deployment, naming fundamental difficulties of sound evaluation among the causes; later work sharpened that concern through temporal, dataset and security-specific evaluation pitfalls: TESSERACT [66] and the “dos and don’ts” of Arp et al. [8] document how temporal and experimental bias inflate security-ML results; Apruzzese et al. [7] systematize the gap between benchmark performance and a pragmatic assessment of ML-based NIDS, and Jacobs et al. [41] show that benchmark-accurate networksecurity models can rest on shortcuts that do not survive a change of conditions. Closest to the update decision itself, Kan et al. [45] show a self-updating malware detector degrading as unverified pseudo-labels feed its own retraining, and Chen et al. [21] show that continuous retraining under drift depends on which samples are labeled and how; both concern the update rather than the alarm. Our pre-specified protocol with temporally ordered progressive windows and paired within-run baselines (§4) follows that evaluation discipline. When to retrain: cost-aware and validation-gated updates. Beyond detection, some work weighs the cost of adaptation or validates a candidate before deployment, e.g. in MLOps model-update pipelines and shadow/champion– challenger evaluation [35, 51, 61, 77]; that a well-intentioned update can be net-harmful is documented beyond security [12]. More generally, automated stream-learning frameworks separate prediction, made with the current model, from the
4
subsequent adaptation step and treat adaptation as a choice among multiple mechanisms, including no update; Bakirov et al. [11] select the mechanism by cross-validation on the most recent batch. Our decision unit is narrower: conditional on a NIDS update proposal, we compare a concrete challenger against the deployed incumbent while controlling how that challenger was constructed and what evidence it received. Champion–challenger evaluation, validation gates and labelefficient model comparison are therefore not new, and this paper claims none of them as novel; their reliance on a small labeled probe connects to active and label-efficient learning for data streams [19, 50, 54]. A complementary line estimates a model’s accuracy under distribution shift without target labels [9, 25, 31, 34, 42, 71]; we instantiate its two standard estimators (ATC, DoC) as promotion rules under the same harness as our gates (§5.6). Positioning and novelty. Table 1 positions the closest decision formulations by objective, decision unit, required information, output and guarantee. The rows are complementary rather than an empirical ranking: their problem definitions, information sets and outputs differ, and this paper reports no head-to-head superiority claim against them. What this study adds is not a new gate but an experimentally supported decomposition of adaptive promotion: adaptive-NIDS systems increasingly expose labeling, monitoring, updating and calibration interfaces [5, 65, 86, 87], whereas we separate promotion from the alarm and show, with pre-specified controls, that upstream challenger construction and evidence conditions change both the apparent value of promotion and the apparent relative value of update policies. Comparability of candidates is a familiar concern in model comparison generally [46, 73]; we claim only that its consequences for this per-proposal decision had not been isolated under the controls used here. Prior work distinguishes drift monitoring from adaptation or retraining decisions [11, 14, 69] and has used validation to calibrate adaptive responses [86], but, to our knowledge, it does not isolate challenger comparability as a factor governing per-proposal incumbent–challenger promotion in adaptive NIDS, and the value of validation as a per-proposal promotion gate under controlled challengercomparability regimes remains underexplored. 3 Candidate promotion as a decision pipeline 3.1 Promotion as a sequential decision We consider a deployed binary classifier h0 (benign vs. attack) monitoring a stream of traffic windows W1 ,W2 , . . . , each a set of flows with (at evaluation time) ground-truth labels; the deployed model degrades as the traffic distribution drifts. A drift monitor D produces a score st = D(Wt ) measuring divergence of Wt from a reference, and the standard adaptiveNIDS loop raises an alarm when st exceeds a threshold and
R. Fernández-Barrios et al.
retrains on recent data upon a confirmed alarm. This loop conflates two questions: – detection: has the distribution changed? — answered by D; – decision: will replacing h with a model retrained on the current window improve accuracy? — which D does not answer. Let h be the incumbent model at proposal time t and h′ the challenger: a model retrained on the current regime and proposed for promotion (we use challenger for the model itself and candidate for its role in the promotion decision, as in candidate construction and candidate evidence). For a deployment horizon H, define the candidate’s realized future value relative to retaining the incumbent as H Vt (h′ , h; H) = ∑ wk BA(h′ ,Wt+k ) − BA(h,Wt+k ) , k=1 H
wk ≥ 0,
∑ wk = 1. k=1
Promotion is beneficial over that horizon iff Vt (h′ , h; H) > 0 (or exceeds a deployment-cost equivalent expressed in the same units). This is a future, generally unobserved quantity at decision time. The labeled probe instead supplies a contemporaneous estimate of the sign of the candidate–incumbent difference and is evaluated here as a surrogate promotion rule; it is not an observation of Vt . Because D measures distributional change rather than the sign of this relative value, a detector-only policy commits every confirmed change, including changes for which retraining does not help. The pipeline of Fig. 1 separates the alarm, the proposal, the evidence used to train the candidate, the evidence used to validate it, and the evidence that determines its future deployment value — five objects the detection-centric loop treats as one. The gates of §3.4 instantiate the validation stage; the controls of §4.2 instantiate the construction and comparability stages.
3.2 Drift monitors We use two-sample drift monitors comparing a reference window (from the pre-drift regime) to the current window: Energy distance, RBF-MMD, per-feature Kolmogorov– Smirnov (max reduction) and histogram Jensen–Shannon divergence, plus classically simulated quantum-kernel MMD monitors with ZZ and Pauli-XZ feature maps used only for the detector-invariance check of §5.8 (Online Resource 1, §S1.2). All monitors expose a scalar score; thresholds are calibrated per detector on held-out calibration windows at a fixed quantile (0.95). The paper’s claims are detector-agnostic.
Candidate Comparability Before Promotion
5
Table 1 Conceptual positioning of related decision formulations. “Guarantee” records the type and scope stated by each formulation, not a common empirical ranking. Approach
Objective
Decision unit
Information required
CARA [57]
Cost-aware stream retraining Retraining opportu- Performance and nity ing/staleness costs
Output
Guarantees / scope
retrain- Retrain or keep
Cost–performance analysis under its assumptions; no concretechallenger audit Forecast when retraining Retraining opportu- Sparse performance history and Retrain or keep Uncertainty-aware forecast within Regol [69] pays nity cost ratio its model; no paired proposal guarantee Automated adapta- Select among stream adapta- Batch / adaptation op- Recent-batch (cross-validation) Selected adaptive Adaptive-strategy selection tion strategies [11] tion mechanisms portunity evidence mechanism, including within its framework; no explicit no update incumbent–challenger comparability audit System-specific decision architecIGPC-MSOS [80] Select NIDS update mode Drift/update episode Dual triggers and online system Update mode feedback ture; no paired concrete-challenger guarantee Label, monitor and retrain in- Labeling window Generated labels, predictions Retrain/update weights End-to-end adaptive system and CARAVAN [87] network models and accuracy proxy public artifact; no held-out challenger comparison ADAWU-IDS [86] Calibrate drift-response pol- Validation stream / Delayed chunk labels, drift Reweight; optional Chronological held-out calibration; icy traffic chunk severity and learner performance stronger response selected configuration is datasetspecific Active compari- Compare fixed predictors Query/evaluation Unlabeled pool and queried la- Estimate or ranking Method-specific sampleson/testing [49, 73] label-efficiently round bels efficiency/statistical guarantees Limited-label model Select among available mod- Selection Candidate predictions, labels Selected model Regret or selection guarantees under selection [36, 46, 64] els round/domain and/or history stated assumptions This work Audit comparability; condi- Drift-triggered pro- Incumbent/challenger pipelines, Commit, reject or defer Comparability evidence is central; tion promotion posal training evidence and a small VBC-SG secondarily controls false probe probe-superiority, not future accuracy
comparability layer made explicit in this paper
Drift signal
Candidate construction
Comparability audit
Optional validation
Commit / reject / defer
Deployment monitoring
proposes an update
own preprocessing? evidence size?
is the comparison informative?
fixed policies; no meta-controller
decision, not reflex
future value is decided here
Fig. 1 Candidate promotion as a decision pipeline. A drift alarm proposes, but does not justify, promotion; candidate construction and evidence comparability precede candidate validation, and the validation stage is optional — conditional on construction, evidence and incumbent health — rather than universal.
3.3 Pool-constructed progressive-drift readaptation protocol
Each stream has 100 post-drift windows. Window t is a classbalanced sample in which each class is a mixture of empirical reference and current regime pools with mixing weight sev(t) ("severity"), ramped linearly from 0 to 1 over the first 80 windows and held at 1 thereafter. We call this pool-constructed progressive drift: the source regimes are empirical, but the temporal trajectory is constructed. On a committed adaptation, the candidate is trained on a fresh balanced labeled sample from the current regime, and the detector reference is reset to the adapted regime. The trigger policy (alarm confirmation and cooldown) is held fixed across all gate conditions, so the decision gate is the only designed difference between arms. On the confirmatory harness every arm additionally processes bit-identical streams, so contrasts are exactly paired; only the initial exploratory study drew a separate pseudo-
random realization per arm (identical in distribution, not in realization; §4).
3.4 The validate-before-commit gate The gate is a decision step interposed between the trigger and the model swap: a deployment (commit) gate, not a retraining gate. On each confirmed trigger the loop retrains a candidate h′ as usual — consuming its training labels and compute regardless of the outcome — and the gate then decides only whether h′ replaces the incumbent h; otherwise we keep h (and keep the detector reference, so a later window can re-propose once an update becomes worthwhile). §5.8 additionally evaluates a two-stage variant that moves the same probe before candidate training, gating the training decision itself. We evaluate three commit gates:
6
R. Fernández-Barrios et al.
– always-deploy (none; written naive in estimand names and tables): always commit — the standard loop. – point gate (labeled_probe(b, ε)): draw a small balanced labeled probe P of b flows from the current window; commit iff BA(h′ , P) ≥ BA(h, P) + ε. The probe is the method’s incremental labeling cost; we report b ∈ {32, 64}, ε = 0. The strict gate is the same rule rejecting probe ties. – disagreement gate (unsup_disagree(τ)): commit iff the fraction of flows on which h and h′ disagree on the unlabeled current window is ≥ τ (a zero-label proxy for "adaptation would change decisions"); we report τ = 0.15. Two harness variants, which differ in exactly the information they may use, are stated in Fig. 2. Variant A is the core simulation gate: the environment supplies candidate batches and probes from its class pools at the current mixture severity, the source of the main tables but an oracle a deployment lacks. Variant B is the observed-data gate: the candidate’s training rows, the probe and the detector’s post-commit reference all come from windows the system has actually seen, and nothing reads sev(t); §5.8 reports B as a replication of A. The point gate estimates the contemporaneous sign of the candidate–incumbent difference from b labels, the surrogate for future value identified in §3.1; the disagreement gate tests whether that estimate can be obtained without labels. A cost-sensitive reading: two decisions, two thresholds. Let each error cost λe per flow, deploying an update λU , and training a candidate λC (compute plus training labels). At commit time the candidate already exists: λC and the probe are sunk, so over a horizon of N post-decision flows the myopic commit rule is c ′ , P) − BA(h, c P) ≥ ε ∗ , commit ⇐⇒ BA(h
ε∗ =
λU , λe N
— the margin ε is the deployment cost in accuracy points over the remaining horizon; λC does not belong in it, because rejecting the candidate cannot recover it. Training costs enter one decision earlier, whether to build a candidate at all, which the two-stage gate of §5.8 instantiates by spending the probe on the incumbent before training. A risk-averse operator replaces the point estimate with a lower confidence bound (the LCB variant of §5.8); raising ε from 0 to 0.01– 0.02 cuts committed updates by ∼30% at equal downstream gain (Online Resource 1, §S1.5), and asymmetric error costs substitute a weighted accuracy for BA in the same rule. 3.5 Risk-controlled commit gates and the scope of their guarantee This subsection states the risk-controlled alternatives to the point gate at the level needed to read the results; implemen-
tation details, the deferral continuation modes, the spending schedules and the stated properties are in Online Resource 1, §S2.13, and the proof of Proposition 1 in §S4. VBC-SG is a secondary decision-support instrument for settings in which validation is chosen; it is not the source of the paper’s main conclusion, and we claim neither that any member of the family dominates nor mathematical novelty for the construction. (A) Point and strict validation. The point gate (ε=0) commits on any observed probe advantage; it maximizes the probe point estimate, not future accuracy, and carries no formal error control: in the logged-trigger analysis of the historical configuration, 23% of its commits had negative realized value (Online Resource 1, §S2.10). The strict variant additionally rejects probe ties at no additional labels, the better buy at some operating points (§5.8). (B) Risk-controlled validation. Let the probe be b flows drawn from a distribution P, and for each flow let di = 1[h′ correct on i] − 1[h correct on i] ∈ {−1, 0, +1}. Three rules commit only on statistical evidence that EP [d] > 0: (i) exact McNemar [59], one-sided on the discordant pairs at level α (ties never commit); (ii) a Robbins normal-mixture lower confidence sequence (CS) [39]; (iii) an empiricalBernstein betting confidence sequence [84], much tighter because the di are mostly ties. The CS gates inspect the probe in 16-flow blocks up to the budget b and reject any proposal whose lower bound never clears 0; they run at α = 0.10, exact McNemar at its pre-specified α = 0.05 (McNemar at 0.10 leaves the zero-drift conclusion unchanged; §5.8). (C) Pooled versus stratified guarantees. The probe is drawn with fixed class quotas (b/2 per class), not i.i.d. from one distribution, so the pooled tests above match the sampling only approximately. Stratified variants lower-bound each class’s mean correctness difference at α/2 (Bonferroni over the two classes) and commit iff 12 (Lben + Latt ) > 0, which targets the balanced-accuracy difference the design estimates; each per-class bound inherits the guarantee of (E) under that class’s conditional null. Only the anytime-valid stratified version, in which each class’s bound is an empiricalBernstein confidence sequence, supports the anytime-valid claim; a fixed-sample per-class LCB variant has no sequential validity. (D) VBC-SG: commit, reject or defer. The ValidateBefore-Commit Sequential Gate carries every layer of control at once. Per-class empirical-Bernstein sequences drive a three-action decision: COMMIT when the balanced-accuracy lower bound clears zero, REJECT when the upper bound falls below zero (futility), and DEFER otherwise, retaining the candidate and continuing at later windows up to a deferral cap D. Because the stream can drift between windows, a deferred sequence must state what it estimates; three continuation modes do so (Cohort-sim, Refresh and Accumulate; Online
Candidate Comparability Before Promotion
7
(A) Gated update, core simulation harness (one stream) h <- h0 ; D <- fit_reference(pools, sev=0) ; alarms <- [] ; cooldown <- 0 for t = 1..T: W_t <- environment.stream[t] # pre-generated, balanced, labeled record BA(h, W_t) s_t <- D.score(W_t) ; alarms.append(s_t > threshold) trigger <- policy(alarms) and cooldown == 0 if trigger: h’ <- train(pools.train, sev(t)) # candidate: current mixture (oracle) P <- sample(pools.probe, sev(t)) # probe: balanced, b flows (oracle) commit <- [ BA(h’,P) >= BA(h,P) + eps ] # Sec 3.4 if commit: h <- h’ ; D <- fit_reference(pools, sev(t)) # deploy + reset alarms <- [] ; cooldown <- C # applied either way else: cooldown <- max(0, cooldown - 1) return {BA(h, W_t)}_t (B) Gated update, observed-data harness (no sev(t), no pools) ... identical control flow, but at a trigger at window t: h’ <- train(rows of the last 8 OBSERVED windows) # candidate P <- 32 rows of the OBSERVED window t-9, natural composition, excluding by row identity any row used to train h’ # probe commit <- [ acc(h’,P) >= acc(h,P) + eps ] if commit: h <- h’ ; D <- fit_reference(last 8 OBSERVED windows), threshold <- quantile_0.95(scores of preceding OBSERVED windows) Fig. 2 The gated update loop. (A) Core simulation harness: candidate batches and probes are drawn from the environment’s class pools at the current mixture severity (an oracle a deployment lacks; source of the main tables). (B) Observed-data harness: candidate, probe and detector recalibration come only from windows the system has already seen, and nothing reads the severity. The commit rule is that of §3.4.
Resource 1, §S2.13). Cohort-sim resamples the proposaltime target in the simulator and is not a retained production cohort, so that acquisition scheme is not claimed to be directly deployable. A deployment-long risk budget allocates each proposal’s level α j by a declared spending schedule (Bonferroni or p-series). (E) What VBC-SG guarantees. For a single proposal and under the probe distribution P, the confidence-sequence gates satisfy Pr ∃n ≤ b : LCBn > 0 EP [d] ≤ 0 ≤ α, uniformly over the data-dependent stopping time, when the conditional weak null of Proposition 1 holds along the adjudication sequence: they control the probability of falsely establishing positive expected paired correctness on the probe distribution. Under the spending schedule the union bound gives a deployment-long version (a false commit ever occurs with probability at most αlife ); without a schedule the guarantee is per proposal only, and labels per proposal are bounded by the per-window budget times (1 + D). Three things should be kept apart. The theorem is the anytime-valid statement below under its conditional weak null. The implementation applies the confidence sequence separately by class in the stratified gates. The limitation is that the probe is drawn with fixed finite class quotas (and, in the pool harness, a fixed reference/current mixture) before random ordering, and it is not proved here that a marginal non-superiority null EP [d] ≤ 0 under such quotas implies the sequential conditional null the
theorem needs; the applied guarantee is therefore stated only under that conditional null. Proposition 1 (conditional validity of the commit e-process). Map each per-flow correctness difference to xi = (di + 1)/2 ∈ [0, 1]; the empirical-Bernstein sequence implemented in the artifact is a wealth process Wn with predictable betting fractions and plug-in means, whose commit condition LCBn > 0 is algebraically the event Wn > 1/α. If E[di | Fi−1 ] ≤ 0 for every i — the weak null; no stationarity or identical distribution is assumed — then Wn is a nonnegative supermartingale with W0 = 1, and Ville’s inequality gives Pr(∃n : Wn > 1/α) ≤ α. The proof (Online Resource 1, §S4) is the standard testing-by-betting argument [39, 84]; we claim no novelty, only the correspondence to this implementation, which a null simulation with time-varying conditional means and variances exercises empirically (a check of the implementation under the conditional null, not a proof for the quota sampling design). (F) What VBC-SG does not guarantee. None of the above bounds the probability that a committed candidate lowers future balanced accuracy: the probe-level bound and a future-deployment bound coincide only when the probe is representative of the windows the candidate will face, and the stale-probe and dependence results of §5.8 show where that premise fails. We therefore claim a bound on false probesuperiority under the stated conditional null, never on future deployment regret or harm, and never a defense against a ma-
8
nipulated probe; §5.8 reports the empirical harmful-commit accounting and the label price of each layer. 4 Experimental design 4.1 Datasets and controlled streams Datasets and regimes. CICIDS2017 [78] (Tuesday reference → Wednesday, Friday-DDoS, Friday-PortScan, ThursdayWebAttacks), UNSW-NB15 [60] (DoS, Reconnaissance) and ToN-IoT [4] (Scanning), spanning benefit, marginal/mixed and harmful readaptation. Features are numeric, standardized and PCA-reduced to 8 dimensions; windows contain 128 flows. To rule out label leakage, identifier and label-derived columns are removed at staging (UNSW: id, label, attack_cat; ToN-IoT: label, type, timestamps and IP/uid identifiers; the CICIDS variant used ships no flow identifiers), and no remaining single feature separates the classes beyond AUC ≈ 0.81. CICIDS2017 has documented trafficgeneration and labeling artifacts [26, 52], and benchmark NIDS datasets more generally carry design smells such as heavy duplication and unclear labeling [28]; the concern is not specific to CICIDS2017, since class imbalance and data overlap are documented for UNSW-NB15 [91], and a recent cross-dataset review compiles published duplication, crosssplit contamination, overlap and label-inconsistency figures for CICIDS2017, UNSW-NB15 and ToN-IoT, obtained under heterogeneous protocols [2]. The exact-feature-disjoint sensitivity of §4.4 addresses exact duplicate-value exposure across evaluation, training and probe roles directly, under its own reproducible definition rather than these sources’ overlap measures; we therefore treat CICIDS2017 as the benefit end of a three-benchmark spectrum rather than as sole evidence — the paper’s central negative results (the marginal and harmful regimes) come from the independent UNSW-NB15 and ToN-IoT benchmarks. The exploratory stage spans all seven regimes; every registered confirmatory block (§4.2–§4.3, and the historical replication of §5.1) uses exactly one primary scenario per benchmark — CICIDS2017 PortScan, UNSWNB15 Reconnaissance and ToN-IoT Scanning — under full drift and, where stated, zero drift. The confirmatory harness. All confirmatory results come from a harness built to make the paired contrasts exact. (i) Common streams: every evaluation stream and calibration draw is pre-generated from the seed alone before any policy runs, so all arms process bit-identical windows and policy actions cannot perturb the environment. (ii) Source-row-disjoint partitions: each per-class pool is split once per seed (50/30/20) into window/train/probe partitions used respectively for evaluation windows, for initial models, candidates and detector references, and for probes; a source-row index cannot cross roles, but distinct source rows carrying an identical cleaned feature vector can, so these
R. Fernández-Barrios et al.
blocks are not exact-feature-value-disjoint. (iii) Separate generators: environment, initial training, detector calibration, candidate training, probe draws and post-commit recalibration each use their own RNG, seeded by (seed) or (seed, t) only. (iv) Per-trigger logging: at every trigger the harness records pre-trigger incumbent degradation, the detector score, and incumbent-vs-candidate BA over the following 1/3/5/10 windows, lookahead used for logging only. The exact-featuredisjoint sensitivity changes only item (ii). The replication protocol (criteria, margins, one-pass stopping rule) was committed and publicly tagged in version control before any confirmatory seed ran; its confirmatory window is 30 held-out seeds (104–133), untouched by any smoke test. Shared protocol constants. The initial model trains on 2,000 flows per class from the reference regime; in the historical configuration the scaler and PCA are fit on that training set only and frozen thereafter, and every candidate trains on a balanced sample of 512 flows per class drawn at the current mixture severity sev(t), never from the final regime pool: the candidate sees the population the deployed model faces at the trigger, a simulator convenience a production system lacks (§7; the observed-data and chronological arms are the counterpoints). The comparability controls of §4.2 vary exactly these two conventions. Detector references use 256 flows per class; thresholds are the 0.95 quantile of detector scores over 30 pre-drift calibration windows; the trigger is 3 consecutive alarms with a 10-window cooldown; windows contain 128 flows. The probe is the gate’s incremental label cost on top of the labeled retraining data any labeled-retraining loop already consumes: we claim label efficiency of the commit decision, not of the retraining pipeline (§5.8 accounts for both). Label and information latency. Several arms model the delay between a flow being observed and its label being available; Table 2 fixes the semantics, read directly off the released code. The probe knob (L p ) draws the probe from the stream as of window t − L p ; the candidate knob (Lc ) shifts the candidate’s whole training batch, features and labels together, to the mixture of window t − Lc (pool generator) or the eight observed windows ending at t − Lc (sliding-window generator). The observed-data probe is inherently nine windows old. Training completion is instantaneous except in the operational sensitivity (§5.8), which evaluates a five-window deployment delay while the incumbent keeps serving. While a deferred VBC-SG proposal is pending, the incumbent remains deployed and new proposals are blocked, exactly like a cooldown window, so a proposal is never silently replaced. The initial exploratory study. The hypotheses, regime taxonomy and robustness texture of Online Resource 1, §S1 come from an earlier harness whose arms drew seed-matched but unpaired stream realizations. Its numbers are exploratory context: every claim the paper rests on was re-established on the hardened harness with fresh seeds, and exploratory
Candidate Comparability Before Promotion
9
Table 2 Temporal (causal-information) availability of every object the update loop touches at a trigger at window t. “Latency knob” names the runner flag; 0 = contemporaneous. Object
Source
Reflects window
Latency knob
Trigger decision Probe (pool arms) Probe (observed arm) Candidate batch (pool) Candidate batch (sliding) Recalibration (observed) Training completion Deferred proposal
detector score on Wt pool sample at sev(t − L p ) observed window t − 9, row-disjoint pool sample at sev(t − Lc ) 8 observed windows ending t − Lc last 8 observed windows instant; delayed only in the operational arm pending state; blocks new proposals
t t − Lp t −9 t − Lc [t − Lc − 7, t − Lc ] ≤t t / t+5 until resolved
— –probe-latency fixed –candidate-latency –candidate-latency — –training-delay ∈ {0, 5} –defer-windows
rankings never govern where confirmatory evidence exists. A pre-specified confirmation/cooldown policy grid with no decision gate is a negative baseline (Online Resource 1, §S3). 4.2 Candidate construction and evidence controls We evaluate how preprocessing ownership and candidate evidence size alter promotion outcomes through staged, prespecified controls: each factor was isolated by a replication whose protocol, estimands, margins and interpretation rules were frozen before any of its confirmatory seeds ran. The controls were designed sequentially, each specified after the previous one’s outcome was known; we present them as the two levels of a construction factor and two drift regimes of an evidence factor rather than as a chronology. 4.2.1 Transformer ownership: self-contained candidate pipelines The historical configuration of this study’s harness fits scaler and PCA once, on the initial training set, and freezes them — every later challenger is trained and scored inside that initial representation. A role-randomized ownership control (Online Resource 1, §S2.12) shows this policy is not neutral for SVC-RBF — giving the incumbent only the feature standardizer reproduces essentially the whole ownership advantage — so the decisive question is dynamic: does the harmful-update result survive when each challenger is a selfcontained pipeline? A pre-specified replication answers it (protocol archived with the public artifact). Throughout, “historical” denotes this study’s own earlier harness: we do not claim that frozen incumbent-owned preprocessing is a standard or widespread policy in published adaptive NIDS; the lesson concerns asymmetric construction in evaluation, not the prevalence of the policy. Design. The environment emits a raw stream, hashverified bit-identical across arms per seed. Each model is a self-contained pipeline (scaler + PCA + classifier) that transforms internally; the drift monitor keeps the initial transformer under both policies, so construction policy is never confounded with monitoring policy. The policies differ in
exactly one intervention: frozen_initial_transformer reuses the initial scaler/PCA for every candidate (reproducing the historical harness bit-for-bit), while own_transformer_per_model fits each challenger’s scaler and PCA on its own candidate batch only, with the same dimension, solver and classifier hyperparameters; raw candidate batches are bit-identical across policies, the probe is served raw to both, and neither probe nor future windows enter any fit. A commit deploys the complete bundle from window t+1; scaler/PCA hashes and the SVC’s fitted γ are recorded per candidate. Estimands and inference. Fresh confirmatory seeds 3001–3030, six scenarios (PortScan, UNSW-Recon, ToN-IoT × full and zero drift), KS-max only, policies never-adapt/always-deploy/point/strict. The pre-specified estimands are ∆naive,own = BAnaive,own − BAnever (does harm persist?), ∆transformer = BAnaive,own − BAnaive,frozen (the ownership interaction, which isolates the ownership effect directly), and ∆point–naive,own , ∆strict–naive,own (does validation retain value?). Four frozen families (harm under own/full; transformer interaction/full; gate value under own/full; a 12-contrast zero-drift secondary family) are tested by the deterministic centered paired bootstrap used throughout, Holm-corrected within family; the inferential unit is the seed, and windows, triggers and commits are never treated as independent units. Margins were frozen in advance: ±0.5 BA points for equivalence, assessed by inclusion of the CI90 within the margin (the interval-inclusion form of the two one-sided tests [75]; ±0.2/ ± 1.0 sensitivities), and non-inferiority guardrails of −1.0 recall points and +0.5 FPR points (onesided 95% bounds) that restrict safety language; balanced accuracy alone determines the pre-specified verdict (Scenario A, B or C). The ±0.5-point margin equals the materiality threshold this study pre-specified for its own harm claims (effects below half a point are treated as operationally immaterial; the smallest harm called material, −0.65, exceeds it), so CI90 inclusion records compatibility of the mean effect with that margin. It is a study-level preregistered materiality and equivalence threshold, not a universal operational security threshold; every equivalence conclusion remains
10
margin-dependent and does not establish absence of an effect. 4.2.2 Candidate evidence size at zero drift: the size-matched control The replication above trains every challenger on 512 flows per class against a 2,000-per-class incumbent, so its residual zero-drift harm admits one mundane explanation: the challenger sees a quarter of the incumbent’s evidence. A prespecified control (protocol, config, seeds, margins, families and machine-evaluable outcome rules committed before any confirmatory seed ran) isolates that variable: does harmful promotion under zero drift persist when self-contained challengers are trained with the same per-class sample size as the incumbent? Design. Zero drift only, with the symmetric replication’s scenario definitions unchanged: random proposal trigger (p=0.05), severity fixed at zero, own_transformer_per_model exclusively. Per benchmark, seven arms — never-adapt, and {naive, point, strict} × candidate size ∈ {512, 2000}/class — give a 21-arm matrix on 30 fresh seeds (4001–4030). The only new variable is the challenger’s training size; trigger, probe (32 labels), gate margins, hyperparameters (SVC C=1.0, γ=scale, PCA dim 8), temporal semantics and sampling policy are unchanged, and the incumbent keeps its 2,000/class training set. One structural property should be stated in advance: at severity zero the 2,000-per-class self-contained challenger is drawn from the same pools, by the same sampler and with the same hyperparameters as the incumbent, so it is close to an exchangeable re-draw of the incumbent’s own training procedure. Near-zero mean effects at nominal parity are therefore the outcome expected of a correct implementation; the control’s informative content lies in the direct 512-versus-2,000 contrast, which is why the same intervention is repeated under pool-constructed progressive drift in §4.2.3, where the challenger is no longer an exchangeable copy of the incumbent. Nested candidate batches. The two size conditions are not independently sampled challengers: at every proposal the draw first produces the unchanged 512-per-class batch (bit-identical to the symmetric replication’s arms), then extends it with 1,488 further draws per class from the same per-trigger RNG stream, so the 512 batch is the first 512 rows per class of the 2,000 batch (999 proposal-level pairs verified). Pool draws are with replacement, so the extension adds nominal draws, not necessarily 1,488 distinct flows or an equal amount of effective information. Each challenger’s scaler/PCA/classifier still fit on its own batch only. This is a controlled pool-based construction, not an observed-data arm, and what it matches is nominal per-class sample size; §5.3 states what nominal parity does and does not equate.
R. Fernández-Barrios et al.
Estimands, families and outcome rules (all frozen). Primary estimands, in BA points per benchmark: E1 sizematched damage (naive2000 − never), E2 candidate-size effect (naive2000 − naive512 ), E3/E4 gate value at matched size, E5 (secondary) the gate×size interaction; four Holmcorrected families (F1–F4: 3/3/6/6 contrasts), CI95 for harm, CI90 for equivalence at the ±0.5-point margin, the same guardrails, the seed as inferential unit. Three machineevaluable outcomes were frozen in advance: PERSISTENCE (material Holm-significant damage at 2,000 plus a guardrailclean gate win), ELIMINATION (equivalence of all three damage contrasts and all six gate contrasts, plus no contradicting harmful-future-value signal, keyed to the sign rate of committed proposals’ five-window future value) and ATTENUATION (anything else). §5.3 applies these rules literally. 4.2.3 Candidate evidence size under pool-constructed progressive drift The zero-drift control cannot, by construction, show that candidate evidence matters where recency carries genuinely shifted information. A pre-specified control therefore repeats the nested size intervention under full progressive drift, the regime in which the frozen-transformer harness had reported that size-matching deepened harm (Online Resource 1, §S1.5). Its protocol, outcome rules and analysis script were committed before the drift-time nested draw was implemented, and the implementation was checked bit-for-bit against stored outputs before the confirmatory block ran. Design. Three full-drift scenarios (PortScan, UNSWRecon, ToN-IoT; mixing ramp 0→1 over 80 of 100 windows), the same seven arms, on 30 fresh seeds (6001–6030), own_transformer_per_model exclusively, every other constant identical to the symmetric replication. Nested draw at drift: at a proposal at window t the draw first produces the 512per-class batch at the proposal-time severity sev(t), then continues the same RNG stream to draw the 1,488-per-class extension at the same sev(t), so both sizes sample the same proposal-time mixture and differ only in nominal candidate evidence (prefix-hash and severity equality verified at every coupled proposal). Coupling scope: the two always-deploy arms share identical trigger and commit timelines, so their proposals are exactly coupled, whereas gated arms may diverge after the first discordant decision, so gate×size contrasts are seed-paired only. Estimands, families and outcome rules (all frozen). G1 value of updating (naive512 − never and naive2000 − never; 6 contrasts), G2 the primary size effect (naive2000 − naive512 ; 3), G3 gate value at 2,000 (6), G4 secondary gate×size interactions (6); Holm within family, the same margin and guardrails. The frozen classification of G2 per scenario is SIZE BENEFIT (Holm-significant, effect ≥ +0.5), SIZE COST (Holm-significant, ≤ −0.5), NO MATERIAL SIZE EFFECT
Candidate Comparability Before Promotion
(CI90 within ±0.5) or otherwise unresolved/sub-material, with a block label HOMOGENEOUS when all three scenarios share a class and HETEROGENEOUS otherwise. Future-value summaries are descriptive only. 4.3 Promotion policies and the common-harness comparison The primary policy matrix is common to every control: neveradapt (the frozen h0 ), always-deploy (naive in estimand names and tables: commit every confirmed proposal), and the point and strict labeled-probe gates of §3.4 (strict additionally rejects probe ties, at no additional labels). The formal sequential alternatives — exact McNemar, the confidencesequence gates and the assembled VBC-SG with its deferral and lifetime-spending machinery (§3.5) — are evaluated as a separate axis under the historical frozen policy, where their guarantees and label costs are priced (§5.8). The common-harness comparison. The commonharness block evaluates published and reference alternatives with self-contained challengers on bit-identical raw streams and source-row-disjoint roles; the exact-feature-disjoint repetition is reported in §5.7. Its protocol was frozen before implementation and, still before any result, placed the primary comparison at nominal 2,000-per-class evidence parity. Policies: never-adapt, always-deploy, point and strict (anchors); ATC [31] and DoC [34], the two standard published label-free accuracy estimators, used as promotion rules (each model estimates its current-window accuracy from its own confidences plus a 512-row labeled validation sample drawn at training time; zero target-window labels); a calibrated soft ensemble and replay 50/50 retraining (standard baselines); and the river reference implementations of DDM and ADWIN [17, 29] as retraining triggers with always-deploy on fire, run at their registered reference parameters — the implementations’ default DDM thresholds and ADWIN δ = 0.002 — on 8 monitoring labels per window (800 per stream); the monitors were not tuned, so their cells are read as evidence about the evaluated reference configuration, not about the methods in general. Selection criteria were frozen before any result: closeness to the same decision problem, public reproducibility, dataset compatibility, no information unavailable to competing methods, and checkable implementation fidelity. No faithfully reproducible end-to-end published adaptive-NIDS system was found that matched the same decision problem and information interface (§5.6); none of the evaluated rows is labelled an adaptive-NIDS state-of-the-art method. Six scenarios (three benchmarks × full and zero drift), 96 arms on 30 fresh seeds (5001–5030): the nine policies at 2,000/class plus a secondary 512/class sensitivity block for {naive, point, strict, ATC, DoC, ensemble}. Information budgets differ legitimately by method definition and are documented, not equalized (Online Resource 1, §S10).
11
Families and rules (frozen). Primary, at 2,000/class: PF1 zero-drift loss avoidance and PF2 full-drift benefit retention (each of {ATC, DoC, ensemble, replay, DDM, ADWIN} minus naive; 18 contrasts each), PF3 the published estimators minus the point gate (12). Secondary: SF4 the 512/class sensitivity (18) and SF5 method×size interactions (30; seedpaired). Holm within family; per-cell classification MATE RIAL GAIN / MATERIAL COST (Holm-significant, |∆ | ≥ 0.5), COMPATIBLE (CI90 within ±0.5) or UNRESOLVED ; anchorversus-anchor contrasts are descriptive here so that the hypotheses of §4.2.2–§4.2.3 are not double-tested. Four prespecified statements (S1–S4, Online Resource 1, §S10) are evaluated literally. 4.4 Exact-cleaned-feature-disjoint integrity sensitivity The historical confirmatory design separated roles by source row: the 50/30/20 splitter prevents reuse of a source-row index, but an audit of the cleaned data found exact cleaned raw feature vectors in more than one role. Exact identity was defined on the complete cleaned raw vector after the experiment’s numeric coercion and NaN/infinity handling, as canonical float64 bytes with signed zero normalized; no rounding, PCA, scaling or approximate matching entered the key, SHA-256 grouping was confirmed by vector equality, and the key used features only (labels, class, reference/current membership and row identity excluded). Across the three scenarios, 13.63%, 36.41% and 11.23% of rows were beyond the first member of an exact-X group; UNSW-NB15 contained 403 conflicting-label groups involving 1,828 rows. Every original row and label was retained. One final protocol, committed before implementation or results, replaced only role assignment: for each seed, all rows sharing exact X are assigned to one role by a deterministic seed-dependent constrained greedy algorithm that targets the pre-specified stratum fractions for reference/current × benign/attack, preserving multiplicity, contradictory labels and every source row while guaranteeing X(window) ∩ X(train) = X(window) ∩ X(probe) = X(train) ∩ X(probe) = ∅. Candidate sampling remains with replacement within its training role; the 512/class batch remains the prefix of the 2,000/class batch, and both use the same proposal-time mixture. All role audits passed, with zero cross-role exact-X overlap and maximum stratum-fraction deviation 0.0058 percentage points. The sensitivity repeated the 21 full-drift arms of the size control under drift (block B2 below) on fresh seeds 7001– 7030 and the 96 arms of the common-harness comparison (block B1) on fresh seeds 8001–8030. B2 froze magnitudeaware classes ROBUST HOMOGENEOUS SIZE BENEFIT, PAR TIAL ROBUSTNESS, NO MATERIAL SIZE EFFECT, SIZE COST and HETEROGENEOUS; B1 reused its original families and per-cell rules and froze POLICY CONCLUSIONS ROBUST,
12 PARTIALLY ROBUST or MATERIALLY CHANGED . Smoke seeds and confirmatory blocks were disjoint; grouping used features only and was fixed before any result was inspected, and no parameter changed after execution began. The exact protocol, configs, implementation tests, audit and complete results are in the artifact and Online Resource 1, §S11.
R. Fernández-Barrios et al.
and under pool-constructed progressive drift (§5.4), the conditional value of validation (§5.5), the common-harness comparison (§5.6), the exact-feature-disjoint sensitivity (§5.7), and mechanism, formal and boundary evidence, including the chronological replays (§5.8). Table 3 maps the sourcerow-disjoint evidence; Table 9 supplies the final sensitivity that governs the size-effect conclusion.
4.5 Outcomes and inference Per window we record balanced accuracy (BA) and attackclass F1; per stream we average over the 100 windows, and per condition over seeds. BA is the primary endpoint and measures the mean effect; attack recall and false-positive rate are pre-specified non-inferiority guardrails that gate safety language, not the verdicts; harmful-proposal summaries are descriptive within-trajectory fractions over seed-clustered commits; probe-level guarantees are not future-deployment guarantees. We report gain vs. no-adaptation (the frozen h0 baseline), committed adaptations and labels used; BApoint differences are not converted into misclassified-flow counts, which would require a prevalence model the balanced pool windows do not supply. The per-stream policy oracle BAoracle = Eseed max(BAno-adapt , BAdetector ) bounds the value of perfect per-trigger decisions from below, the regret of always-deploy is BAoracle −BAnaive , and the harm frequency is the fraction of streams on which adapting scored below noadaptation. All confirmatory comparisons are truly paired on bit-identical pre-generated streams, with cluster (per-seed) bootstrap 95% confidence intervals; the initial exploratory study uses seed-matched bootstrap CIs (conservative rather than fully paired). Generative-AI assistance was used in manuscript editing and in implementation and review of analysis scripts under author supervision; the tools, scope and humanaccountability statement are disclosed in the Declarations. Evidence stages. The study combines an exploratory discovery stage, the frozen-policy replication, two comparability controls (transformer ownership; zero-drift candidate size), the source-row-disjoint full-drift size control and commonharness comparison, their final exact-feature-disjoint sensitivity, supporting pre-specified analyses, and chronological boundary evaluations. The final claims use the exact-featuredisjoint result to qualify the earlier blocks; the remaining blocks test mechanisms, costs or boundaries. The full evidence hierarchy, per-block configuration, seeds and provenance are catalogued in Online Resource 1, §S0.
5 Results Results follow the pipeline of Fig. 1: the historical configuration that exposed the promotion problem (§5.1), candidate construction (§5.2), candidate evidence at zero drift (§5.3)
5.1 The historical frozen configuration: where the promotion problem appeared Under the study’s historical configuration — a frozen, incumbent-owned scaler and PCA reused by every challenger, and 512-per-class challengers against a 2,000-per-class incumbent — a replication on the hardened harness (seeds 104–133; full table in Online Resource 1, §S2.11) established the reference behaviour that motivated what follows: alwaysdeploy updating is net-harmful in the harm regime (−1.64 [−2.72, −0.64] balanced-accuracy points on ToN-IoT) and the 32-label commit gate converts it to a net benefit (+0.79; +2.43 above always-deploy) while preserving the benefit and marginal regimes, under both a classical and a quantumkernel detector. The same configuration makes always-deploy net-harmful in all three benchmarks once the incumbent stays healthy, under mild drift (−0.46, −0.15, −0.65) and under zero drift with random proposals (−2.76, −0.75, −4.75; Online Resource 1, §S2.12), and a statistical commit rule that refuses to commit without evidence recovers the zero-drift loss almost entirely. Absolute harm magnitudes in this configuration vary between seed blocks. The frozen configuration above was executed on three independent 30-seed blocks with identical data, flags, trigger semantics, candidate sampler and learner (the released code reproduces this block’s per-seed outputs bit-for-bit), and the absolute always-deploy harm on ToNIoT full drift differs materially between them: −1.64 [−2.72, −0.64] here (seeds 104–133), −1.97 [−4.00, −0.14] for the always-deploy anchor of the registered budget frontier (seeds 501–530; frontier_anchors.csv in the artifact) and −4.95 [−7.15, −2.86] in the frozen arm of the symmetric replication (seeds 3001–3030; Table 4); the zero-drift frozen cells show the same pattern with overlapping intervals (−2.76 here against −4.09 on PortScan in Table 3). The per-seed distribution is heavy-tailed: in the symmetric replication’s frozen arm the median stream loses 2.4 points while seven of thirty streams lose more than ten, because a committed challenger occasionally collapses, so a block’s mean depends on how many such streams it draws. Any single absolute frozen-harm magnitude is therefore quoted only as a descriptive block-level value; the comparative conclusions rest on paired within-block contrasts (the same streams under both policies), and the self-contained
Candidate Comparability Before Promotion
13
Table 3 Central evidence matrix: how the apparent value of promotion depends on challenger construction and evidence. Each cell reports always-deploy minus never-adapt in balanced-accuracy points (PortScan / UNSW-Recon / ToN-IoT) and whether point/strict validation adds Holm-significant average value over always-deploy in that configuration; every value names the 30-seed block it was estimated in, and where a cell’s validation summary comes from a different registered block than its ∆ , that block is named in the cell. Rows 1–2 are the historical frozen incumbent-owned preprocessing configuration (descriptive; the frozen arm of the symmetric replication, seeds 3001–3030, and, for the 2,000/class zero-drift cell, the registered frozen size-matched control on seeds 104–133). Rows 3–4 are self-contained challengers: zero-drift ∆ s from the registered size-matched control (seeds 4001–4030; 512 column descriptive, 2,000 column registered F1), full-drift ∆ s from the registered size-matched control under drift (seeds 6001–6030; registered G1). Absolute frozen-configuration magnitudes differ between seed blocks (§5.1); only within-block contrasts are paired. “not evaluated” cells were never run under a registered protocol. Configuration
512/class challenger
2,000/class challenger
Frozen incumbent-owned pre- ∆ vs never: +6.31 / +1.39 / −4.95 (seeds 3001– not evaluated under a registered protocol processing (historical), full 3030; descriptive) drift validation: point − naive +0.33 / −0.00 / +5.56 (same block; descriptive) Frozen incumbent-owned pre- ∆ vs never: −4.09 / −0.68 / −5.87 (seeds 3001– processing (historical), zero 3030; descriptive) drift validation: point − naive +2.93 / +0.21 / +5.47 (same block; descriptive)
∆ vs never: −4.81 / −0.13 / −5.76 (seeds 104–133; descriptive) validation: point − naive +4.46 / +0.03 / +5.35 (same block; descriptive)
Self-contained zero drift
challenger, ∆ vs never: −1.70 / −0.65 / −0.24 (seeds 4001– 4030; descriptive) validation: 6/6 gate contrasts Holm-significant gains (seeds 3001–3030, Table 4)
∆ vs never: +0.19 / −0.02 / −0.01 (seeds 4001– 4030; registered F1; CI90 within ±0.5 in 3/3) validation: 0/6 Holm-significant gains (same block; registered F3)
Self-contained full drift
challenger, ∆ vs never: +8.90 / +2.13 / +1.44 (seeds 6001– 6030; registered G1) validation: 1/6 Holm-significant gain, 1 resolved cost (seeds 3001–3030, Table 4)
∆ vs never: +9.72 / +3.79 / +2.44 (seeds 6001– 6030; registered G1) validation: 0/6 Holm-significant gains, 1 resolved cost (strict, UNSW −0.34; same block; registered G3)
size-matched and common-harness results of §5.3–§5.6 do not use these historical absolute values. Read on its own, this block says “promotion harms a healthy incumbent, and a gate rescues it”. The rest of the paper shows why that reading is a statement about the configuration: who owns the preprocessing, and how much evidence the challenger sees, drive the harm and most of the gate’s measured value. The frozen-policy numbers are retained as diagnostic evidence only (full grids: Online Resource 1, §S2.11–S2.12).
5.2 Candidate construction: preprocessing ownership changes the apparent value of promotion Table 4 reports the replication of §4.2.1; its pre-specified verdict was Scenario A — persistence with a shifted locus: preprocessing asymmetry amplified harmful promotion, while a residual zero-drift harm survived for self-contained challengers. All of its challengers train on 512 flows per class against a 2,000/class incumbent, the asymmetry §5.3 then isolates; the gate-value family is read in §5.5. (1) The mean full-drift harm does not persist under self-contained pipelines. With each challenger owning its preprocessing, always-deploy is beneficial in all three fulldrift regimes: PortScan +7.21 [4.89, 9.52], UNSW-Recon +2.55 [2.34, 2.75], and — the regime where the harm was
originally reported — ToN-IoT +1.03 [0.55, 1.53], all Holmsignificant. (2) Preprocessing ownership is a major amplifier — not a complete explanation of candidate-promotion risk. The ownership interaction BAnaive,own − BAnaive,frozen , on bitidentical raw streams and candidate batches, is positive everywhere it was harmful: +5.98 [3.82, 8.26] on ToN-IoT full, +0.90 and +1.15 on PortScan and UNSW full, and +5.49/+2.36 on the ToN/PortScan zero-drift streams. The frozen configuration had been handicapping every challenger, most severely where the harm was reported: the original full-drift harm was configuration-dependent and strongly amplified by incumbent-owned preprocessing. Freezing a representation can be a legitimate production policy; the issue is generalizing results obtained under that treatment beyond it. (3) But ownership does not explain all of the risk: zero-drift promotion harm persists at the historical candidate size. With no drift at all, self-contained 512-per-class challengers still hurt: naive−never is −1.74 [−2.35, −0.96] on PortScan and −0.65 [−0.77, −0.55] on UNSW-Recon — both beyond the preregistered ±0.5-point materiality margin, Holm-significant — and −0.38 [−0.53, −0.25] on ToN-IoT, sub-material but significant and not equivalent to zero (its CI90 is not inside ±0.5). These challengers were also trained on a quarter of the incumbent’s evidence, and §5.3 shows that asymmetry, not re-estimation per se, accounts for the mean effect.
14
R. Fernández-Barrios et al.
Table 4 The symmetric-pipeline dynamic replication (registered; seeds 3001–3030). Shared raw stream per seed (hash-verified across all seven arms); frozen = historical frozen-initial-transformer policy, own = self-contained pipelines (each challenger fits its own scaler/PCA on its own raw batch, bit-identical across policies; commits deploy the complete bundle). BA points, paired within seed, 30 seeds, deterministic paired-bootstrap CI95; † = Holm-significant within its registered family (own rows); frozen rows are descriptive (uncorrected). Zero drift: random trigger p=0.05, no drift; the detector representation stays the initial transformer under both policies.
Naive − never
Point − naive
Strict − naive
87.44 87.44
+6.31 [4.11, 8.53] +7.21 [4.89, 9.52]†
+0.33 [-0.05, 1.00] +0.19 [-0.06, 0.41]
+0.36 [-0.12, 1.07] +0.05 [-0.33, 0.41]
frozen own
83.03 83.03
+1.39 [1.17, 1.62] +2.55 [2.34, 2.75]†
−0.00 [-0.11, 0.10] −0.21 [-0.36, -0.06]†
−0.03 [-0.24, 0.16] −0.15 [-0.35, 0.05]
ToN-IoT full
frozen own
92.40 92.40
−4.95 [-7.15, -2.86] +1.03 [0.55, 1.53]†
+5.56 [3.50, 7.75] +0.64 [0.26, 1.06]†
+5.38 [3.34, 7.56] +0.18 [-0.31, 0.67]
PortScan zero
frozen own
95.44 95.44
−4.09 [-6.01, -2.56] −1.74 [-2.35, -0.96]†
+2.93 [1.34, 4.84] +0.75 [0.41, 1.16]†
+4.03 [2.49, 5.95] +1.68 [1.23, 2.13]†
UNSW-Recon zero
frozen own
89.09 89.09
−0.68 [-0.80, -0.57] −0.65 [-0.77, -0.55]†
+0.21 [0.12, 0.32] +0.19 [0.09, 0.31]†
+0.44 [0.31, 0.57] +0.51 [0.38, 0.64]†
ToN-IoT zero
frozen own
92.58 92.58
−5.87 [-9.20, -3.29] −0.38 [-0.53, -0.25]†
+5.47 [2.79, 8.86] +0.11 [0.03, 0.20]†
+5.87 [3.29, 9.21] +0.34 [0.20, 0.49]†
Regime
Transf.
Never BA
PortScan full
frozen own
UNSW-Recon full
Security guardrails and future-negative signs. Every BA-winning self-contained gate cell passes both noninferiority margins except one: on zero-drift UNSW-Recon, strict validation improves balanced accuracy and falsepositive rate, but its attack-recall lower bound crosses the 1-point margin (∆ recall −0.99, bound −1.26), and that cell is not described as a security improvement anywhere in this paper; the FPR point estimate improves in each of the other winning cells (up to −3.33 points for strict on zero-drift PortScan), while several non-winning full-drift gate cells fail the FPR margin (Online Resource 1, §S7). Under selfcontained always-deploy updating at 512/class, the descriptive fraction of committed proposals with future-negative five-window sign is 65%/61%/46% on the zero-drift streams (42% on ToN full); commits cluster within seeds and support no population-prevalence or deployment-probability inference. The quantum monitors, the VBC-SG frontier and the mild-drift matrix were not re-run under self-contained pipelines; their evidence remains scoped to the frozen policy (§7).
5.3 Candidate evidence at zero drift: nominal candidate-evidence size accounts for the residual zero-drift mean harm Table 5 reports the control of §4.2.2; the gate family (F3) is read in §5.5. (1) The 512 harm replicates; matched-size means are compatible with the margin. On thirty fresh seeds, 512per-class challengers reproduce the residual harm almost exactly (naive512 −never: −1.70 [−2.05, −1.36] PortScan, −0.65 [−0.77, −0.55] UNSW, −0.24 [−0.39, −0.10] ToN; descriptive). Give the same challengers the incumbent’s 2,000
flows per class, as a pure extension of the nested batch, and naive2000 −never is +0.19 [−0.15, +0.55], −0.02 [−0.10, +0.06] and −0.01 [−0.05, +0.04], all compatible with the ±0.5-point margin (CI90 inside it) and none approaching Holm significance. PortScan is margin-sensitive at ±0.2: its CI90 upper bound is 0.494, only 0.006 points inside the margin, so that cell is margin-dependent and does not demonstrate absence of an effect; UNSW-NB15 and ToN-IoT pass at ±0.2, and all three at ±1.0. (2) Candidate size is the driver, and it is Holm-significant everywhere. The paired size effect naive2000 −naive512 is +1.89 [+1.54, +2.23] on PortScan, +0.63 [+0.53, +0.73] on UNSW and +0.23 [+0.10, +0.38] on ToN — essentially the mirror image of the 512 harm, all three Holm-significant. Increasing candidate evidence made each mean effect compatible with the margin in the evaluated zero-drift control; it did not establish absence of an effect, nor equivalence in every respect a proposal can differ (Online Resource 1, §S8). What nominal sample-size parity does and does not match. The matched condition equates the challenger’s and the incumbent’s nominal per-class training size: 2,000 rows per class against 2,000 rows per class. It does not establish equality in effective sample size, temporal coverage, temporal diversity, subtype support, duplication, label quality, prevalence or information content. Pool draws are with replacement, so nominal rows can contain duplicates and should not be read as effective sample sizes; the candidate batch is a controlled balanced sample from the current-regime pool, not a time-ordered slice of observed traffic; and neither condition equates the temporal span the two training sets cover. Within these controlled conditions the experiment isolates the nominal row-count factor, and that factor accounts for the residual
Candidate Comparability Before Promotion
mean harm. Nominal parity is not information parity: evidence comparability in deployment remains an operational property to be audited, and §7 records the corresponding scope limits. Pre-specified outcome: ATTENUATION — a property of the rule, not evidence of residual harm. The frozen rules classify this result mechanically. PERSISTENCE fails because there is no material mean damage at the matched size (no F1 contrast is ≤ −0.5 with CI95 below zero). ELIMINATION fails on exactly one criterion: the harmful-future-value check (E3) keys on the sign rate of committed proposals’ five-window value, and at the matched size that rate is ≈48– 52% in two benchmarks, so the outcome is ATTENUATION. When the mean future value of committed challengers is approximately zero — which is what near-exchangeable challengers make expected (§4.2.2) — negative and positive signs occur at close to equal frequency, so a sign-rate threshold of this form is structurally unable to certify elimination in exactly the situation it was registered to detect. The label is preserved mechanically and should be read as a property of the rule, not as evidence of directional residual harm; the same logs show a small mean five-window future value of committed 2,000-challengers (−0.06 to +0.03 points, against −0.13 to −1.03 at 512), and a near-50% negative-sign rate must be read as proposal-level variability within trajectories, not a deployment-risk probability. The formal classification and the margin-based mean comparison are distinct; neither demonstrates absence of an effect, and the full-drift control of the next subsection does not retroactively change this classification. Guardrails at the matched size. All six matched-size gate cells pass both non-inferiority margins vs. naive2000 , while at 512 the zero-drift UNSW strict cell again fails recall non-inferiority, replicating the earlier trade-off on fresh seeds (Online Resource 1, §S8). There is no security trade-off at the matched size; there is also no BA gain, and the absence of a trade-off is not evidence of utility. 5.4 Candidate evidence under pool-constructed progressive drift: the source-row-disjoint result The zero-drift null was expected of any correct implementation, because a 2,000-per-class challenger drawn from the incumbent’s own pools at severity zero is close to an exchangeable copy of the incumbent. The full-drift control of §4.2.3 answers in the only regime where candidate evidence is a non-trivial hypothesis: the challenger now samples a drifted mixture the incumbent has never seen. Table 6 reports it (21 arms, seeds 6001–6030; nesting and severity audits verified at all 707 coupled proposals). (1) Source-row-disjoint primary result. In the sourcerow-disjoint block, increasing the candidate from 512 to 2,000 rows per class changes BA by +0.82 [0.67, 0.98] on
15
PortScan, +1.66 [1.51, 1.81] on UNSW-Recon and +1.00 [0.60, 1.39] on ToN-IoT — all Holm-significant and above the +0.5-point materiality margin. Its frozen classification is HOMOGENEOUS - SIZE BENEFIT. The measurements remain valid for that design, but the design is source-rowdisjoint rather than exact-feature-disjoint; the exact-featuredisjoint repetition of §5.7 (Table 9) is the governing estimate — +0.53, +1.67 and +0.38 — and narrows the conclusion. (2) What this establishes, and what it does not. Within this pool-based design, increasing nominal challenger evidence improves promotion under a constructed drift trajectory as well as under the zero-drift control, and the frozentransformer precedent in which size-matching deepened fulldrift harm does not reproduce with self-contained pipelines. It does not establish effective information parity, a universal monotonic benefit from more data, causal generality beyond the nested sampling design, or an effect free of exact-value exposure; the exact-feature-disjoint estimate in §5.7 supersedes the homogeneity claim. (3) Validation at the matched size under drift. Point and strict validation add no positive average value at 2,000/class in any of the six G3 contrasts (point +0.15, −0.06, +0.01 over always-deploy, none Holm-significant; strict −0.00, −0.34, −0.14, the UNSW-Recon cell a resolved cost, −0.34 [−0.53, −0.15]), and the secondary gate×size interactions are ≤ 0 in every point estimate. Every matchedsize point cell passes both guardrails; strict at 2,000 fails the FPR margin on UNSW-Recon (∆ FPR +0.86), so no safety language attaches to that cell, and at 512 strict fails the recall margin on PortScan and ToN-IoT. The only resolved gate gain of this block is the descriptive point gate on ToN-IoT at 512 (+0.46 [0.09, 0.88] over naive512 ), consistent with validation earning its keep where the challenger is evidencedisadvantaged (Online Resource 1, §S9). 5.5 Validation has conditional average value The gate families of the three comparability controls, read together, locate where validation earns its keep — and where it measurably does not. Under a nominal-evidence disadvantage, validation retains material value. At self-contained 512/class in the symmetric replication (seeds 3001–3030), point and strict validation improve over always-deploy in all six zero-drift comparisons (Holm-significant), recovering much or nearly all of the loss depending on the cell: strict gains +1.68 [1.23, 2.13] over always-deploy on PortScan, +0.51 [0.38, 0.64] on UNSW, +0.34 [0.20, 0.49] on ToN (point +0.75/ + 0.19/ + 0.11). “Recover” means improvement over always-deploy: strict returns to within 0.06–0.14 points of never-adapt, the point gate still sits 0.27–0.99 points below it, and no zerodrift cell converts to a net benefit over never adapting, nor should it where there is nothing to gain. At full drift the point
16
R. Fernández-Barrios et al.
Table 5 The size-matched self-contained challenger control (registered; seeds 4001–4030). Zero drift, random proposal trigger, own-transformer only; the sole new variable is the challenger’s per-class training size (512 vs. the incumbent’s 2,000), with nested candidate batches (§4.2.2). BA points, paired within seed, 30 fresh seeds, paired-bootstrap CI95; † = Holm-significant within its registered family; the 512-vs-never column is descriptive (uncorrected). compat.: CI90 fully inside the preregistered ±0.5-point margin; PortScan is boundary-close (upper bound 0.494, 0.006 inside) and does not demonstrate absence of an effect. Regime PortScan zero UNSW-Recon zero ToN-IoT zero
Never BA 95.66 89.08 92.56
Naive512 −never
Naive2000 −never
compat.
Naive2000 −naive512
Point2000 −naive2000
Strict2000 −naive2000
−1.70 [-2.05, -1.36] −0.65 [-0.77, -0.55] −0.24 [-0.39, -0.10]
+0.19 [-0.15, 0.55] −0.02 [-0.10, 0.06] −0.01 [-0.05, 0.04]
yes yes yes
+1.89 [1.54, 2.23]† +0.63 [0.53, 0.73]† +0.23 [0.10, 0.38]†
+0.05 [-0.00, 0.12] +0.01 [-0.02, 0.04] +0.02 [0.00, 0.04]
+0.13 [-0.07, 0.34] +0.06 [-0.01, 0.12] +0.03 [-0.01, 0.08]
Table 6 Candidate evidence under pool-constructed progressive drift: the registered source-row-disjoint size-matched control (seeds 6001–6030, 21 arms). Self-contained challengers, nested candidate batches drawn at the proposal-time mixture (the 512 batch is the first 512 rows per class of the 2,000 batch). BA points; paired within seed, 30 seeds; CI95 from the deterministic centered paired bootstrap; † = Holm-significant within its registered family (G1: naive vs never at both sizes; G2: the primary size effect; G3: gate value at 2,000). Historical registered classification of G2 per the frozen protocol; block outcome HOMOGENEOUS – SIZE BENEFIT. Columns 3–4: always-deploy minus never-adapt at each size; column 5: the size effect; columns 7–8: gate minus always-deploy at 2,000/class. Full matrices in Online Resource 1, §S9. Regime (full drift) Never BA Naive512 Naive2000 Size effect [CI95] PortScan UNSW-Recon ToN-IoT
85.58 83.36 91.79
+8.90† +2.13† +1.44†
+9.72†
G2 class
Point2000 [CI95]
Strict2000 [CI95]
+0.82 [0.67, 0.98]†
size benefit +0.15 [-0.03, 0.39] −0.00 [-0.17, 0.23] +3.79† +1.66 [1.51, 1.81]† size benefit −0.06 [-0.16, 0.03] −0.34 [-0.53, -0.15]† +2.44† +1.00 [0.60, 1.39]† size benefit +0.01 [-0.06, 0.09] −0.14 [-0.56, 0.28]
gate still filters harmful candidates on ToN-IoT (+0.64 [0.26, 1.06]), adds nothing material on PortScan, and has a small resolved cost on UNSW-Recon (−0.21 [−0.36, −0.06]): where updating is already clearly beneficial, validation’s marginal value can be zero or slightly negative (Online Resource 1, §S7). At nominal evidence parity, validation has nothing left to recover — at zero drift and under drift alike. All six F3 gate contrasts at the matched size under zero drift are tiny (+0.01 to +0.13 points), none Holm-significant and every one CI90-equivalent to always-deploy within ±0.5; the secondary F4 interactions are uniformly negative and Holmsignificant (strict on PortScan −1.54 [−1.89, −1.20]): the gates’ value at 512 was, to the extent the interaction measures it, compensation for the challenger’s evidence disadvantage (Online Resource 1, §S8). Under full drift the G3 family repeats the pattern with fresh seeds: 0/6 positive gate effects at 2,000/class and a resolved strict-gate cost on UNSW-Recon (§5.4). No average benefit under these controls is not the same as no value: it does not rule out value against individual harmful candidates, tail risk, temporally uncertain streams, or configurations not evaluated here. Evidence versus validation: what each nominal label buys. Table 7 compares, per benchmark, the 512-per-class policies against always-deploy at nominal 2,000-per-class parity, from the size-matched control (seeds 4001–4030), so its 512-side gate values re-estimate on fresh seeds the contrasts quoted above from seeds 3001–3030. In nominal adjudication counts, moving from 512 to 2,000 samples per class requires 2 × 1,488 = 2,976 additional candidate-training labels per proposal, whereas the point/strict decision uses a 32-label probe. At 512 the strict gate recovers most of the mean loss
(PortScan: naive512 −1.70 vs strict512 −0.03 against never, descriptive); at 2,000 the mean effects are compatible with the ±0.5-point margin without a probe. This establishes no economic dominance in either direction: labels are counted, not priced, and the inspected-flow cost of acquiring balanced attack labels is not modeled. Under the historical frozen policy the same conditionality appears from the other side: with a healthy incumbent and asymmetric challengers, point/strict and the risk-controlled rules recover most or all of the zero-drift loss (§5.8), while the chronological replays (§5.8) show what conservative validation forgoes when the incumbent has collapsed.
5.6 Source-row-disjoint common-harness comparison with published and reference baselines Table 8 reports the source-row-disjoint comparison of §4.3 at its primary condition: self-contained challengers at nominal 2,000-per-class evidence parity, 96 arms on seeds 5001–5030, every policy on bit-identical streams. Recent end-to-end adaptive NIDS have different decision units and label/update interfaces (§2; Online Resource 1, §S11), so transplanting them would not reproduce their published decision problem; cross-policy information budgets differ by definition and are reported, never equalized (Online Resource 1, §S10). The exact-feature-disjoint B1 sensitivity in §5.7 governs the final policy interpretation. Zero drift at parity: nothing left to rescue. With 2,000per-class challengers always-deploy is already approximately neutral relative to never-adapt (+0.25, +0.07, +0.02; descriptive), while the 512-per-class anchor replicates the residual harm on a third fresh seed block (−1.66, −0.58, −0.35).
Candidate Comparability Before Promotion
17
Table 7 Nominal evidence–validation trade-off under the preregistered zero-drift size-matched control (own-transformer, random proposals, balanced pools, SVC-RBF, seeds 4001–4030). “Cand.”/“Probe” are nominal adjudicated labels per proposal; moving from 512 to 2,000 per class adds 2 × 1,488 = 2,976 candidate-training labels per proposal versus the 32-label point/strict probe — counts, not costs: inspected-flow acquisition is not modeled and no economic dominance is claimed. Paired per-seed BA contrasts (CI95, deterministic centered paired bootstrap): naive-2,000 vs never is registered F1, gate gains at 2,000 are registered F3 (none Holm-significant), 512-side and gate-vs-never cells are descriptive, uncorrected. The supplementary point/strict-2,000 policies differ from naive-2,000 by < 0.14 pp (non-significant, CI90 compatible within ±0.5 pp). Guardrails: all matched-size gate cells pass recall/FPR non-inferiority; at 512 the UNSW strict cell fails recall non-inferiority (Online Resource 1, §S8).
Dataset
Policy
Cand. Probe BA vs never [CI95]
Gate gain vs naive [CI95] Commits/seed
CICIDS2017-PortScan
naive 512 point 512 + 32-label probe strict 512 + 32-label probe naive 2,000
1,024 1,024 1,024 4,000
0 32 32 0
−1.70 [-2.05, -1.36] −0.73 [-1.10, -0.38] −0.03 [-0.24, +0.16] +0.19 [-0.15, +0.55]
— +0.97 [+0.67, +1.31] +1.67 [+1.36, +2.00] —
3.70 1.87 0.23 3.70
UNSW-NB15
naive 512 point 512 + 32-label probe strict 512 + 32-label probe naive 2,000
1,024 1,024 1,024 4,000
0 32 32 0
−0.65 [-0.77, -0.55] −0.47 [-0.61, -0.33] −0.19 [-0.30, -0.10] −0.02 [-0.10, +0.06]
— +0.19 [+0.10, +0.30] +0.46 [+0.35, +0.58] —
3.70 2.27 0.67 3.70
ToN-IoT
naive 512 point 512 + 32-label probe strict 512 + 32-label probe naive 2,000
1,024 1,024 1,024 4,000
0 32 32 0
−0.24 [-0.39, -0.10] −0.12 [-0.24, -0.01] +0.00 [-0.08, +0.06] −0.01 [-0.05, +0.04]
— +0.12 [+0.04, +0.23] +0.24 [+0.12, +0.38] —
3.70 3.13 0.27 3.70
Table 8 Registered common-harness comparison with published and reference baselines (primary condition: self-contained 2,000/class challengers; seeds 5001–5030). Balanced-accuracy points over never-adapting (from the sealed per-arm means; the never-adapt row gives its absolute mean BA; all arms share bit-identical raw streams per seed). Markers record the registered per-cell classification of the policy−naive contrast (families PF1 zero drift / PF2 full drift; Holm within family): † material gain, ‡ material cost, ≈ compatible (CI90 within ±0.5), ? unresolved. Anchor rows (naive, point, strict) are descriptive here by amendment. Labels = target labels per decision; ATC/DoC additionally use a 512-row labeled validation sample at each model’s training time; DDM/ADWIN consume 8 monitoring labels per window. Origin: published generic method, reference implementation (river 0.25.0), standard baseline, or authors’ policy — none is an adaptive-NIDS system reproduced end to end. Full families, the 512/class sensitivity block and statements in Online Resource 1, §S10. Full drift Policy
Origin
Labels at decision
PortScan UNSW ToN-IoT PortScan UNSW ToN-IoT
Never-adapt anchor 0 Always-deploy (naive) anchor 0 Point gate, b=32 authors’ policy 32 Strict gate (reject ties), b=32 authors’ policy 32 ATC [31] published generic 0 (+512-row val.) DoC [34] published generic 0 (+512-row val.) Calibrated soft ensemble standard baseline 0 Replay 50/50 retraining standard baseline 0 river-DDM trigger reference impl. 0; 800/stream monitoring river-ADWIN trigger reference impl. 0; 800/stream monitoring
No policy attains the pre-specified statement S1 (“avoids the zero-drift loss”): every PF1 cell is COMPATIBLE or UNRESOLVED except the calibrated ensemble on PortScan (MATERIAL GAIN, +1.46); the zero-drift problem that labelfree alternatives were previously priced against disappears once the challenger is comparably evidenced. Full drift at parity: two compatible alternatives, four that pay as configured. ATC is COMPATIBLE with alwaysdeploy on UNSW-Recon (−0.22) and ToN-IoT (+0.10) and UNRESOLVED on PortScan (−0.65); the calibrated ensemble is COMPATIBLE in all three (−0.09, −0.42, −0.23). DoC pays a MATERIAL COST on PortScan (−2.59) and UNSWRecon (−0.59) and is COMPATIBLE on ToN-IoT (+0.21); replay (−0.89, −2.07, −1.22), river-DDM (−0.78, −2.67, −1.81) and river-ADWIN (−8.73, −4.06, −2.50) are MA -
Zero drift
86.0 83.1 91.9 +9.44 +4.06 +2.50 +9.53 +4.05 +2.70 +9.28 +3.83 +2.28 +8.80? +3.84≈ +2.60≈ ‡ ‡ +6.85 +3.47 +2.71≈ +9.35≈ +3.64≈ +2.26≈ +8.55‡ +1.99‡ +1.28‡ +8.66‡ +1.39‡ +0.69‡ +0.71‡ +0.00‡ +0.00‡
95.1 89.0 +0.25 +0.07 +0.49 +0.09 +0.64 +0.06 +0.52≈ +0.03≈ +0.64? +0.06≈ +1.71† +0.12≈ +0.40≈ +0.05≈ +0.40? +0.00≈ +0.00? +0.00≈
92.5 +0.02 +0.03 −0.01 +0.02≈ +0.05≈ +0.04≈ +0.01≈ −0.06≈ +0.00≈
TERIAL COST in every full-drift scenario, the two monitors while also consuming 800 monitoring labels per stream (ADWIN’s cost is largely under-triggering: at its registered reference parameters it barely fires, so its arm stays close to never-adapt; neither monitor was tuned, so these cells characterize the evaluated reference configuration, not the methods in general).
Statement S2 (“pays for it at full drift”) therefore holds for DoC, replay, DDM and ADWIN and for neither ATC nor the ensemble. ATC is the strongest published zero-targetlabel competitor; the calibrated ensemble is the strongest standard label-free baseline (it commits every trigger and cannot decline an update, and it fails recall non-inferiority on PortScan). Compatibility here is the pre-specified CI90within-±0.5 criterion, not a demonstration of equality.
18
Against the point gate. ATC is COMPATIBLE with the point gate on five of six scenarios and UNRESOLVED on PortScan full drift (−0.73), so statement S3 (“matches the point gate”) is narrowly not attained; DoC is a MATERIAL COST relative to the gate on PortScan (−2.67) and UNSWRecon (−0.58) full drift and COMPATIBLE elsewhere. The exploratory-harness observation that DoC beat the point gate in the harm regime (Online Resource 1, §S1.5) does not reproduce under the final harness at parity (ToN-IoT full drift: DoC−point +0.01, COMPATIBLE). Neither point nor strict validation dominates (descriptive full-drift differences to always-deploy at 2,000: +0.08/−0.00/+0.20 and −0.16/−0.23/−0.22, the UNSW strict cell resolved negative). Many full-drift cells of the label-free and monitor policies fail the FPR guardrail (ADWIN ∆ FPR up to +9.5 points; Online Resource 1, §S10), and no safety language attaches to them. The 512/class sensitivity: method ordering depends on the candidate generator. With evidence-disadvantaged 512-per-class challengers every label-free alternative gains at zero drift (the ensemble +2.93/+0.20/+0.36, DoC +1.93/+0.40/+0.44, ATC +1.65/+0.25/+0.26 over naive512 ), because naive512 is itself harmful there, and ATC/DoC surrender full-drift benefit on PortScan (−1.82, −3.63) while gaining on ToN-IoT (+0.48, +0.78). All fifteen Holm-significant method×size interactions (SF5) are negative, and the pre-specified ordering-change rule S4 fires for ATC (PortScan zero drift: MATERIAL GAIN at 512, COMPATIBLE at 2,000) and for the calibrated ensemble (PortScan and ToN-IoT full drift), not for point, strict or DoC. The value of every evaluated safeguard, labeled gates and label-free alternatives alike, concentrates where candidate evidence is asymmetric. This is the paper’s central methodological point in its sharpest form: even conclusions about which update policy appears preferable depend on the comparability of the candidate generator, so a policy ranking obtained with evidence-disadvantaged challengers need not survive at parity. The historical policy blocks — the frozen-policy comparison, the frozen zero-drift control, the exploratory-harness estimates and the lifetime-budgeted frontier — are retained in Online Resource 1, §S2.12 with their provenance and evidence tiers; where the same method and question are evaluated here, the common-harness evidence supersedes them, and cross-block comparisons remain descriptive. The comparison is a map of evaluated accuracy/label/update trade-offs, not a state-of-the-art ranking. 5.7 Final exact-feature-disjoint sensitivity: positive size effect, partial policy robustness The audit verified the exposure rather than assuming it (§4.4): under the source-row splitter, exact-X groups span window,
R. Fernández-Barrios et al.
train and probe roles, and candidate/future exposure differs with candidate size (Online Resource 1, §S11). These diagnostics establish exposure, not its causal effect. B2: the direction survives, homogeneity does not. Under exact-feature-disjoint roles, the paired size effect remains positive and resolved in all three benchmarks: +0.53 [0.27, 0.79] on PortScan, +1.67 [1.47, 1.89] on UNSW-Recon and +0.38 [0.21, 0.57] on ToN-IoT, all Holm-resolved (Table 9). PortScan and UNSW meet the pre-specified +0.5-point materiality rule on the registered mean estimand, although the PortScan seed distribution remains heterogeneous around that margin; ToN-IoT is resolved but sub-material. The pre-specified verdict is PARTIAL ROBUSTNESS; the sourcerow block’s HOMOGENEOUS - SIZE BENEFIT classification is therefore not retained as the headline. Point gating adds no resolved benefit at parity and strict gating has one resolved sub-material cost on UNSW-Recon. The betweenblock sensitivity-minus-historical contrast on the size effect is compatible on UNSW (+0.01), unresolved on PortScan (−0.29) and materially attenuated on ToN-IoT (−0.62, Holm p = 0.0149); the pre-specified rule therefore does not classify the original size effect as materially inflated overall, and these independent-block changes are not causal estimates of duplicate exposure. The absolute value of updating also moved between blocks. Always-deploy at 2,000/class retains a positive mean effect over never-adapt in all three benchmarks (+7.10 [5.26, 8.96], +3.90 [3.56, 4.26] and +1.06 [0.37, 1.92]; ToN-IoT Holm p = 0.017), although the ToN-IoT seed distribution is highly heterogeneous and its median effect is near zero; at 512/class the corresponding mean gains are +6.57, +2.23 and +0.68, the last no longer statistically resolved (CI95 [0.01, 1.52], Holm p = 0.074). The exact-feature-disjoint block therefore also yielded smaller absolute always-deploy gains in PortScan and ToN-IoT than the earlier independent block (Table 6: +9.72 and +2.44 at 2,000/class, +8.90 and +1.44 at 512/class), whereas UNSW-NB15 was similar (+3.79 and +2.13). Because these are independent seed blocks, this comparison is descriptive rather than a causal attribution to duplicate exposure. Security metrics behind the size effect. In the same sealed cells the balanced-accuracy advantage of 2,000/class over 512/class always-deploy is driven primarily by a lower false-positive rate — 5.44 → 4.70 (PortScan), 31.20 → 27.51 (UNSW-Recon) and 28.39 → 27.81 (ToN-IoT) percentage points — while attack recall is approximately stable (89.82 → 90.13 and 97.16 → 97.34) and slightly lower on UNSW-Recon (97.86 → 97.51; Table 9). The size benefit should therefore be read as fewer false alarms at approximately unchanged attack recall, not as improved attack detection; a lower false-positive rate is not called better wherever a recall guardrail is crossed. The absolute false-positive rates of roughly 27–31% in the UNSW-Recon and ToN-IoT full-drift
Candidate Comparability Before Promotion
19
Table 9 Exact-feature-disjoint sensitivity of the full-drift size effect. Always-deploy with 2,000/class minus always-deploy with 512/class, balanced-accuracy points, 30 fresh seeds (7001–7030). Exact cleaned raw feature groups are assigned wholly to window, candidate-training or probe roles; multiplicity and original labels are retained. CI95 and Holm-adjusted p are from the registered deterministic paired bootstrap. The historical source-row-disjoint estimate is shown for context; “change” is sensitivity minus historical, estimated between independent seed blocks and is not a causal duplicate- leakage effect. The last two columns give always-deploy attack recall and false-positive rate (%) at 512 → 2,000 per class from the same sealed cells: the balanced-accuracy advantage is driven mainly by the lower false-positive rate, with recall approximately stable and slightly lower on UNSW-Recon. Benchmark PortScan UNSW-Recon ToN-IoT
Historical Exact-feature-disjoint [CI95] +0.82 +0.53 [0.27, 0.79] +1.66 +1.67 [1.47, 1.89] +1.00 +0.38 [0.21, 0.57]
pHolm Class
Change [CI95]
Change class
Recall 512→2,000 FPR 512→2,000
0.00018 material benefit −0.29 [-0.60, 0.01] unresolved 89.82→90.13 0.00003 material benefit +0.01 [-0.24, 0.28] compatible 97.86→97.51 0.00018 resolved sub-material −0.62 [-1.04, -0.19] material attenuation 97.16→97.34
cells are not deployment-grade: this study evaluates promotion behaviour under the chosen benchmark pipeline, not a production-ready detector. B1: policy interpretation is partially robust. Four of six pre-specified robustness predicates hold, with no direct historical material-gain/material-cost reversal. No evaluated policy globally dominates, policy ordering remains candidate-sizedependent, ATC remains compatible with the point gate in five of six scenarios, and material costs remain among alternatives. The stronger source-row retention claims do not survive intact: at full drift ATC is unresolved on PortScan/UNSW and compatible on ToN-IoT, and the calibrated ensemble is unresolved/material cost/compatible; DoC costs materially on PortScan and UNSW, replay and DDM on UNSW, and ADWIN in all three, always at the registered reference parameters for DDM/ADWIN. Point/strict anchors add no positive material effect at parity. The pre-specified B1 verdict is PARTIALLY ROBUST: validation and policy rankings remain conditional on construction and evidence, and no policy is globally dominant. The thesis that survives the sensitivity is therefore narrower than the source-row blocks suggested: nominal evidence has a positive resolved effect in all three exact-featuredisjoint benchmarks, but material benefit is benchmarkdependent; policy ordering remains size-dependent; and validation remains conditional. No causal claim is made about exact duplicates, and no generalization extends beyond exact duplicate-value exposure.
5.8 Mechanism, formal instruments and external boundaries The remaining blocks support the mechanism behind the results above, price the formal and operational cost of validating, and delimit external validity; Table 10 states each block’s question, main finding and scope, and the full grids are in Online Resource 1. Except where noted, these blocks are evidence under the historical frozen-transformer configuration at 512/class, and their numbers should not be assumed to transfer to self-contained, size-matched pipelines. Mechanism: update value tracks incumbent degradation, not detector score. At each of 250 triggers the harness
5.44→4.70 31.20→27.51 28.39→27.81
measured the incumbent’s degradation over the five preceding windows and the candidate-minus-incumbent BA over the five following windows — predictor and outcome share no algebraic term. Pre-trigger degradation predicts the future value of committing in every regime (regime-specific r = −0.67 to −0.70, every clustered CI95 away from zero; pooled over the three regimes r = −0.57, Fig. 3), while the detector’s score at the same triggers shows no consistent association (regime-specific r = −0.01 to +0.05, every CI95 straddling zero; pooled +0.02). A mixed-effects model controlling for severity and stream time confirms it: degradation is the only predictor whose CI95 consistently excludes zero (βdeg = −1.02 [−1.61, −0.43] per BA point; VIF ≤ 2.4), and the analysis replicates on the quantum-detector logs; within triggered decisions the evaluated detectors’ scores carry no consistent incremental signal, not provably none. The association is descriptive: triggers are nested in seeds and regimes, past and future performance are autocorrelated, and a bounded accuracy series gives a degraded incumbent more headroom mechanically, so this is predictive support for the degradation–headroom account and does not identify a causal mechanism (Fig. 3; Online Resource 1, §S1.1, §S2.10). Supporting robustness. Observed-data: with every oracle privilege and leakage path removed (candidate, probe and detector recalibration drawn from observed past traffic only, without replacement from a value-deduplicated pool), the leakage-free observed-data arm reproduces the harm-regime rescue (+4.05 and +4.83 above always-deploy on ToN-IoT full and mild drift); it is the only tier free of simulator-oracle information (Online Resource 1, §S2.7). Classifier and generator controls: at 512/class the harmful zero-drift sign appears for random forest, logistic regression, MLP and full-feature SVC and across replacement generators; size-matching removes it for the robust learners, localizing the residual magnitude to the fragile SVC-RBF pipeline (Online Resource 1, §S1.6, §S2.12). Quantum detector: the harm, the gate’s rescue and the per-trigger mechanism all replicate under a quantum-kernel monitor at a ∼114× simulated cost, a detector-invariance check rather than a quantumadvantage claim (Online Resource 1, §S1.2).
20
R. Fernández-Barrios et al.
Table 10 Supporting evidence blocks. Each is detailed in Online Resource 1 at the section shown; scopes are stated in §7. Extension
Question
Main result
Scope / limitation
Observed-data (§S2.7)
Survives without simulator privileges?
Harm-regime rescue reproduces leakagefree (+4.05/+4.83 above always-deploy)
Classifier / generator (§S1.6, S2.12)
Is the harm SVC-specific?
Sign general at 512; size-matching clears robust learners
Mild drift (§S2.12)
Is harm dataset-specific?
Quantum monitor (§S1.2) VBC-SG frontier (§S2.12) Chronological replays (§S2.6)
Does a more expressive detector change conclusions? What do formal guarantees cost?
Healthy incumbents make always-deploy harmful in all three No
Frozen, 512/class; UNSW below no-adapt at 64-flow windows Magnitude pipelinedependent; frozen transformer Frozen, 512/class
Prevalence / acquisition (§S2.3)
What does finding attack labels cost?
External time-ordered boundary?
Non-vacuous from b=256; abstention/labels priced No net harm observed in any of 13 replays (7 in Table 11, 6 earlier); gates pay a premium on collapsed incumbents ≈1/π inspected flows; alert-enrichment cuts it 5–8×
VBC-SG: what validating formally costs. The sequential family of §3.5 controls, under the conditional weak null of Proposition 1, the per-proposal probability of falsely establishing probe-superiority; it does not bound future deployment harm. A pre-specified budget frontier prices it (Online Resource 1, §S2.12): the lifetime-budgeted pooled sequence recovers 93% of always-deploying’s benefit at cap 512 (≈578 adjudicated labels per proposal, deferrals included) under the approximate pooled analysis, while the fully stratified VBC-SG-Cohort-sim, which carries the formal deploymentlong guarantee, reaches 81%. The deployment-long stratified guarantee is thus non-vacuous within the evaluated balanced-probe budget, from b=256 onward, at an abstention of 74–86%. Empirically the risk-controlled gates committed zero observed harmful updates over a complete 520-commit accounting (506 evaluable) against 39% for always-deploy; these commits cluster within shared seeds and are not treated as 506 independent Bernoulli trials, so, being not independent trials, they support no population-rate bound and “zero observed” is not “eliminates”. A pooled sequential gate provides useful empirical risk control at tens of adjudicated labels, while the formally aligned stratified guarantee requires substantially larger balanced-probe budgets. All of these numbers are evidence under the historical frozentransformer policy; the frontier was not re-evaluated under the size-matched self-contained controls, and nothing here shows VBC-SG to be necessary in that setting (Online Resource 1, §S2.9, §S4). Chronological replays: the external boundary. These replays ask whether the controlled-condition harm and the value of validation reappear on real timelines. Thirteen registered replays on real, chronologically ordered streams were evaluated in total: the seven of the pre-enumerated final matrix reported in Table 11 (seeds 601–630) and six earlier reg-
∼114× simulated cost; invariance check only Frozen; not necessary under the size-matched controls No ToN-IoT analogue; no prevalence estimate Not end-to-end; starved trigger is the upstream limit
istered replays reported in Online Resource 1, §S2.6. Each processes 200 windows of 256 flows strided to span the capture: ordering, labels and the temporal availability of every probe are real; the sampling density is not. Every CICIDS incumbent collapses on a later window (no-adaptation BA 49–72%) as each stream presents an unseen attack sub-type, and drift-triggered retraining recovers +13.6 to +43.8 BA points; on these deep-benefit streams the point and strict gates retain 73–100% of that recovery (48% for strict on the shallow Thursday intra-day split), paying a real, metric-dependent premium in balanced accuracy on three streams, and a pre-specified interventional test rules out probe class composition as the cause (probe staleness remains the hypothesis consistent with the data). Where the incumbent stays healthy, on the two UNSW-NB15 timelines (82.9% and 84.2% BA), updating helps moderately and the point and strict gates have higher point estimates than always-deploy (strict +2.78 and +1.34, descriptive, committing 2.5 and 3.6 times per stream against 13.0 and 17.3), while the healthy Wednesday intra-day split is an unresolved counterexample with the gates 0.5–0.7 points below always-deploy. These are descriptive comparisons, not inferential superiority claims: the pre-specified chronological family tests strict against noadaptation, a registration that is structurally easy to satisfy wherever the incumbent collapses, so its confirmatory content is confined to the healthy-incumbent replays. Net harm remains unobserved: on none of the thirteen chronological streams does always-deploy lose to never adapting, so the harmful-update finding stays a controlledcondition result. This is an absence of observed harm in thirteen replays, not an estimate of harm frequency and not evidence that harm cannot occur; the replays estimate no deployment harm frequency, and ToN-IoT ships no timestamps, so the controlled harm benchmark has no chronological analogue. The price of that conservatism is equally visible in
Candidate Comparability Before Promotion
(a) KS-max (classical) pooled r = -0.57, n = 250
40
Candidate − incumbent BA over the 5 windows AFTER (pts)
Candidate − incumbent BA over the 5 windows AFTER (pts)
40
21
20
0
−20
−40
(b) KS-max (classical) pooled r = +0.02, n = 250
20
0
−20
−40
0.5
0.6
0.7
0.8
0.9
0.14
0.16
Incumbent BA over the 5 windows BEFORE the trigger (c) QK-ZZ (quantum) pooled r = -0.59, n = 311
40
Candidate − incumbent BA over the 5 windows AFTER (pts)
Candidate − incumbent BA over the 5 windows AFTER (pts)
40
0.18
0.20
0.22
0.24
0.26
KS-max detector score at the trigger
20
0
−20
−40
(d) QK-ZZ (quantum) pooled r = +0.01, n = 311
20
0
−20
−40
0.5
0.6
0.7
0.8
0.9
1.0
0.000
Incumbent BA over the 5 windows BEFORE the trigger
PortScan (benefit)
0.005
0.010
0.015
0.020
0.025
0.030
QK-ZZ detector score at the trigger
UNSW Recon (marginal)
ToN-IoT (harm)
Fig. 3 Per-trigger test on the hardened harness (seeds 104–133; 250 KS-max triggers): incumbent balanced accuracy over the five windows before the trigger (left) and detector score at the trigger (right), each against the candidate-minus-incumbent balanced accuracy over the five following windows, for the KS-max (top) and QK-ZZ (bottom) detectors; colours and marker shapes mark the three regimes. Panel annotations give the correlation pooled over the three regimes (r = −0.57 and −0.59 for degradation, +0.02 and +0.01 for the score); the regime-specific correlations quoted in §5.8 (−0.67 to −0.70) are computed within each regime and are a different statistic. Predictor and outcome share no algebraic term (disjoint past/future windows); clustered CIs and the hierarchical model are in §5.8.
Table 11: on the five CICIDS replays VBC-SG-Cohort-sim commits at most once per stream and retains only 0–9% of always-deploy’s recovery, whereas on the two healthy UNSW timelines it retains 91% and 129% at 1,232 and 1,634 labels per stream. Formal lifetime risk control can become extremely conservative exactly where the incumbent has the most recovery headroom; this is the operational price of the guarantee. Operational feasibility. Every count above is an adjudicated label; an operator pays in inspected flows, and finding an attack label at operating prevalence costs about 1/π inspections. A pre-specified acquisition-yield simulation prices that search: alert-enriched inspection cuts it 5–8× at every prevalence while the commit rule sees only an independent uniform validation sample (32 adjudications), so enrichment improves discovery without touching decision validity; the arm is bounded (candidate batch and detector calibration remain balanced) and is not an end-to-end de-
ployment cost. A separate prevalence sensitivity shows the gate holds down to π ≈ 0.05 but its harm-regime protection dissolves at π = 0.01, where two-thirds of 32-flow probes contain no attack; there the binding constraint is a starved trigger, not the gate (Online Resource 1, §S2.3). The historical frozen-policy diagnostic further shows that simply rejecting probe ties (reject-ties) recovers most of the residual zero-drift loss at no additional labels (Online Resource 1, §S2.12).
6 Discussion The study — historical diagnostic, ownership replication, zero-drift and full-drift size controls, common-harness comparison, exact-feature-disjoint sensitivity, chronological matrix — answers three questions. Q1: How do challenger construction and evidence condition promotion conclusions? Promotion conclusions, and
22
R. Fernández-Barrios et al.
Table 11 The registered chronological matrix. Seven pre-enumerated time-ordered replays (CICIDS2017 train→future day pairs, two intra-day splits, UNSW-NB15 at two training fractions), 200 windows of 256 flows in capture order, seeds 601–630. Columns: balanced-accuracy points over never-adapting; no-adapt is the frozen incumbent’s absolute BA (the health indicator). No stream shows net harm. Where the incumbent collapses (CICIDS), the gates preserve part of always-deploying’s recovery but can impose substantial costs; where the incumbent stays healthy, point estimates for point and strict gates are above always-deploying on both UNSW timelines (VBC-SG on the 20% split but not the 40%), with the healthy Wednesday intra-day replay the unresolved counterexample. These are descriptive comparisons, not direct inferential superiority claims. Replay Tue → Wed+Thu+Fri Wed → Fri Thu → Fri Wed intra-day Thu intra-day UNSW (train 20%) UNSW (train 40%)
no-adapt BA
naive
point
strict
VBC-SG
59.0 51.8 49.0 84.7 48.3 82.9 84.2
+14.78 +35.29 +36.07 +6.47 +0.88 +8.35 +7.64
+11.52 +27.64 +27.71 +6.02 +0.88 +9.90 +8.38
+11.50 +28.81 +26.51 +5.77 +0.42 +11.13 +8.98
+0.16 +0.91 +1.56 +0.56 +0.00 +10.76 +6.92
the apparent ranking of update policies, are conditional on challenger construction and evidence; these factors should therefore be controlled, reported and interpreted explicitly. The sharpest form of the result is the ordering dependence: with evidence-disadvantaged 512-per-class challengers every label-free alternative appeared to help at zero drift, whereas at parity those gains vanished, and the pre-specified orderingchange rule fired for ATC and the ensemble in the sourcerow block and for ATC, DoC and the ensemble under exactfeature-disjoint roles (§5.6, §5.7). A policy ranking obtained with one candidate generator need not survive another; this is a conditionality result, not a claim that any one construction is mandatory. Frozen incumbent-owned preprocessing accounted for the mean full-drift harm in its paired control (the ownership interaction reaches +6.0 points where the harm was reported), and the challengers’ 4× nominal per-class evidence disadvantage accounts for the residual zero-drift mean harm (all size effects Holm-significant; matched-size mean effects compatible with ±0.5 points in 3/3). Self-contained preprocessing is the controlled alternative evaluated here, not a universal production requirement, and nominal row-count parity is not effective information parity. The full-drift source-row-disjoint control first found a homogeneous +0.82 to +1.66-point effect. The final exactfeature-disjoint sensitivity retains a positive resolved effect in all three (+0.53, +1.67, +0.38), but the ToN-IoT effect is sub-material; the defensible conclusion is therefore benchmark-dependent rather than homogeneous. In security terms the effect is a lower false-positive rate at approximately unchanged attack recall (§5.7), obtained on a benchmark pipeline whose absolute false-positive rates under full drift are not deployment-grade. The same block also yielded smaller absolute always-deploy gains on PortScan and ToNIoT than the earlier independent block; being a betweenblock comparison, that change is descriptive. The controls identify representation and nominal row-count effects only within the evaluated pool construction; they do not establish effective information parity or attribute the between-design change causally to duplicates.
Q2: When does validation add value? Validation has its greatest observed average value when candidate construction or evidence is asymmetric or candidate quality is uncertain. Under the frozen transformer the gate partly compensated a representation bias; at self-contained 512/class it compensated a nominal-evidence disadvantage (up to +1.68 points, guardrail-clean in all but one cell). At matched 2,000-perclass evidence the source-row blocks show no positive resolved gate effect and one strict-gate cost; the exact-featuredisjoint B2 and B1 repetitions preserve that qualitative conclusion. The experiments therefore do not support universal validation, and we do not call validation necessary, generally safer, superior, or the recommended default. Where it earns its keep, the gate works because it estimates a contemporaneous surrogate for the sign of the candidate–incumbent difference from a small labeled probe whose validity does not depend on the model being judged; the realized future value remains observable only downstream. A defensible operational interpretation, which extrapolates beyond the fixed-policy evidence, is to validate when upstream comparability cannot be established cheaply or when tail-risk or governance requirements justify the caution, and to treat a demonstrably comparable, well-evidenced challenger as one that validation will mostly delay; we did not evaluate a meta-controller that selects a gate or its strength online. The label bill is layered: retraining labels are unavoidable for any labeled-retraining loop; when candidate and validation labels are acquired jointly, reserving a pre-specified disjoint holdout of that same (deduplicated) labeled batch as the probe need not require label acquisition beyond the batch, at the opportunity cost of the held-out flows, which are not used to train the challenger; and converting average harm into average benefit in the controlled harm regime costs a fresh probe of ≈32 flows per confirmed drift under the controlled balanced-probe protocol, an adjudication count compatible with analyst-assisted workflows, though the inspected-flow acquisition cost of the balanced candidate batch and of the formal stratified probes was not evaluated. On collapsed
Candidate Comparability Before Promotion
chronological incumbents every conservative layer forfeits part of a large recovery, VBC-SG almost all of it (§5.8): where degradation is strong and well established, excessive validation delays beneficial adaptation, and the operator’s protection is bounded deferral, adequate probe budgets and rollback capability rather than more gating. Q3: What does the common-harness comparison say about alternative update policies? Different alternatives occupy different information/accuracy/update-cost trade-offs, and none dominates. Under exact-feature-disjoint roles ATC is unresolved on PortScan and UNSW and compatible on ToN-IoT at full drift, the calibrated ensemble is unresolved/material cost/compatible, DoC retains material costs on PortScan and UNSW, replay and DDM on UNSW, and ADWIN in all three (at the registered reference parameters for DDM/ADWIN), while ATC remains compatible with the point gate in five of six scenarios. These are statements about fixed evaluated configurations, not general properties of the methods. Implications for operational NIDS, and what is not implied. Detection is not decision: a detector reports that the distribution moved, while promotion concerns the sign of the candidate–incumbent difference, with which the evaluated detector scores showed no consistent association within triggered decisions (regime-specific r between −0.01 and +0.05 at 250 logged triggers, pooled +0.02; §5.8); the more expensive quantum-kernel monitor did not change promotion outcomes in its frozen-policy check. The controlled harm results license no prevalence claim about production, and the pool-constructed trajectories are not real deployment timelines. The contribution is evaluation discipline: challenger representation, training evidence and exact-value role separation should be stated before attributing harm, benefit or policy ordering, confined to the network-security benchmarks studied. Trust assumptions, security significance and scope. The asset is the integrity and performance of the deployed intrusion detector; the decision is whether a challenger replaces the incumbent. In a deployed NIDS that promotion is a security-relevant integrity decision: it replaces the model responsible for all subsequent attack detection, so a poorly supported update can alter the detector’s future security behaviour — its attack recall and false-positive rate, which is why both are carried as guardrails throughout — even when the drift alarm itself was correct. In every confirmatory block the candidate-training data, the trigger process and the validation evidence are assumed not to be adaptively manipulated by an adversary. Automated retraining does expose, in principle, an update surface to data manipulation: an adversary able to influence the traffic that raises alarms or trains challengers could steer what gets promoted, a risk studied in the poisoning literature [18]. Adversarial candidate poisoning, trigger manipulation and adaptive attacks on the validation
23
probe are out of scope: the present experiments do not model an adversary manipulating the update process — candidate training data, drift triggers and validation probes are all assumed honest — and the evaluated validation policies are not claimed as poisoning defenses. The only related evidence is a bounded, non-adaptive robustness probe from the initial study, in which uniformly random flips of up to 40% of the probe labels left the gate harm-avoiding (Online Resource 1, §S1.5); it does not model an adaptive adversary. The security contribution of this study is an evaluation discipline for a decision whose outcome determines the detector that will face the next attack. Operational guidance, not a validated stack. The full combination — a self-contained challenger, nominal size matching, observed-data construction, a detector alarm process, chronological deployment and natural prevalence — was not evaluated jointly. The sequence below is therefore evaluation guidance and future-work hypotheses derived from separate controls, not a demonstrated or recommended deployment configuration: 1. Audit candidate construction. Do not treat incumbentowned frozen preprocessing as a neutral evaluation convention; when comparability is intended, use a selfcontained challenger, and if production deliberately freezes the incumbent’s representation, state and validate that policy explicitly (§5.2). 2. Audit evidence comparability. Match the challenger’s nominal training evidence to the incumbent’s, or treat any shortfall as a reason to validate (§5.3, §5.4); nominal parity is necessary for a fair reading, not sufficient for full comparability. 3. Assess incumbent health. Degradation, not detector score, predicts update value (§5.8); the cost and calibration of such monitoring were not evaluated here. 4. Condition validation strength on uncertainty. Validate conservatively when candidate quality or incumbent health is uncertain; when the challenger is comparable and degradation is strong and well established, avoid unnecessary delay, since conservative gates measurably forfeit recovery there (§5.8). 5. Monitor post-promotion outcomes and retain rollback capability. Mean compatibility within a margin is not a per-proposal guarantee (§5.8, §5.3). Under this design hypothesis, and conditional on a comparable proposal being generated, the detector acts mainly as a scheduler; at extreme class imbalance, detector calibration and alarm starvation determine whether any of this is invoked at all. Computational overhead of self-contained challengers (operational implication, not a benchmark). A selfcontained challenger refits its own preprocessing bundle (a feature standardizer and an 8-component PCA) on its candidate batch before training the classifier, and a commit de-
24
ploys the complete bundle; inference cost after deployment is structurally similar under both policies. We did not run an isolated timing study of the two policies. The only measured figures are coarse per-arm wall-clock times of sequential single-machine runs: across the 18 matched scenario×policy pairs of the symmetric-pipeline replication, self-contained arms took on average 0.97× the wall-clock of their frozen counterparts (range 0.89–1.06), so the transformer refit is not visible at whole-arm granularity, and the 2,000-per-class arms of the size-matched control took 1.2–1.5× the wallclock of their 512-per-class counterparts, consistent with SVC-RBF cost growing with the batch. These whole-arm timings do not isolate preprocessing overhead, and that a self-contained arm occasionally runs faster than its frozen counterpart is run-to-run noise, not a speedup caused by selfcontained preprocessing; they are operational context, not a benchmark, and distinct from the ∼114× simulated overhead of the quantum-kernel monitor (§5.8), which concerns the drift detector rather than the challenger pipeline. 7 Limitations Benchmarks and drift construction. Three public benchmarks in binary benign-versus-attack form, with scenarios constructed from regime pools, do not constitute broad external validation; the strongest benefit evidence is concentrated in CICIDS2017, and the marginal and harmful ends come from UNSW-NB15 and ToN-IoT. The core drift trajectories are gradual mixing ramps between real regime pools (covariate/regime drift); we do not isolate a p(y | x) change, and recurrent drift is untested. Candidate batches and probes are sampled at the true current severity, information a production system lacks; the leakage-free observed-data arm (§5.8; Online Resource 1, §S2.7) reproduces the harmful-regime rescue from observed past traffic only, but equivalence to the oracle-probe arm was not tested and the size-matched controls were not repeated in observed-data form. Learners and pipelines. The primary pipeline is SVCRBF with an 8-dimensional PCA chosen for quantumsimulation tractability. At full drift the historical net-harm result is specific to that fragile classifier, the fragile-model tail of the evaluated learner spectrum, and does not appear for more robust classifiers (Online Resource 1, §S1.6; §5.8); incremental or weighted online learners with no discrete replacement step remain untested, and the full-drift size control and the common-harness comparison were run on SVCRBF only, so effect magnitudes are pipeline-dependent and generalization to other learners requires further evaluation. The pipeline’s absolute false-positive rates under full drift (roughly 27–31% on UNSW-Recon and ToN-IoT) are not deployment-grade; the study evaluates promotion behaviour under this benchmark pipeline, not a production-ready detector. Neither the frozen-policy full-drift harm nor the 512-
R. Fernández-Barrios et al.
challenger zero-drift harm should be quoted as a general property of drift-triggered updating: harm is what happens when a healthy model is replaced by a challenger whose construction or evidence is materially asymmetric, a condition on the proposal, not on the dataset. Nominal evidence is not effective information. The size-matched controls equate nominal per-class sample size; temporal coverage, diversity, subtype support, label quality, duplication, prevalence and effective sample size are not equated by equal row counts, and with-replacement candidate draws mean nominal rows can repeat even in the exactfeature-disjoint role design (§5.3). The final sensitivity prevents the same exact cleaned feature vector from crossing window/train/probe roles but preserves within-role multiplicity; it addresses exact duplicate-value exposure only, not approximate similarity, semantic duplicates or dependence among distinct vectors. The full-drift sensitivity establishes positive resolved size effects only within its nested pool design, with a material benefit in two of three benchmarks; we make no claim that larger candidates always help or that the effect is monotone beyond the two sizes. The zero-drift trigger is a random proposal rate, not a detector alarm process. The controls isolate the design factors individually; the complete self-contained + size-matched + observed-data + real-alarm + natural-prevalence + exact-feature-disjoint combination was not evaluated jointly, so their joint deployment behaviour is untested and the operational sequence in §6 is a design hypothesis derived from separate controls. The contribution is evaluation methodology, not the validation of a deployment recipe. Mean compatibility at the matched size does not demonstrate absence of an effect, nor imply that no individual committed candidate can be worse than the incumbent; the future-negative sign percentages of §5.3 are descriptive within-trajectory fractions over seed-clustered commits, not population prevalences or deployment probabilities. The common-harness comparison. Information budgets differ legitimately by method definition (ATC and DoC use a 512-row training-time validation sample per model and zero target labels, DDM/ADWIN consume 800 monitoring labels per stream, the gates 32 target labels per decision) and are reported rather than equalized; policies with probability outputs train Platt-calibrated SVCs, which leaves the classifiers’ predictions unchanged. DDM and ADWIN were run at their registered reference parameters and were not tuned; their cells characterize that configuration, and a parameter sweep would be required before reading them as properties of the methods. Recent adaptive-NIDS systems were reviewed as of September 2026 (Online Resource 1, §S11); their triggers, labels, update semantics and decision units are not decision-equivalent to a per-proposal incumbent–challenger comparison, so they are reported as related systems rather than reproduced unfaithfully. Comparisons are between fixed
Candidate Comparability Before Promotion
policies; no learned meta-controller was evaluated. Crosssize contrasts are seed-paired only. Scope of the formal, diagnostic and chronological instruments. The quantum detector, the full sequential frontier and the mild-drift matrix remain evaluated under the historical frozen-transformer policy; VBC-SG was not re-run under size-matched own pipelines, so none of those numbers should be assumed to transfer. VBC-SG’s guarantee — where it applies — controls false probe-superiority on the probe distribution under the conditional weak null of Proposition 1, not future deployment harm; the probe’s fixed class and mixture quotas are not proved to convert a marginal non-superiority null into that conditional null, and the bound equals a bound on future harm only when the probe is representative, an assumption our own stale-probe and coverage results show can fail (the empirical-Bernstein sequence exceeds nominal α under strong within-probe autocorrelation), and its deployment-long budget is prohibitive in its maximally-stacked stratified form at b=64. The gate assumes a small labeled probe at decision time; its size (≈32–64 adjudicated flows per confirmed drift) is an incremental adjudication count under the evaluated protocol, not an estimate of inspected-flow acquisition cost in an operationally imbalanced deployment, which is not modeled; under genuinely random inspection the gate’s harm-regime protection dissolves at π = 0.01, and the initial study’s stronger claim, which rested on a probe conditioned to contain one attack, is not retained. Label latency is modeled for the probe (up to 20 windows stale) and the candidate batch (5–20 windows; Table 2); asynchronous or partial labeling and candidate-aware label-flipping adversaries are untested. The thirteen chronological replays (seven in the registered final matrix, six earlier) are external boundary evidence: they provide no estimate or bound on how frequently harm occurs in deployment, most CICIDS streams collapse into deep-benefit territory, ToN-IoT ships no timestamps, so the controlled harm benchmark has no chronological counterpart, and the pre-specified chronological family is structurally easy to satisfy where the incumbent collapses. Quantum kernels are classically simulated, so the 114× monitoring overhead is a simulation figure. Statistical granularity and multiplicity. The 30 seeds of each block estimate stochastic variability conditional on that benchmark and pool design; they are not 30 independent deployment environments. The external study units are the three benchmarks (seven regimes in the exploratory stage), and confidence intervals should be read at that granularity (Online Resource 1, §S3 reports leave-one-out behaviour); the narrative therefore leads with effect sizes, intervals, heterogeneity across benchmarks and robustness across sensitivity blocks rather than with adjusted p-values. Multiplicity-adjusted pvalues are computed directly from paired per-seed contrasts using a deterministic centered bootstrap, with Holm [38]
25
within each pre-specified family of the core, construction, size, common-harness and exact-feature-disjoint blocks (Online Resource 1, §S6–S11), and Benjamini–Hochberg [16] separately for the pre-declared budget-frontier and chronological families. The exact-feature-disjoint B2-vs-historical contrasts use an independent-block bootstrap and Holm correction; they are sensitivity contrasts, not causal duplicate effects. Frozen-side cells, 512-vs-never columns and commonharness anchor contrasts remain descriptive and uncorrected.
8 Conclusion A drift alarm can justify constructing a challenger, but not promoting it. In this study the apparent outcome of promotion, and the apparent ranking of update policies, changed with two upstream conditions. With incumbent-owned frozen preprocessing removed, the mean full-drift harm did not persist; with nominal candidate evidence matched to the incumbent’s, the residual zero-drift deficit disappeared; and the pre-specified ordering-change rule fired for label-free and ensemble policies when the challenger’s evidence changed. Under exact-cleaned-feature-disjoint roles and pool-constructed progressive drift, the 512→2,000 intervention retained positive resolved effects of +0.53, +1.67 and +0.38 balancedaccuracy points, material in two of three benchmarks and driven mainly by fewer false positives. At nominal parity, point and strict validation provided no resolved positive average effect, and strict validation incurred a sub-material cost in one cell. The common-harness interpretation is partially robust: policy ordering remains candidatesize-dependent and no evaluated policy globally dominates, while the earlier ATC and calibrated-ensemble compatibility claims narrow. Promotion conclusions are conditional on challenger construction and evidence; these factors should be controlled, reported and interpreted explicitly. Validation is conditional rather than universal, the net-harm findings are configuration-dependent, and thirteen chronological replays showed no net harm from always deploying while bounding neither its frequency nor its impossibility. The integrated operational stack remains future work rather than an empirically validated recommendation.
Statements and Declarations Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-forprofit sectors. Competing interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
26
Ethics approval and consent to participate Not applicable: the study uses three public benchmark datasets and involves no human participants or animals. Consent for publication Not applicable: the manuscript contains no data or images of individual persons. Data availability The three public benchmarks (CICIDS2017, UNSW-NB15, ToN-IoT) are cited in the text. Source code, frozen protocols, configs, environment specification, reproducibility instructions, confirmatory result summaries and the hash manifest are archived at Zenodo in artifact version v1.24.0: the exact version DOI is 10.5281/zenodo.22239106, and the concept DOI 10.5281/zenodo.21322256 resolves to the latest version [27]. The previous v1.23.0 release remains immutable. Public benchmark datasets and voluminous raw/generated outputs are not redistributed; the released pipeline regenerates them, and every released result CSV used by the manuscript is pinned by SHA-256 in the manifest. Code availability The source code, frozen protocols, experiment runner and analysis scripts are part of the Zenodo artifact identified under Data availability. Author contributions Roberto Fernández-Barrios: Conceptualization, Methodology, Software, Data curation, Formal analysis, Visualization, Investigation, Writing – original draft, Writing – review & editing. Iker Pastor-López: Supervision, Validation, Writing – review & editing. Amaia PikatzaHuerga: Validation, Project administration, Writing – review & editing. Pablo García Bringas: Supervision, Resources, Writing – review & editing. Generative AI and AI-assisted technologies During preparation of this work the authors used Claude (Anthropic) and OpenAI Codex to assist with drafting and editing text and with implementation and review of analysis scripts. The authors defined the protocols and scientific decisions, reviewed and verified all AI-assisted output, and take full responsibility for the work.
References 1. Abdel Wahab, O.: Intrusion detection in the IoT under data and concept drifts: Online deep learning approach. IEEE Internet of Things Journal 9(20), 19706–19716 (2022). https://doi.org/10. 1109/JIOT.2022.3167005 2. Aburomman, A.A., Reaz, M.B.I.: AutoML for network-based intrusion detection: Evaluation practice, dataset quality, and deployment constraints. Future Internet 18(8), 383 (2026). https: //doi.org/10.3390/fi18080383 3. Aguiar, G.J., Cano, A.: A comprehensive analysis of concept drift locality in data streams. Knowledge-Based Systems 289, 111535 (2024). https://doi.org/10.1016/j.knosys.2024.111535
R. Fernández-Barrios et al. 4. Alsaedi, A., Moustafa, N., Tari, Z., Mahmood, A., Anwar, A.: TON_IoT telemetry dataset: A new generation dataset of IoT and IIoT for data-driven intrusion detection systems. IEEE Access 8, 165130–165150 (2020). https://doi.org/10.1109/ACCESS.2020. 3022862 5. Amalapuram, S.K., Tamma, B.R., Channappayya, S.S.: SPIDER: A semi-supervised continual learning-based network intrusion detection system. In: IEEE INFOCOM 2024 – IEEE Conference on Computer Communications, pp. 571–580. IEEE (2024). https://doi.org/10.1109/INFOCOM52122.2024.10621428 6. Andresini, G., Pendlebury, F., Pierazzi, F., Loglisci, C., Appice, A., Cavallaro, L.: INSOMNIA: Towards concept-drift robustness in network intrusion detection. In: Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec), pp. 111– 122 (2021). https://doi.org/10.1145/3474369.3486864 7. Apruzzese, G., Laskov, P., Schneider, J.: SoK: Pragmatic assessment of machine learning for network intrusion detection. In: 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P), pp. 592–614. IEEE (2023). https://doi.org/10.1109/ EuroSP57164.2023.00042 8. Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., Rieck, K.: Dos and don’ts of machine learning in computer security. In: 31st USENIX Security Symposium (USENIX Security 22), pp. 3971–3988 (2022) 9. Baek, C., Jiang, Y., Raghunathan, A., Kolter, J.Z.: Agreementon-the-line: Predicting the performance of neural networks under distribution shift. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 19274–19289 (2022) 10. Baena-García, M., del Campo-Ávila, J., Fidalgo-Merino, R., Bifet, A., Gavaldà, R., Morales-Bueno, R.: Early drift detection method. In: 4th International Workshop on Knowledge Discovery from Data Streams (IWKDDS) (2006) 11. Bakirov, R., Fay, D., Gabrys, B.: Automated adaptation strategies for stream learning. Machine Learning 110(6), 1429–1462 (2021). https://doi.org/10.1007/s10994-021-05992-x 12. Bansal, G., Nushi, B., Kamar, E., Weld, D.S., Lasecki, W.S., Horvitz, E.: Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 2429–2437 (2019). https://doi.org/10.1609/aaai.v33i01.33012429 13. Barbero, F., Pendlebury, F., Pierazzi, F., Cavallaro, L.: Transcending TRANSCEND: Revisiting malware classification in the presence of concept drift. In: 2022 IEEE Symposium on Security and Privacy (S&P), pp. 805–823 (2022). https://doi.org/10.1109/SP46214.2022. 9833659 14. Bayram, F., Ahmed, B.S., Kassler, A.: From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Based Systems 245, 108632 (2022). https://doi.org/10. 1016/j.knosys.2022.108632 15. Bellante, A., Fioravanti, T., Carminati, M., Zanero, S., Luongo, A.: Evaluating the potential of quantum machine learning in cybersecurity: A case-study on PCA-based intrusion detection systems. Computers & Security 154, 104341 (2025). https: //doi.org/10.1016/j.cose.2025.104341 16. Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) 57(1), 289– 300 (1995). https://doi.org/10.1111/j.2517-6161.1995.tb02031.x 17. Bifet, A., Gavaldà, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of the 2007 SIAM International Conference on Data Mining (SDM), pp. 443–448 (2007). https://doi.org/10.1137/1.9781611972771.42 18. Biggio, B., Nelson, B., Laskov, P.: Poisoning attacks against support vector machines. In: Proceedings of the 29th International Conference on Machine Learning (ICML), pp. 1467–1474 (2012). ArXiv:1206.6389
Candidate Comparability Before Promotion 19. Cacciarelli, D., Kulahci, M.: Active learning for data streams: a survey. Machine Learning 113, 185–239 (2024). https://doi.org/10. 1007/s10994-023-06454-2 20. Camarda, F., De Paola, A., Drago, S., Ferraro, P., Lo Re, G.: Managing concept drift in online intrusion detection systems with active learning. In: Proceedings of ITASEC & SERICS 2025, CEUR Workshop Proceedings, vol. 3962 (2025) 21. Chen, Y., Ding, Z., Wagner, D.: Continuous learning for Android malware detection. In: 32nd USENIX Security Symposium (USENIX Security 23), pp. 1127–1144. USENIX Association (2023). URL https://www.usenix.org/conference/usenixsecurity23/ presentation/chen-yizheng 22. Constantinides, C., Shiaeles, S., Ghita, B., Kolokotronis, N.: A novel online incremental learning intrusion prevention system. In: 2019 10th IFIP International Conference on New Technologies, Mobility and Security (NTMS) (2019). https://doi.org/10.1109/ NTMS.2019.8763842 23. Darling, D.A.: The Kolmogorov–Smirnov, Cramér–von Mises tests. The Annals of Mathematical Statistics 28(4), 823–838 (1957). https: //doi.org/10.1214/aoms/1177706788 24. De Paola, A., Drago, S., Ferraro, P., Lo Re, G.: HOIDS: Concept drift aware hybrid online intrusion detection system. Journal of Network and Computer Applications 254, 104556 (2026). https: //doi.org/10.1016/j.jnca.2026.104556 25. Deng, W., Zheng, L.: Are labels always necessary for classifier accuracy evaluation? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15069– 15078 (2021) 26. Engelen, G., Rimmer, V., Joosen, W.: Troubleshooting an intrusion detection dataset: the CICIDS2017 case study. In: 2021 IEEE Security and Privacy Workshops (SPW), pp. 7–12 (2021). https: //doi.org/10.1109/SPW53761.2021.00009 27. Fernández-Barrios, R., Pastor-López, I., Pikatza-Huerga, A., García Bringas, P.: Validate before commit: reproducibility artifact (code and protocols). Zenodo [software]; concept DOI 10.5281/zenodo.21322256 resolves to the latest version (2026). https://doi.org/10.5281/zenodo.22239106. URL https://github.com/ roberto-fernandez-barrios/Validate-Before-Commit/tree/v1.24.0. Version 1.24.0 28. Flood, R., Engelen, G., Aspinall, D., Desmet, L.: Bad design smells in benchmark NIDS datasets. In: 2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P), pp. 658–675. IEEE (2024). https://doi.org/10.1109/EuroSP60621.2024.00042 29. Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Advances in Artificial Intelligence – SBIA 2004, pp. 286–295 (2004). https://doi.org/10.1007/978-3-540-28645-5_29 30. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Computing Surveys 46(4), 44:1–44:37 (2014). https://doi.org/10.1145/2523813 31. Garg, S., Balakrishnan, S., Lipton, Z.C., Neyshabur, B., Sedghi, H.: Leveraging unlabeled data to predict out-of-distribution performance. In: International Conference on Learning Representations (ICLR) (2022). URL https://openreview.net/forum?id=o_ HsiMPYh_x 32. Gouveia, A., Correia, M.: Towards quantum-enhanced machine learning for network intrusion detection. In: 2020 IEEE 19th International Symposium on Network Computing and Applications (NCA) (2020). https://doi.org/10.1109/NCA51143.2020.9306691 33. Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. Journal of Machine Learning Research 13, 723–773 (2012) 34. Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., Schmidt, L.: Predicting with confidence on unseen distributions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1134–1144 (2021). https://doi.org/10.1109/ ICCV48922.2021.00117
27 35. Guissouma, H., Zink, M., Sax, E.: Continuous safety assessment of updated supervised learning models in shadow mode. In: 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C) (2023). https://doi.org/10.1109/ICSA-C57050. 2023.00069 36. Han, E., Huang, C., Wang, K.: Model assessment and selection under temporal distribution shift. In: Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 235, pp. 17374–17392 (2024) 37. Havlíček, V., Córcoles, A.D., Temme, K., Harrow, A.W., Kandala, A., Chow, J.M., Gambetta, J.M.: Supervised learning with quantumenhanced feature spaces. Nature 567, 209–212 (2019). https: //doi.org/10.1038/s41586-019-0980-2 38. Holm, S.: A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2), 65–70 (1979) 39. Howard, S.R., Ramdas, A., McAuliffe, J., Sekhon, J.: Timeuniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics 49(2), 1055–1080 (2021). https://doi.org/10. 1214/20-AOS1991 40. Huang, H.Y., Broughton, M., Mohseni, M., Babbush, R., Boixo, S., Neven, H., McClean, J.R.: Power of data in quantum machine learning. Nature Communications 12, 2631 (2021). https://doi.org/ 10.1038/s41467-021-22539-9 41. Jacobs, A.S., Beltiukov, R., Willinger, W., Ferreira, R.A., Gupta, A., Granville, L.Z.: AI/ML for network security: The emperor has no clothes. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS ’22), pp. 1537– 1551. ACM (2022). https://doi.org/10.1145/3548606.3560609 42. Jiang, Y., Nagarajan, V., Baek, C., Kolter, J.Z.: Assessing generalization of SGD via disagreement. In: International Conference on Learning Representations (ICLR) (2022). URL https: //openreview.net/forum?id=WvOGCEAQhxl 43. Jordaney, R., Sharad, K., Dash, S.K., Wang, Z., Papini, D., Nouretdinov, I., Cavallaro, L.: Transcend: Detecting concept drift in malware classification models. In: 26th USENIX Security Symposium (USENIX Security 17), pp. 625–642 (2017) 44. Kaissar, A., Nassif, A.B., Bouridane, A.: Enhancing network intrusion detection with quantum machine learning: A comprehensive survey of methods, metrics, and applications. Future Internet 18(5), 234 (2026). https://doi.org/10.3390/fi18050234 45. Kan, Z., Pendlebury, F., Pierazzi, F., Cavallaro, L.: Investigating labelless drift adaptation for malware detection. In: Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec ’21), pp. 123–134. ACM (2021). https://doi.org/10.1145/ 3474369.3486873 46. Karimi, M.R., Gürel, N.M., Karlaš, B., Rausch, J., Zhang, C., Krause, A.: Online active model selection for pre-trained classifiers. In: Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, vol. 130, pp. 307–315 (2021) 47. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., GrabskaBarwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017). https://doi.org/10.1073/pnas.1611835114 48. Komorniczak, J., Zyblewski, P., Ksieniewicz, P.: Statistical drift detection ensemble for batch processing of data streams. KnowledgeBased Systems 252, 109380 (2022). https://doi.org/10.1016/j. knosys.2022.109380 49. Kossen, J., Farquhar, S., Gal, Y., Rainforth, T.: Active testing: Sample-efficient model evaluation. In: Proceedings of the 38th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 139, pp. 5753–5763 (2021) 50. Krawczyk, B., Cano, A.: Adaptive ensemble active learning for drifting data stream mining. In: Proceedings of the Twenty-Eighth
28 International Joint Conference on Artificial Intelligence (IJCAI), pp. 2763–2771 (2019). https://doi.org/10.24963/ijcai.2019/383 51. Kreuzberger, D., Kühl, N., Hirschl, S.: Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access 11, 31866–31879 (2023). https://doi.org/10.1109/ACCESS.2023. 3262138 52. Lanvin, M., Gimenez, P.F., Han, Y., Majorczyk, F., Mé, L., Totel, É.: Errors in the CICIDS2017 dataset and the significant differences in detection performances it makes. In: Risks and Security of Internet and Systems: 17th International Conference, CRiSIS 2022, Revised Selected Papers, Lecture Notes in Computer Science, vol. 13857, pp. 18–33. Springer (2023). https: //doi.org/10.1007/978-3-031-31108-6_2 53. Lin, J.: Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 37(1), 145–151 (1991). https: //doi.org/10.1109/18.61115 54. Liu, W., Zhang, H., Ding, Z., Liu, Q., Zhu, C.: A comprehensive active learning method for multiclass imbalanced data streams with concept drift. Knowledge-Based Systems 215, 106778 (2021). https://doi.org/10.1016/j.knosys.2021.106778 55. Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G.: Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering 31(12), 2346–2363 (2019). https://doi.org/ 10.1109/TKDE.2018.2876857 56. Lukats, D., Zielinski, O., Hahn, A., Stahl, F.: A benchmark and survey of fully unsupervised concept drift detectors on real-world data streams. International Journal of Data Science and Analytics (2024). https://doi.org/10.1007/s41060-024-00620-y 57. Mahadevan, A., Mathioudakis, M.: Cost-aware retraining for machine learning. Knowledge-Based Systems 293, 111610 (2024). https://doi.org/10.1016/j.knosys.2024.111610 58. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press (1989) 59. McNemar, Q.: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153– 157 (1947). https://doi.org/10.1007/BF02295996 60. Moustafa, N., Slay, J.: UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In: 2015 Military Communications and Information Systems Conference (MilCIS) (2015). https://doi.org/10.1109/MilCIS.2015. 7348942 61. Nath, S.V.: Champion-challenger based predictive model selection. In: Proceedings of IEEE SoutheastCon (2007). https://doi.org/10. 1109/SECON.2007.342897 62. Nguyen, H., Kashef, R.: TS-IDS: Traffic-aware self-supervised learning for IoT network intrusion detection. Knowledge-Based Systems 279, 110966 (2023). https://doi.org/10.1016/j.knosys. 2023.110966 63. Nicesio, O.K., Leal, A.G., Gava, V.L.: Quantum machine learning for network intrusion detection systems, a systematic literature review. In: 2023 IEEE 2nd International Conference on AI in Cybersecurity (ICAIC) (2023). https://doi.org/10.1109/ICAIC57335. 2023.10044125 64. Okanovic, P., Kirsch, A., Kasper, J., Hoefler, T., Krause, A., Gürel, N.M.: All models are wrong, some are useful: Model selection with limited labels. In: Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, vol. 258, pp. 2035–2043 (2025) 65. Pederzoli, S., Paganelli, M., Contalbo, M.L., Benassi, R., Tiano, D., Iannucci, S., Guerra, F.: NOCTOWL: Adaptive tree-based model for network anomaly detection under delayed and sampled label availability. IEEE Access 13, 197899–197911 (2025). https://doi. org/10.1109/ACCESS.2025.3633419
R. Fernández-Barrios et al. 66. Pendlebury, F., Pierazzi, F., Jordaney, R., Kinder, J., Cavallaro, L.: TESSERACT: Eliminating experimental bias in malware classification across space and time. In: 28th USENIX Security Symposium (USENIX Security 19), pp. 729–746 (2019) 67. Raab, C., Heusinger, M., Schleif, F.M.: Reactive soft prototype computing for concept drift streams. Neurocomputing 416, 340– 351 (2020). https://doi.org/10.1016/j.neucom.2019.11.111 68. Rabanser, S., Günnemann, S., Lipton, Z.C.: Failing loudly: An empirical study of methods for detecting dataset shift. In: Advances in Neural Information Processing Systems (NeurIPS) (2019). ArXiv:1810.11953 69. Regol, F., Schwinn, L., Sprague, K., Coates, M., Markovich, T.: When to retrain a machine learning model. In: Proceedings of the 42nd International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 267, pp. 51369– 51404 (2025) 70. Rizzo, M.L., Székely, G.J.: Energy distance. WIREs Computational Statistics 8(1), 27–38 (2016). https://doi.org/10.1002/wics.1375 71. Rosenfeld, E., Garg, S.: (Almost) provable error bounds under distribution shift via disagreement discrepancy. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 28761–28784 (2023). URL https://papers.nips.cc/paper_files/paper/2023/hash/ 5bacb12bf81e98e2ee0eed953a23c656-Abstract-Conference.html 72. Roshan, S., Miche, Y., Akusok, A., Lendasse, A.: Adaptive and online network intrusion detection system using clustering and extreme learning machines. Journal of the Franklin Institute 355(4), 1752–1779 (2018). https://doi.org/10.1016/j.jfranklin.2017.06.006 73. Sawade, C., Landwehr, N., Scheffer, T.: Active comparison of prediction models. In: Advances in Neural Information Processing Systems 25 (NeurIPS), pp. 1763–1771 (2012) 74. Schnabel, J., Roth, M.: Quantum kernel methods under scrutiny: A benchmarking study. Quantum Machine Intelligence 7(1), 58 (2025). https://doi.org/10.1007/s42484-025-00273-5 75. Schuirmann, D.J.: A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics 15(6), 657–680 (1987). https://doi.org/10.1007/ BF01068419 76. Schuld, M., Killoran, N.: Quantum machine learning in feature hilbert spaces. Physical Review Letters 122, 040504 (2019). https: //doi.org/10.1103/PhysRevLett.122.040504 77. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.F., Dennison, D.: Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems (NeurIPS) (2015) 78. Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A.: Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP), pp. 108–116 (2018) 79. Shyaa, M.A., Ibrahim, N.F., Zainol, Z., Abdullah, R., Anbar, M., Alzubaidi, L.: Evolving cybersecurity frontiers: A comprehensive survey on concept drift and feature dynamics aware machine and deep learning in intrusion detection systems. Engineering Applications of Artificial Intelligence 137, 109143 (2024). https://doi.org/10.1016/j.engappai.2024.109143 80. Shyaa, M.A., Ibrahim, N.F., Zainol, Z.B., Abdullah, R., Anbar, M., Alzubaidi, L.: IGPC-MSOS: A knowledge-preserving transfer learning framework with dynamic mode-switching for handling concept drift in network intrusion detection systems. KnowledgeBased Systems 337, 115361 (2026). https://doi.org/10.1016/j. knosys.2026.115361 81. Sommer, R., Paxson, V.: Outside the closed world: On using machine learning for network intrusion detection. In: 2010 IEEE Symposium on Security and Privacy (S&P), pp. 305–316. IEEE (2010). https://doi.org/10.1109/SP.2010.25
Candidate Comparability Before Promotion 82. Székely, G.J., Rizzo, M.L.: Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference 143(8), 1249–1272 (2013). https://doi.org/10.1016/j.jspi.2013.03. 018 83. Thanasilp, S., Wang, S., Cerezo, M., Holmes, Z.: Exponential concentration in quantum kernel methods. Nature Communications 15, 5200 (2024). https://doi.org/10.1038/s41467-024-49287-w 84. Waudby-Smith, I., Ramdas, A.: Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology 86(1), 1–27 (2024). https://doi.org/10.1093/jrsssb/qkad009 85. Yang, L., Guo, W., Hao, Q., Ciptadi, A., Ahmadzadeh, A., Xing, X., Wang, G.: CADE: Detecting and explaining concept drift samples for security applications. In: 30th USENIX Security Symposium (USENIX Security 21), pp. 2327–2344 (2021) 86. Yuan, J., Yang, Y., Wang, M.: ADAWU-IDS: A dynamic adaptive weight update mechanism for concept drift-aware network intrusion detection systems. Journal of King Saud University – Computer and Information Sciences 38, 546 (2026). https://doi.org/10.1007/ s44443-026-00964-4 87. Zhang, Q., Imran, A., Bardhi, E., Swamy, T., Zhang, N., Shahbaz, M., Olukotun, K.: CARAVAN: Practical online learning of in-network ML models with labeling agents. In: 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 325–345. USENIX Association (2024). URL https: //www.usenix.org/conference/osdi24/presentation/zhang-qizheng 88. Zhang, W., Deng, L., Zhang, L., Wu, D.: A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica 10(2), 305–329 (2023). https://doi.org/10.1109/JAS.2022.106004 89. Zhang, X., Zhao, R., Jiang, Z., Chen, H., Ding, Y., Ngai, E.C.H., Yang, S.H.: Continual learning with strategic selection and forgetting for network intrusion detection. In: IEEE INFOCOM 2025 – IEEE Conference on Computer Communications, pp. 1–10. IEEE (2025). https://doi.org/10.1109/INFOCOM55648.2025.11044615 90. Žliobaitė, I., Budka, M., Stahl, F.: Towards cost-sensitive adaptation: When is it worth updating your predictive model? Neurocomputing 150, 240–249 (2015). https://doi.org/10.1016/j.neucom. 2014.05.084 91. Zoghi, Z., Serpen, G.: UNSW-NB15 computer security dataset: Analysis through visualization. Security and Privacy 7(1), e331 (2024). https://doi.org/10.1002/spy2.331
29