Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT\&CK-Aligned Triage as a Worked Instance

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

1

Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT&CK-Aligned Triage as a Worked Instance

arXiv:2608.12444v1 [cs.CR] 12 Aug 2026

Zhenpeng Li

containment procedures, waste analyst time on the wrong investigation track, or mask a genuine threat behind a benignlooking attribution. Existing approaches to LLM-based IDS offer no formal control over the misattribution rate. Practitioners either accept all model outputs at face value (risking misattribution) or apply ad hoc confidence thresholds (which require manual tuning and lack guarantees). Conformal risk control (CRC) [3] offers a principled alternative: wrap the classifier with a calibrationbased abstention layer so that alerts whose nonconformity scores exceed a data-driven threshold are deferred, and the probability of a wrong automated attribution among the rest is controlled at a user-specified level α. This risk statement, however, is weaker than it looks. It is an unconditional bound: R ≤ α holds trivially if the system abstains on everything. We initially observed exactly this failure mode in one training run of an ML+CRC baseline, where a LightGBM classifier satisfied its FAR target by never automating a single alert; a subsequent stability check (Section VII-D0d) shows this specific instance does not reproduce across retrainings, but the underlying vacuity is a property Index Terms—Intrusion detection, MITRE ATT&CK, Confor- of the risk bound itself, not of that one run, and is what mal prediction, Large language models, Risk control, Alert triage motivates the theory developed next. This is not a bug in one classifier’s calibration; it is a structural consequence of what an unconditional risk bound can and cannot certify, one that applies to conformal risk control wherever it is used to I. I NTRODUCTION gate automation, not only in this triage setting. We show that Modern Security Operations Centers (SOCs) process thou- risk can be driven down through exactly two mechanisms that sands of intrusion detection system (IDS) alerts daily. A have nothing to do with predictive competence—abstaining critical step in incident response is attributing each alert more (shrinking the set of inputs the system is responsible to a specific attack technique within the MITRE ATT&CK for) and reporting at a coarser semantic resolution (redefining framework, which determines the appropriate containment and some errors as correct)—and that these two mechanisms admit remediation actions. Manual attribution is accurate but does a common formal treatment. not scale: the median SOC analyst handles 20–30 alerts per Our contributions are: hour [1], while enterprise IDS deployments generate orders of 1) A decision-contract theory of risk-certificate degenmagnitude more. eracy. We formalize a decision contract as a selectorLarge language models (LLMs) have recently demonstrated plus-correctness-relation pair, prove an error-conservation strong performance on network traffic classification tasks [2], law under which a base classifier’s error is only ever raising the prospect of automated ATT&CK-aligned alert reassigned among harmful automation, human deferral, and triage. However, an LLM classifier can still return a confident semantic masking (Theorem 3), and derive an exact riskbut wrong technique label. In the SOC context, a misattributed transfer identity for semantic coarsening together with an alert is worse than an unattributed one: it can trigger incorrect impossibility result for its reverse direction (Theorems 4– 5). Zhenpeng Li is with Guangzhou Health Science College, No. 248 Guangyuan Middle Road, Guangzhou, Guangdong 510405, China (e-mail: 2) Singleton capacity: a label-free deployability limit, and 2025301001@gzws.edu.cn). Corresponding author: Zhenpeng Li. its risk-feasible refinement. For the singleton-set deci0000–0000/00$00.00 © sion 2021 rule IEEE used throughout the paper, we derive an exact Abstract—An unconditional risk bound on automated decisions can be satisfied without automating anything, since a selector that never acts drives the bound to zero. We show this is structural: any risk certificate is defined over a decision contract, the inputs a system acts on plus the semantic relation under which an output counts correct, and weakening either hides base-classifier error. We develop a decision-contract theory: an error-conservation law showing error is only reassigned among harmful automation, human deferral, and semantic masking; a label-free singleton capacity certifying structural incapacity, with a risk-feasible refinement separating recoverable threshold misalignment from risk-constrained incapacity; and a non-degenerate actionability certificate excluding all-abstain solutions by construction. We instantiate this on ATT&CK-aligned alert triage for LLM-based intrusion detection, the setting that exposed the vacuity failure. Across 3 IDS datasets, 6 LLMs, and 4 error-rate thresholds, empirical false-attribution risk stays at or below target in 90.3% of configurations, with 83.4% mean correct automation. The capacity diagnostic explains every low-utility configuration; its refinement separates genuine misalignment from risk-constrained incapacity, confirmed by an exhibited alternative threshold; a training-stability re-run finds no confirmed structural-incapacity instance; and real fine-grained attack-subtype labels confirm the coarsening-transfer identity under a genuine many-to-one map, with small but non-zero masking mass.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

geometric characterization of automation (Proposition 2) and a resulting capacity κ(f ) that upper-bounds the action rate attainable by any global threshold (Theorem 6), with a finite-sample, unlabeled certificate (Proposition 3) and a formal counterexample ruling out score-variancebased diagnostics (Proposition 4). κ(f ) is necessary but not sufficient to diagnose a shortfall as recoverable: we introduce the risk-feasible capacity κα (f ) (Definition 4), which requires labels but correctly separates threshold misalignment from a risk-constrained limit that κ(f ) alone cannot see. 3) A non-degenerate actionability certificate. We define (α, ρ)-actionability, which jointly bounds harmful risk and floors the action rate, and show it converts an unconditional risk bound into both a correct-automation lower bound and a conditional-error upper bound (Theorem 8)—a guarantee the risk bound alone cannot provide (Proposition 1). Unlike the risk side, the action-rate floor is not certified by exchangeability alone; we close this with a finite-sample Hoeffding lower bound on the deployed threshold’s action rate (Proposition 5). This makes the certificate two-sided but not symmetric: the risk side keeps the standard marginal conformal guarantee of Theorem 1, and the action-rate side receives a genuine (1 − δ) finite-sample lower confidence bound for the already-realized τ̂ —a different probabilistic statement, not one certified bound paired with an unqualified point estimate. 4) Instantiation on ATT&CK-aligned triage. We formalize ATT&CK-aligned alert triage as a selective prediction problem and evaluate a CRC wrapper across 6 LLMs (7B–32B), 3 IDS benchmark datasets, and 4 error-rate thresholds. At the representative deployment floor ρ = 0.5 used throughout, the singleton-capacity diagnostic rules out structural incapacity for every LLM configuration (all 18, Table V), and the risk-feasible refinement κα (f ) (Definition 4) further separates the three lowest-utility cases into two that are risk-constrained incapacity and one genuine, recoverable threshold misalignment—a distinction κ(f ) alone gets wrong for two of the three; a 20-seed trainingstability re-run of an ML+CRC baseline finds no confirmed instance of true structural incapacity, which we report as an open empirical question rather than a confirmed case (Section VII-D0d). II. R ELATED W ORK A. ATT&CK Technique Mapping MITRE ATT&CK provides a taxonomy of adversarial tactics and techniques observed in real-world intrusions. Several systems map unstructured cyber threat intelligence (CTI) text to ATT&CK techniques: rcATT [4] uses TF-IDF with multilabel classification, TRAM fine-tunes BERT-based models on CTI reports. These systems operate on textual input (threat reports, CVE descriptions) and solve a text classification problem. Our work addresses a complementary task: mapping structured IDS alerts (network flow features) to ATT&CK techniques. The input modality, inference pipeline, and label

2

granularity differ fundamentally from CTI text mapping, and the two lines of work are not directly comparable. What our approach adds is not a better mapping method, but a riskcontrolled decision layer that quantifies and bounds attribution uncertainty. B. Conformal Prediction for Safety-Critical Systems Conformal prediction [5] constructs prediction sets with finite-sample coverage guarantees under exchangeability. Angelopoulos et al. [3] generalized this to conformal risk control (CRC), enabling control of arbitrary monotone loss functions—including the false discovery rate and selective prediction risk—and subsequent work extends the same calibration principle to general risk-controlling algorithms via learn-then-test procedures [6]. Applications in safety-critical domains include medical diagnosis [7], autonomous driving [8], and natural language generation [9]. In cybersecurity, conformal prediction has been applied to malware detection [10] and network anomaly detection [11], mostly for coverage-style guarantees. C. Selective Classification and the Object of Certification Selective classification [12], [13] equips a classifier with a reject option and characterizes the resulting risk–coverage trade-off; SelectiveNet [14] integrates the reject decision into training to optimize this trade-off directly, and one-sided formulations [15] give alternative selector constructions with their own coverage guarantees. Decision-theoretic treatments of selective labeling [16] independently arrive at a threshold policy as the optimal selector under an explicit cost model— a policy-optimization question our decision-contract framing does not address, since we take the selector as given by the conformal calibration procedure and ask instead what a certificate over that selector actually establishes. This literature already establishes that a risk bound alone is insufficient without a coverage (action-rate) guarantee alongside it—the same tension that motivates Proposition 1 and the actionability certificate of Section IV. Recent work on LLM outputs specifically shows that simple confidence statistics are unreliable selection signals [17], echoing Proposition 4’s formal argument that true-label score statistics do not determine singleton capacity, and that risk control can fail to certify what practitioners actually need from structured LLM generations [18], an impossibility-flavored result adjacent to but distinct from Theorem 5: theirs concerns what CRC can certify about structured outputs in general, ours concerns what a coarsening of a fixed selector’s outputs can certify about the selector’s finer-grained behavior. Selective risk control for LLM action policies is an active target for conformal methods more broadly [19]. Our contribution is not that risk and coverage both matter, which selective classification already shows; it is a generalization of what is being certified. The selective-classification line of work cited above studies the selector: when to abstain, at a fixed correctness relation. We generalize the certified object to a decision contract (g, E), in which both the selector and the

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

correctness relation can vary, and show that this second axis— semantic coarsening—hides base-classifier error by exactly the same accounting as abstention (Theorem 3). This is why a coverage guarantee at one label resolution does not, by itself, answer whether a report at a coarser resolution is trustworthy: Theorem 4 and its reverse-impossibility counterpart (Theorem 5) give an exact accounting and a limit for that question specifically, complementary to work on varying prediction specificity in a label hierarchy rather than a substitute for it. Class-conditional conformal prediction [20] addresses a related but distinct concern—per-class coverage can be masked by a favorable marginal average across many classes—and our within-fiber confusion mass Mϕ (g) (Theorem 4) is the analogous accounting object for masking induced by coarsening the label space itself rather than by aggregating coverage across existing classes. Conformal risk control [3] gives our risk side its finite-sample guarantee; selective classification’s risk– coverage lens gives our action-rate side its natural counterpart (Proposition 5); the decision-contract formalism is what places both, plus semantic coarsening, under a single certified object. D. LLMs for Network Intrusion Detection Recent work has explored LLMs as network intrusion detectors, serializing flow features into natural-language prompts for classification [2]. These approaches achieve competitive accuracy with traditional ML models (XGBoost, LightGBM) on standard benchmarks, but inherit the calibration challenges of LLMs: softmax probabilities are often poorly calibrated, and confident errors are common. Our framework is model-agnostic: it wraps any LLM-based (or ML-based) classifier and provides post-hoc FAR control without modifying the base model. This composability is a practical advantage, since the abstention layer can be deployed on top of existing LLM-IDS pipelines. III. P ROBLEM F ORMULATION A. Task Definition Let X denote the space of network flow feature vectors and T = {t1 , . . . , tK } denote a set of ATT&CK techniques. An IDS alert x ∈ X has a benchmark-derived, ATT&CK-aligned operational technique label y(x) ∈ T , obtained by mapping the dataset’s own attack-category annotation through ϕ (Section III-B); this is not an independently sourced ATT&CK ground-truth annotation, and we use y(x) throughout in this operational sense. A classifier f : X → ∆K produces a probability vector over techniques, where fk (x) estimates P (y = tk | x). We consider the selective prediction setting: the system either outputs a single technique attribution t̂(x) ∈ T , or abstains (t̂(x) = ⊥), deferring the alert to a human analyst. B. ATT&CK Label Mapping Standard IDS benchmark datasets use attack-category labels (e.g., DoS, Probe, Exploitation) rather than ATT&CK identifiers. We define a deterministic bijective mapping ϕ : Cattack → T from attack categories to ATT&CK techniques,

3

shown in Table I. Since ϕ is a bijection, the classification problem over Cattack and the attribution problem over T are equivalent: P (y = tk | x) = P (c = ϕ−1 (tk ) | x). This mapping is intentionally coarse-grained (4 techniques from 4 tactics), reflecting the label granularity available in existing IDS benchmarks. We discuss the implications of this granularity in Section VIII-D. C. Nonconformity Score Design We define the nonconformity score for sample x and candidate technique tk as: s(x, tk ) = 1 − P

fk (x)

j:tj ̸=Normal fj (x)

,

(1)

where the denominator normalizes over attack classes only (excluding the Normal/benign class). A low s(x, tk ) indicates high confidence that x corresponds to technique tk . The normalization in Eq. (1) serves two purposes: (i) it removes the influence of the Normal-class probability, which is irrelevant for technique attribution among known-attack alerts; and (ii) it improves cross-model comparability, since raw softmax magnitudes vary across LLM architectures. D. Risk Guarantee Definition 1 (False Automated Attribution Risk). For a selective classifier that outputs t̂(x) ∈ T ∪ {⊥}, we define:
FAR = P t̂(x) ̸= ⊥, t̂(x) ̸= y(x) . (2) FAR is therefore an unconditional risk over all attack alerts: it counts only alerts that are automatically attributed to the wrong technique. For diagnostics, we also report the conditional automated-attribution error, P (t̂(x) ̸= y(x) | t̂(x) ̸= ⊥), but this conditional quantity is not the formal target controlled by the conformal threshold. Theorem 1 (Conformal bound for wrong automated attribution). Let Dcal = {(xi , yi )}ni=1 be an exchangeable calibration set. Define the threshold: ( ) n 1 X q̂ = inf q : 1[s(xi , yi ) ≤ q] ≥ 1 − α , (3) n + 1 i=1 and the conformal prediction set: Γq̂ (x) = {tk ∈ T : s(x, tk ) ≤ q̂}.

(4)

The selective decision rule is: ( the unique element of Γq̂ (x) if |Γq̂ (x)| = 1, t̂(x) = (5) ⊥ otherwise. Then the probability of a wrong automated attribution is bounded by the conformal miscoverage risk:

P t̂(x) ̸= ⊥, t̂(x) ̸= y(x) ≤ P y(x) ∈ / Γq̂ (x) ≤ α, (6) up to the standard finite-sample quantile convention. The guarantee is distribution-free in the usual conformal sense: it requires no parametric assumptions on P (X, Y ), only exchangeability between calibration and test samples. The

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

4

TABLE I D ETERMINISTIC MAPPING FROM IDS ATTACK CATEGORIES TO ATT&CK TECHNIQUES . E ACH ATTACK CATEGORY MAPS TO EXACTLY ONE TECHNIQUE ( BIJECTION ), PRESERVING THE CLASSIFICATION PROBLEM STRUCTURE . IDS Label

ATT&CK Tactic

Technique ID

Mapping Basis

DoS CredentialAccess Exploitation Probe

Impact Credential Access Initial Access Discovery

T1498 T1110 T1190 T1046

Volumetric traffic patterns Authentication failure rate Exploit payload signatures Port scanning behavior

bound applies to the unconditional probability of a wrong automated attribution. It does not imply that every realized split has empirical FAR below α, nor does it directly bound the conditional error rate among non-abstained predictions. We therefore treat “empirical FAR ≤ α across all random seeds” as a stringent operational check, and report conditional automated-attribution error only as a diagnostic quantity. E. ATT&CK Hierarchy Guarantee ATT&CK organizes techniques under tactics (e.g., T1498 ⊂ Impact). Since our mapping ϕ assigns each technique to a unique tactic, correct technique attribution implies correct tactic attribution. Formally, if ψ : T → A maps techniques to tactics (a surjection in general, a bijection in our 4-class setting), then: t̂(x) = y(x) =⇒ ψ(t̂(x)) = ψ(y(x)).

(7)

Therefore, the FAR bound at the technique level automatically propagates to the tactic level. Theorem 1 and this hierarchy propagation are standard consequences of split conformal prediction, restated here for the ATT&CK attribution setting. They establish that FAR can be controlled; they do not establish that a controlled FAR is operationally meaningful. Section IV shows that an unconditional bound of this form can be satisfied without any automation at all, and develops the additional conditions under which a low FAR corresponds to genuine, non-degenerate triage. IV. D ECISION -C ONTRACT T HEORY FOR N ON -D EGENERATE R ISK C ERTIFICATION The central premise of this section is that a risk bound such as Theorem 1 is not, by itself, a property of a good predictor. It is a property of a decision contract: the set of alerts on which the system agrees to act, and the semantic relation under which an automated output is counted as correct. A system can reduce its measured FAR without improving its underlying predictive competence, either by abstaining more often or by reporting at a coarser semantic resolution. We formalize these two mechanisms, show that they are the only two mechanisms by which the base classifier’s fine-grained error can be hidden from the risk bound, and use the result to define a deployment certificate that cannot be satisfied by either mechanism alone. A. Decision contracts Let Y = T be the finest available label space (Section III) and let h : X → Y denote the top-one prediction of the base

classifier f . A selector g : X → {0, 1} determines whether the system acts: g(x) = 1 recovers t̂(x) ̸= ⊥, and g(x) = 0 recovers t̂(x) = ⊥. Correctness is represented by a reflexive acceptance relation E ⊆ Y × Y. The finest relation, matching Theorem 1, is Efine = {(y, y) : y ∈ Y}. A coarser contract is induced by the tactic map ψ of Section III: Eψ = {(y, y ′ ) : ψ(y) = ψ(y ′ )}. We use a relation rather than a function so that the same object accommodates ATT&CK structures in which a technique belongs to more than one tactic. Definition 2 (Decision contract). A decision contract is a pair C = (g, E). Its harmful automated risk, action rate, accepted utility, and conditional error are
R(C) = P g(X) = 1, (Y, h(X)) ∈ /E , (8)
A(C) = P g(X) = 1 , (9)
U (C) = P g(X) = 1, (Y, h(X)) ∈ E = A(C) − R(C), (10)
Q(C) = P (Y, h(X)) ∈ / E | g(X) = 1 , (11) where Q(C) is defined when A(C) > 0. Under E = Efine , R(C) is exactly the FAR of Theorem 1 and A(C) = 1 − AR. Proposition 1 (Risk factorization and vacuity). For every decision contract with A(C) > 0, R(C) = A(C) Q(C). Consequently, a bound R(C) ≤ α alone implies neither a positive action rate nor a small conditional error: g ≡ 0 gives R(C) = 0 trivially, and a contract with A(C) ≤ α may have Q(C) = 1 while still satisfying R(C) ≤ α. Proof. The identity is the product rule for probability; the two constructions instantiate it at the boundary A = 0 and at Q = 1. Proposition 1 is not a hypothetical concern. Section VII-D0d exhibits a real ML+CRC configuration in which A(C) = 0 and the FAR bound is satisfied by construction. B. Contract weakening and an error-conservation law Definition 3 (Contract order). C1 = (g1 , E1 ) ⪰ C2 = (g2 , E2 ) if g1 (x) ≥ g2 (x) almost everywhere and E1 ⊆ E2 : the stronger contract acts on at least as many alerts and accepts no more semantic errors as correct. Theorem 2 (Risk monotonicity under contract weakening). If C1 ⪰ C2 , then R(C2 ) ≤ R(C1 ), and the reduction decomposes exactly as
R(C1 ) − R(C2 ) = P g1 =1, g2 =0, (Y, h) ∈ / E1
+ P g2 =1, (Y, h) ∈ / E1 , (Y, h) ∈ E2 , (12)

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

where the first term is risk removed by additional deferral and the second is risk removed by semantic relaxation. Proof. Since g2 ≤ g1 and E1 ⊆ E2 , the pointwise difference of risk indicators equals (g1 − g2 )1{(Y, h) ∈ / E1 } + g2 1{(Y, h) ∈ / E1 , (Y, h) ∈ E2 }, which is non-negative; take expectations. A direct consequence is an accounting identity for the base classifier’s fine-grained error. Define B(h) = P(h(X) ̸= Y ), deferred error D(C) = P(g=0, h ̸= Y ), semantic masking M (C) = P(g=1, h ̸= Y, (Y, h) ∈ E), and genuine correct automation G(C) = P(g=1, h=Y ). Theorem 3 (Error conservation under a decision contract). For every reflexive E ⊇ Efine , B(h) = R(C)+D(C)+M (C),

U (C) = G(C)+M (C). (13)

Proof. On {h(X) ̸= Y }, exactly one of three disjoint events occurs: the system defers (D); the system acts and E still rejects the error (R); the system acts and E accepts the error (M ). Among acted-upon cases accepted by E, either h = Y or the case is a masked error, giving the second identity. Eq. (13) is the paper’s central accounting result: a fixed base classifier’s error B(h) does not disappear under a risk certificate, it is only reassigned among harmful automation, human deferral, and semantic masking. A reported reduction in risk, or a reported gain in utility from coarsening (e.g., techniqueto tactic-level reporting), is therefore only informative once M (C) is reported alongside it. C. Risk transfer under semantic coarsening Let ϕ : Y → Z be a deterministic, possibly many-to-one map, and fix the same selector g at both resolutions. Theorem 4 (Exact fine-to-coarse risk transfer). R (g) = R where Mϕ (g) = fine (g) − Mϕ (g), Pϕ P ′ P(g=1, Y =y, h(X)=y ) is the within-fiber z y̸=y ′ ∈ϕ−1 (z) confusion mass. Hence Rϕ (g) ≤ Rfine (g), with equality iff Mϕ (g) = 0. Proof. A fine-label error stops being an error after coarsening exactly when the true and predicted labels share a fiber of ϕ; subtract this disjoint mass from Rfine . Theorem 5 (Impossibility of reverse risk transfer). If some fiber of ϕ contains two distinct labels y ̸= y ′ , then Rϕ (g) = 0 does not imply any non-trivial bound on Rfine (g): taking g ≡ 1, Y = y a.s., h(X) = y ′ a.s. gives Rϕ (g) = 0 and Rfine (g) = 1. In our 4-technique, 4-tactic mapping (Table I), every fiber of ψ is a singleton, so Mψ (g) ≡ 0 identically: the techniqueto-tactic identity of Section III, and the empirical equality reported in Section VII-E, are the zero-masking boundary case of Theorem 4, not an independent finding. Theorem 5 is stated for the general many-to-one case relevant to the full ATT&CK matrix, where multiple techniques share a tactic; evaluating Mϕ (g) > 0 empirically requires technique-level labels finer than the 4-category benchmarks used here (Section VIII-D),

5

and we leave a direct empirical demonstration of non-trivial masking to future work on finer-grained taxonomies. D. Exact geometry of singleton automation P Write pk (x) = fk (x)/ j̸=Normal fj (x) so that s(x, tk ) = 1 − pk (x) (Eq. (1)) and τ = 1 − q̂. The prediction set is Γτ (x) = {k : pk (x) ≥ τ }. Let p(1) (x) ≥ p(2) (x) be the largest and second-largest attack-normalized probabilities. Proposition 2 (Singleton interval characterization). |Γτ (x)| = 1 ⇐⇒ p(2) (x) < τ ≤ p(1) (x). Hence every x defines a singleton interval I(x) = (p(2) (x), p(1) (x)], and a global threshold automates x iff τ ∈ I(x). Let Fj− (t) = P(p(j) (X) < t) for j ∈ {1, 2}. Theorem 6 (Singleton capacity). The global-threshold action rate is A(τ ) = F2− (τ ) − F1− (τ ). The singleton capacity
κ(f ) = sup F2− (τ ) − F1− (τ ) (14) τ ∈[0,1]

is the maximum action rate attainable by any global singleton threshold. If κ(f ) < ρ, no calibration method using a single global threshold can satisfy an action-rate requirement A ≥ ρ, regardless of α or calibration-set size. Proof. P(p(2) < τ ≤ p(1) ) = P(p(2) < τ ) − P(p(1) < τ ) since {p(1) < τ } ⊆ {p(2) < τ }; take the supremum over τ . κ(f ) is unconstrained by risk: it is the largest action rate reachable by some threshold, without regard to how many of the resulting singleton predictions are wrong. It therefore yields a one-directional diagnosis. If κ(f ) < ρ, no threshold— calibrated or otherwise—can reach the action-rate requirement A ≥ ρ at any risk level, and this is certified from the classifier’s probability output f alone, without labels: structural incapacity. The converse does not hold: κ(f ) ≥ ρ rules out this particular geometric obstruction, but it does not imply that a threshold exists which is simultaneously risk-feasible (R(τ ) ≤ α) and reaches ρ, because the action-maximizing τ in Eq. (14) need not be the same τ that keeps risk below α. Definition 4 (Risk-feasible capacity). κα (f ) = sup{A(τ ) : τ ∈ [0, 1], R(τ ) ≤ α}, the largest action rate attainable among thresholds that also satisfy the risk target. By definition κα (f ) ≤ κ(f ), and κ(f ) < ρ =⇒ κα (f ) < ρ, so structural incapacity certified via κ(f ) remains valid without reference to κα . Unlike κ(f ), κα (f ) requires labeled data to evaluate R(τ ) and is not a pre-deployment, label-free quantity. We use the two together to distinguish three regimes rather than two: structural incapacity (κ(f ) < ρ, no threshold at any risk level reaches ρ); risk-constrained incapacity (κ(f ) ≥ ρ but κα (f ) < ρ, the geometry admits a highaction threshold only at risk levels the target α forbids); and threshold misalignment (κα (f ) ≥ ρ but the calibrated τ̂ has A(τ̂ ) < ρ, a genuinely recoverable shortfall). The actionability gap AG(τ̂ ) = κ(f ) − A(τ̂ ) remains a useful label-free upper bound on how much automation could conceivably be at stake, but Section VIII-A shows it routinely overstates what is riskfeasible; κα (f ) − A(τ̂ ) is the quantity that actually determines whether recalibration can close the gap.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

Proposition 3 (Finite-sample capacity certificate). For an unlabeled readiness sample of size m, the empirical gap estimator κ b = supτ (Fb2− − Fb1− ) satisfies, by the DKW inequality and a union bound over the two CDFs, |b κ − κ(f )| ≤ p b plus this 2 log(4/δ)/(2m) with probability at least 1−δ. If κ margin falls below ρ, structural incapacity is certified without any test labels. This certificate applies to κ(f ) only; κα (f ) has no label-free analogue, since it is defined through the risk constraint R(τ ) ≤ α. Proposition 4 (True-label score statistics do not determine capacity). Neither the mean nor the variance of the truelabel nonconformity score, alone or combined with top-one accuracy, determines κ(f ). Proof. Let K = 3, Y = 1 a.s. Predictor A outputs (0.3, 0.7, 0) on every input; predictor B outputs (0.3, 0.35, 0.35). Both have true-label score 1 − 0.3 = 0.7 (equal mean, zero variance) and zero top-one accuracy. For A, τ ∈ (0.3, 0.7] singletons every input, so κ = 1. For B, the top two probabilities are tied for every input, so no τ yields a singleton and κ = 0. Proposition 4 is a formal argument against the heuristic used in an earlier draft of this analysis (nonconformity-score variance combined with base accuracy) to diagnose deployability; Section VIII-A replaces that heuristic with κ(f ) as a label-free necessary condition and κα (f ) (Definition 4) as the labeled quantity that determines whether a shortfall is actually recoverable by recalibration. E. Operational risk and surrogate slack Let e(X, Y ) = 1{h(X) ̸= Y }, gτ (X) = 1{p(2) (X) < τ ≤ p(1) (X)}, and C(τ ) = P(Y ∈ / Γτ (X)) = P(pY (X) < τ ) the miscoverage event controlled directly by Theorem 1. Theorem 7 (Surrogate-slack decomposition). For the singleton decision rule, C(τ ) = R(τ ) + S(τ ), where S(τ ) = P(gτ =0, Y ∈ / Γτ (X)) is the deferred-miscoverage mass. Proof. Partition the miscoverage event by gτ . When gτ = 1 the set is a singleton, so the true label is excluded iff the unique output is wrong, giving R(τ ); the remainder is S(τ ). Conformal miscoverage control is therefore a valid but generally conservative surrogate for wrong automated attribution: the conservatism is exactly S(τ ), the risk budget spent on alerts that are already deferred and hence operationally harmless. Section VII-D0b measures S(τ ) directly on the calibrated B2/B3 configurations. F. Non-degenerate actionability certificates Definition 5 ((α, ρ)-actionable contract). C = (g, Ereq ) is (α, ρ)-actionable if R(C) ≤ α and A(C) ≥ ρ for a required action-rate floor ρ > 0. Theorem 8 (Bridge from unconditional risk to useful automation). If C is (α, ρ)-actionable, then U (C) ≥ ρ − α and Q(C) ≤ α/ρ. Proof. U = A − R ≥ ρ − α directly; Q = R/A ≤ α/ρ by Proposition 1.

6

Theorem 8 excludes the all-abstain solution by construction: e.g., α = 0.05, ρ = 0.80 jointly certify at least 75% correct automation and at most 6.25% conditional error among automated outputs — a guarantee that an unconditional FAR bound alone cannot provide (Proposition 1). Definition 5 and Theorem 8 are population-level statements: R(C) ≤ α is certified at deployment by the conformal calibration of Theorem 1, but A(C) ≥ ρ as written requires knowing the true population action rate, which a practitioner does not have. Checking A(τ̂ ) ≥ ρ against a single realized test set, as an empirical measurement, is not itself a certificate: it carries no stated confidence level and can be violated by an unlucky sample. We close this gap with a finite-sample bound on the deployed threshold’s action rate, giving a certificate whose action-rate side is exchangeability-based in the same sense as its risk side. Proposition 5 (Finite-sample action-rate certificate). Let τ̂ be fixed independently of a certification sample {Xi }m i=1 b ) = drawn from the deployment distribution, and let A(τ̂ p P 1 log(1/δ)/(2m), the i 1{gτ̂ (Xi ) = 1}. For εm (δ) = m one-sided Hoeffding bound gives, with probability at least 1 − δ, b ) − εm (δ)}. A(τ̂ ) ≥ Aδ := max{0, A(τ̂

(15)

If, in addition, R(Cτ̂ ) ≤ α (Theorem 1’s standard marginal conformal guarantee) and Aδ ≥ ρ, then substituting Aδ for ρ in Theorem 8 gives U (Cτ̂ ) ≥ Aδ − α and Q(Cτ̂ ) ≤ α/Aδ on the probability-(1 − δ) event {A(τ̂ ) ≥ Aδ }. Proof. 1{gτ̂ (Xi ) = 1} are i.i.d. Bernoulli(A(τ̂ )) since τ̂ is fixed before the certification draw; the one-sided Hoeffding b ) − εm (δ)) ≤ δ, which inequality gives P(A(τ̂ ) < A(τ̂ rearranges to Eq. (15). Substituting Aδ for ρ in Theorem 8’s proof, which uses only A(C) ≥ ρ, gives the stated bounds on the same probability-(1 − δ) event. The two sides of the certificate carry genuinely different guarantees, and we do not combine them into a single confidence level. R(Cτ̂ ) ≤ α is the standard split-conformal marginal statement of Theorem 1: a probability over the joint draw of calibration set and test point, not a (1 − δ)-probability statement about the population risk of this specific realized τ̂ . A(τ̂ ) ≥ Aδ is a genuine finite-sample, (1−δ)-confidence lower bound for the fixed, already-realized τ̂ , since τ̂ is held constant while {Xi } is redrawn. Reporting both bounds together gives a practitioner two guarantees of different character, not one guarantee at a single stated level; a training-conditional or PAC-style high-probability version of the risk side, combined with δR + δA union-bounding, would be needed to state a single joint confidence level, which we leave for future work. Proposition 5 converts (α, ρ)-actionability from a definition that a realized action rate happens to satisfy into a certificate a practitioner can compute before trusting a deployment: fix τ̂ b ) on a disjoint certification on a calibration split, measure A(τ̂ split, and report Aδ rather than the raw empirical rate. We compute this bound for the paper’s 18 LLM configurations in Section VII-C0d.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

7

G. Testable predictions The framework yields pre-specified predictions checked against existing D2 data with no new inference runs, except where noted: P1 The observed automation rate equals Fb2− (τ ) − Fb1− (τ ) up to numerical ties (Theorem 6). P2 Utility(α) ≤ κ(f ) for every configuration and α. P3 Low-utility configurations are not structural incapacity (κ(f ) near zero); Definition 4’s risk-feasible capacity κα (f ), not the unconstrained κ(f ), determines whether the remaining shortfall is genuine threshold misalignment (κα (f ) ≥ ρ, recoverable by recalibration) or riskconstrained incapacity (κα (f ) < ρ, not recoverable at the given α); a genuine structural-incapacity instance, if one occurs in this benchmark suite, requires a dedicated persample re-run of the ML pipeline and a training-stability check to confirm it is not an artifact (Section VII-D0d). P4 C(τ ) − R(τ ) = S(τ ) ≥ 0 across all 18 LLM configurations, not only the single illustrative configuration of Section VII-D. P5 Under a genuinely many-to-one ϕ (real fine-grained attack subtypes collapsed onto the 4-category ATT&CK mapping), Rϕ (g) = Rfine (g) − Mϕ (g) holds exactly, with Mϕ (g) > 0 (Section IV-C). P6 Configurations whose certified action-rate lower bound Aδ (Proposition 5) meets a floor ρ, jointly with R ≤ α, satisfy U ≥ Aδ − α and Q ≤ α/Aδ ; configurations with low FAR but near-zero action rate fail the certificate by construction, and the certificate is now stated at a declared confidence level rather than against a single realized test set. P7 B(h) = R(C) + D(C) (the E = Efine special case of Theorem 3, where M (C) ≡ 0) holds exactly for every configuration and α. V. I NSTANTIATING THE D ECISION C ONTRACT ON ATT&CK-A LIGNED T RIAGE We now instantiate the framework of Section IV on the ATT&CK attribution task of Section III. The deployed decision contract is Cτ̂ = (gτ̂ , Efine ): the selector gτ̂ is the singleton rule of Proposition 2 at threshold τ̂ = 1 − q̂, and correctness is judged at the finest ATT&CK technique resolution (Section VII-E instantiates the coarser tactic-level contract Eψ as a comparison point). Calibrating q̂ (Algorithm 1) certifies R(Cτ̂ ) ≤ α by Theorem 1; Section VII additionally reports A(Cτ̂ ), κ(f ), the actionability gap, and the certificate of Proposition 5 for every configuration, rather than treating R(Cτ̂ ) ≤ α as sufficient on its own. A. CRC Calibration Procedure Figure 1 illustrates the calibration and inference pipeline that produces gτ̂ , and Algorithm 1 gives the procedure in full. The procedure requires only a held-out calibration set of labeled attack alerts; the base LLM classifier f is not retrained or fine-tuned.

IDS alert → LLM/ML classifier → attack-class probabilities → conformal set Γq̂ (x) |Γq̂ (x)| = 1: automated ATT&CK attribution |Γq̂ (x)| ̸= 1: defer to analyst Fig. 1. CRC-based ATT&CK alert triage framework. The abstention layer wraps the base classifier without modifying it. The system emits an automated ATT&CK attribution only when the conformal set contains a single technique; empty or multi-technique sets are deferred.

Algorithm 1 CRC-based ATT&CK Alert Triage Require: Calibration set Dcal = {(xi , yi )}ni=1 , target FAR α, classifier f Ensure: Threshold q̂ 1: Compute NC scores: si ← s(xi , yi ) for all i ∈ [n] // Eq. (1) 2: Sort: s(1) ≤ s(2) ≤ · · · ≤ s(n) 3: q̂ ← s(⌈(1−α)(n+1)⌉) 4: At test time, for each alert x: 5: Γq̂ (x) ← {tk ∈ T : s(x, tk ) ≤ q̂} 6: if |Γq̂ (x)| = 1 then 7: Output the unique technique in Γq̂ (x) 8: else 9: Abstain: t̂(x) = ⊥ // Defer to analyst 10: end if

B. Proxy Validity Our nonconformity score s(x, tk ) is derived from the LLM’s softmax probabilities over IDS attack categories, not from a model trained directly on ATT&CK labels. The validity of this proxy rests on the bijective mapping ϕ (Section III-B): since each attack category corresponds to exactly one ATT&CK technique, the ordering induced by fk (x) over attack categories is identical to the ordering over ATT&CK techniques. The conformal risk statement does not require calibrated probabilities, but utility depends strongly on whether the score ranks plausible techniques ahead of implausible ones—this ranking quality is exactly what κ(f ) (Theorem 6) measures directly from p(1) (x) and p(2) (x), so we do not additionally require a separate calibration diagnostic (Section VIII-D reports one supplementary check). C. Practical Considerations a) Calibration set size.: The CRC threshold q̂ depends on the calibration set size n. Small n yields conservative thresholds (high abstention rate), while large n gives tighter thresholds. We investigate the n–utility tradeoff in Section VII-G. b) Model selection.: CRC is a post-hoc wrapper: it does not constrain the choice of base classifier. A model whose top-1 and top-2 attack-class probabilities are well-separated on most inputs (high singleton capacity κ(f ), Theorem 6) admits lower abstention rates for the same risk target, but the risk bound itself does not require this separation. A model with poor separation simply triggers more abstentions, which is the correct conservative behavior—and, if κ(f ) is itself

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

8

TABLE II DATASET CHARACTERISTICS . O NLY ATTACK - CLASS SAMPLES ARE USED FOR ATT&CK ATTRIBUTION . Dataset CIC-IDS-2018 [21] HIKARI-2021 [22] RT-IoT2022 [23]

Attack Classes

ATT&CK Techniques

Test Samples

3 2 3

T1498, T1110, T1190 T1110, T1190 T1498, T1190, T1046

1,488 226 1,779

low, no calibration choice can recover a higher action rate (Section VIII-A). VI. E XPERIMENTAL S ETUP A. Datasets We use three IDS benchmark datasets spanning different network environments and attack types (Table II). Only attackclass samples are used (Normal traffic is excluded), since ATT&CK attribution applies only to detected threats.

Automation rate: 1−AR, the fraction of alerts receiving any automated singleton attribution, whether correct or incorrect. • Conditional error: Fraction of automated singleton attributions that are wrong. This diagnostic metric is reported to characterize model behavior, but it is not the formal conformal risk target. • Coverage: Fraction of alerts where the true technique is in the prediction set (for multi-label evaluation). •

F. Configuration B. LLM Backbones We evaluate six open-source LLMs spanning three model families and parameter scales: Gemma-2 9B [24], LLaMA3 8B [25], Mistral 7B [26], Qwen-3 8B, Qwen-3 14B, and Qwen-3 32B [27]. All models perform zero-shot IDS classification: network flow features are serialized into naturallanguage prompts, and the model outputs logit-based probabilities over attack categories. No fine-tuning is performed.

All LLM+CRC experiments use ncal = 200 calibration samples (stratified by class), the remaining attack samples for testing, 5 random seeds for calibration/test splits, and α ∈ {0.01, 0.05, 0.10, 0.20}. The primary evaluation point is α = 0.05. All computations are CPU-only post-processing of pre-computed LLM logit outputs. VII. R ESULTS A. RQ1: Does CRC Guarantee FAR ≤ α?

C. ML Baselines To evaluate whether the CRC framework benefits from LLM backbones over traditional classifiers, we include two ML baselines with the same CRC abstention layer: XGBoost [28] and LightGBM [29], trained on the same attackclass features with predict_proba providing the softmaxequivalent probability vector. D. Baselines B1-Argmax: The LLM’s top-1 prediction with no abstention. FAR equals the model’s misclassification rate. • B2-Threshold: Abstain when maxk fk (x) < τ for a handtuned threshold τ . We sweep τ to find the best FAR–utility tradeoff. • B3-CRC (ours): CRC-calibrated abstention with a formal false-attribution risk target. • B4-ML-CRC: XGBoost or LightGBM with the same CRC abstention framework, using identical NC score design and calibration procedure. •

E. Evaluation Metrics FAR (False Automated Attribution Risk): Fraction of all attack alerts that are automatically attributed to the wrong technique. • AR (Abstention Rate): Fraction of alerts deferred to human analysts. • Utility: Fraction of all attack alerts that receive a correct automated attribution. •

Table III summarizes the empirical false automated attribution risk across all configurations. We apply a stringent operational criterion—FAR ≤ α across all 5 random seeds— which is stricter than the marginal risk statement itself. Under this criterion, 65 of 72 configurations pass (90.3%). The 7 exceedances are concentrated in four model–dataset pairs: CIC-IDS-2018 × Mistral-7B (2 exceedances at α ∈ {0.10, 0.20}), CIC-IDS-2018 × Qwen-3 14B (1 at α = 0.05), HIKARI-2021 × Gemma-2 9B (2 at α ∈ {0.05, 0.20}), and RT-IoT2022 × Gemma-2 9B / Qwen-3 14B (2 combined). In these cases the mean FAR is often below α while at least one seed exceeds it, consistent with finite-sample variability under a stringent per-seed check. We diagnose these exceedances directly in Section VII-C using the singleton-capacity framework of Section IV, which replaces an earlier nonconformity-scorevariance heuristic that Proposition 4 shows is not sufficient in general. B. RQ2: How Much Utility Does CRC Retain? Table IV reports per-configuration results at α = 0.05. Across 18 configurations, the mean utility is 0.834, meaning that 83.4% of attack alerts receive a correct automated technique attribution. The mean automation rate, which also includes wrong singleton attributions, is 0.850. Among the 12 configurations with utility above 0.90, FAR is simultaneously held below α = 0.05, showing that high correct automation and low false-attribution risk are jointly achievable when the base classifier’s score geometry favors singleton predictions.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

9

TABLE III CRC EMPIRICAL VERIFICATION ACROSS 18 MODEL – DATASET CONFIGURATIONS AND 4 α LEVELS . FAR IS THE FALSE AUTOMATED ATTRIBUTION RISK OVER ALL ATTACK ALERTS . “PASS ” INDICATES EMPIRICAL FAR ≤ α ACROSS ALL 5 RANDOM SEEDS . ± DENOTES STANDARD DEVIATION OVER SEEDS . α

FAR (mean ± std)

Utility (mean)

Coverage (mean)

Pass Rate

0.01 0.05 0.10 0.20

0.002 ± 0.003 0.015 ± 0.014 0.025 ± 0.031 0.040 ± 0.058

0.704 0.834 0.813 0.745

0.995 0.960 0.910 0.820

17/18 16/18 16/18 16/18 65/72

Overall

TABLE IV P ER - CONFIGURATION FAR, U TILITY, AND C OVERAGE AT α = 0.05. VALUES ARE MEANS ± STD OVER 5 RANDOM SEEDS . B OLD INDICATES U TILITY ≥ 0.90. Dataset

Model

FAR (mean ± std)

Utility

Coverage

CIC-IDS-2018

Gemma-2 9B LLaMA-3 8B Mistral 7B Qwen-3 8B Qwen-3 14B Qwen-3 32B

0.035 ± 0.002 0.031 ± 0.005 0.004 ± 0.001 0.024 ± 0.006 0.049 ± 0.014 0.033 ± 0.001

0.516 0.961 0.378 0.952 0.838 0.954

0.954 0.961 0.996 0.952 0.950 0.954

HIKARI-2021

Gemma-2 9B LLaMA-3 8B Mistral 7B Qwen-3 8B Qwen-3 14B Qwen-3 32B

0.023 ± 0.031 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000

0.831 0.985 0.985 0.946 0.962 0.946

0.977 0.985 0.985 0.946 0.962 0.946

RT-IoT2022

Gemma-2 9B LLaMA-3 8B Mistral 7B Qwen-3 8B Qwen-3 14B Qwen-3 32B

0.036 ± 0.004 0.001 ± 0.000 0.000 ± 0.000 0.003 ± 0.003 0.029 ± 0.008 0.006 ± 0.002

0.063 0.959 0.954 0.959 0.904 0.927

0.960 0.959 0.954 0.959 0.951 0.952

The three low-utility outliers—CIC × Mistral (0.378), CIC × Gemma (0.516), and RT-IoT × Gemma (0.063)— correspond to models whose base classification accuracy or score separation is weak on those datasets. CRC responds by increasing abstention, reducing the number of wrong automated attributions but also reducing usable automation. This behavior is operationally conservative, but it is not a substitute for choosing a competent base model. The best-performing configurations on HIKARI achieve 98.5% utility with FAR = 0, so the calibrated layer abstains on only 1.5% of alerts. This suggests that the abstention layer has low overhead when the base model already separates the attack classes well. C. RQ2.5: Verifying the Decision-Contract Predictions We test predictions P1–P7 of Section IV-G against the existing per-sample logit data for all 18 LLM configurations, with no new inference. For each configuration, p(1) (x) and p(2) (x) are recovered from the stored attack-normalized nonconformity scores, and κ̂ is computed as the empirical CDF-gap supremum of Eq. (14). P1, P2, P4, and P7 check exact accounting identities (Propositions 2 and 1, Theorems 6, 3, and 7) that must hold by construction on any correctly computed data; we report them briefly, as an implementation check, before turning to the non-trivial empirical questions— P3, P5, and P6—that the identities alone do not answer:

does the capacity bound explain the observed low-utility configurations, how large is semantic masking in practice, and does a genuine structural-incapacity instance exist in this benchmark suite. a) P1, P2, P4, P7 (accounting identities).: All four hold with zero violations across the 18-configuration, 4-α matrix (difference < 10−9 throughout): the realized automation rate matches the CDF-gap prediction Fb2− (τ ) − Fb1− (τ ) exactly (P1, Proposition 2); Utility(α) ≤ κ̂ (P2, Theorem 6); S(τ ) = C(τ ) − R(τ ) ≥ 0 and is strictly positive wherever any deferral occurs, extending the single-configuration check of Table VII (P4, Theorem 7); and, at the finest label resolution used throughout this paper where M (C) ≡ 0, B(h) = R(C) + D(C) (P7, the E = Efine special case of Theorem 3, with the general M (C) > 0 case reported as P5 below). b) P5 (semantic masking under genuine coarsening).: The 4-technique ATT&CK mapping used elsewhere in this paper (Table I) is bijective at both the technique and tactic level, so Mϕ (g) ≡ 0 in every experiment reported so far— the identity of Theorem 4 is verified only in its trivial boundary case up to this point. To test the identity under a genuinely many-to-one map, we exploit real (non-synthetic) fine-grained attack subtypes present in the raw CIC-IDS2018 labels before the dataset-specific loader collapses them into the four coarse categories: the DoS category aggre-

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

gates 7 raw subtypes (e.g., DDoS-HOIC, DoS-Hulk, DoSSlowHTTPTest), CredentialAccess aggregates 4 (e.g., FTPBruteForce, SSH-Bruteforce), and Exploitation aggregates 3 (Bot, SQL Injection, Infilteration)—the coarse-to-fine map used by the benchmark curators themselves, not a construction of this paper. We train an XGBoost classifier at this fine resolution (the sampled training data realizes 8 of the 14 raw attack subtypes, including a genuine 4-element DoS fiber: GoldenEye, Hulk, SlowHTTPTest, Slowloris), apply the singleton CRC rule to obtain a fixed selector g, and measure Rfine (g), Rϕ (g), and Mϕ (g) directly under the coarsening map ϕ back to the 4-category taxonomy, on 47,425 held-out attack alerts. The identity Rϕ (g) = Rfine (g) − Mϕ (g) holds exactly at all 4 α levels (difference < 10−9 in every case), confirming Theorem 4 under a real, non-synthetic many-to-one map rather than only its trivial single-fiber boundary case. The realized masking mass is small in this instance—Mϕ (g) = 2.1 × 10−5 at α = 0.05 (the only α level with a non-zero fine-grained error among automated alerts) and Mϕ (g) ≈ 0 elsewhere— because this classifier’s rare fine-grained confusions occur predominantly across coarse categories rather than within the same fiber. This does not weaken the theorem: Theorem 4 is an exact accounting identity regardless of the masking magnitude, and Theorem 5 remains a worst-case existence result rather than a claim about typical deployments. The practical reading is that coarsening-induced risk-hiding is a real, exactly quantifiable phenomenon whose magnitude is classifier- and taxonomy-dependent, and Theorem 4 gives the tool to measure it directly rather than assume it is negligible. c) Finite-sample capacity certificate at readiness-sample size.: Proposition 3 is only useful if its DKW margin is tight enough, at a realistic unlabeled readiness-sample size, to distinguish structural incapacity from threshold misalignment before deployment—the question a pre-deployment practitioner actually faces, not whether the bound holds in the infinite-sample limit. We test this directly: for each of the 18 configurations, we draw m = 200 (matching the paper’s calibration-set size, Section VI) unlabeled subsamples from the full test population 500 times, compute κ̂ on each subsample, and check whether it falls within the claimed margin 2εm (δ) of the population κ(f ) at δ = 0.1. Across all 18 × 500 = 9,000 trials, the margin holds in every case (pooled coverage 9,000/9,000), against the ≥ 90% nominal target—the DKW bound is conservative here, as expected, but not so loose as to be uninformative: at m = 200 the margin is 2ε200 (0.1) ≈ 0.192, tight enough to certify structural incapacity (κ̂ + 2εm < ρ) whenever the true κ(f ) is more than about 0.19 below the action-rate floor ρ. For the smallest test population in this study (HIKARI-2021, Table V), the achievable margin is correspondingly wider, illustrating that the certificate’s practical bite depends on the available readiness-sample size, not only on δ. d) P3 (structural incapacity, risk-constrained incapacity, or threshold misalignment).: Table V reports κ̂, the riskfeasible capacity κ̂α (Definition 4, α = 0.05), and A(τ̂ ), all computed on one fixed 200-sample calibration / held-out test split (seed 42, matching Section VI), with κ̂ and κ̂α evaluated by exact enumeration of the singleton-interval breakpoints of

10

Proposition 2 together with τ̂ itself, rather than a uniform grid: since τ̂ is always a feasible candidate in κ̂α ’s supremum when R(τ̂ ) ≤ α on this split, this construction guarantees κ̂α ≥ A(τ̂ ) by definition, with no discretization gap. These single-split values differ slightly from Table IV’s 5-seed means (e.g. CIC × Gemma-2: 0.552 here vs. 0.516 there); we use one fixed split throughout this diagnostic so that κ̂, κ̂α , and A(τ̂ ) are directly comparable to each other. κ̂α itself is a post-hoc oracle diagnostic, not a deployable decision rule: it uses test labels to characterize whether a better threshold exists at all, the same question Section VII-D correctly refuses to let B2Threshold answer by tuning τ on held-out labels. Concluding that a specific configuration is fixable in practice additionally requires exhibiting a threshold selected without test labels; we do this separately for the one case where it matters, below. Every configuration in the LLM matrix has κ̂ ≥ 0.80, ruling out structural incapacity at any deployment floor ρ ≤ 0.80, and in particular at the representative ρ = 0.5 used throughout this section (P2); a stricter floor (e.g. ρ = 0.9) would place RT-IoT × Gemma-2’s κ̂ = 0.803 into the structural-incapacity regime instead, since incapacity is a property of the pair (ρ, f ), not of f alone. But κ̂ overstates what CRC could actually reach at α = 0.05: for CIC × Mistral, κ̂ = 1.000 suggests substantial unused capacity, while κ̂α = 0.380 is almost exactly A(τ̂ ) = 0.380—the calibrated threshold is already the best any threshold on this split can do without exceeding the risk target, and this is risk-constrained incapacity, not a fixable misalignment. RT-IoT × Gemma-2 shows the same pattern (κ̂ = 0.803, κ̂α = 0.122, A(τ̂ ) = 0.111). CIC × Gemma-2 is different: the oracle κ̂α = 0.637 remains well above A(τ̂ ) = 0.552, an 8.5-point gap suggesting the shortfall may be recoverable. We verify this without appeal to test labels: searching Proposition 2’s singleton intervals for a threshold satisfying R(τ ) ≤ α using only the 200-sample calibration split (the same split that produced τ̂ ) yields an alternative threshold with calibration-split action rate 0.585; freezing this threshold and evaluating it on the disjoint heldout test split gives A = 0.605 (vs. CRC’s 0.552 on the same split) at empirical risk 0.036 ≤ α—a genuine, label-free-atselection-time improvement, confirming recoverable threshold misalignment for this configuration specifically rather than inferring it from the oracle κ̂α alone. This distinction matters because κ̂ and the actionability gap AG = κ̂ − A(τ̂ ), while still a valid label-free necessary condition and upper bound (Theorem 6), are not sufficient on their own to diagnose which configurations are recoverable: two of the three cases with the largest AG turn out not to be, and only a labeled check of κ̂α against A(τ̂ ) resolves it. This is still a strictly stronger and falsifiable replacement for the retracted NC-variance heuristic (Proposition 4), which offered no such distinction at all; Section VII-D0d additionally tests whether genuine structural incapacity (κ̂ near zero) occurs anywhere in the ML+CRC baselines. e) P6 (finite-sample actionability certificate).: We apply Proposition 5 rather than the raw empirical action rate: τ̂ is fixed on a 200-sample calibration split, and Aδ is computed at δ = 0.1 from the disjoint certification split (the remaining test samples; m = 1,288 for CIC-IDS-2018, 1,579 for RT-

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

11

TABLE V S INGLETON CAPACITY κ̂, RISK - FEASIBLE CAPACITY κ̂α , AND REALIZED ACTION RATE AT α = 0.05 FOR THE THREE LOWEST- UTILITY LLM CONFIGURATIONS , PLUS THE RANGE ACROSS THE REMAINING 15, ALL ON ONE FIXED CALIBRATION / TEST SPLIT ( SEED 42) WITH κ̂, κ̂α COMPUTED BY EXACT BREAKPOINT ENUMERATION SO THAT κ̂α ≥ A(τ̂ ) HOLDS BY CONSTRUCTION . κ̂ ALONE WOULD MISCLASSIFY TWO OF THE THREE AS RECOVERABLE THRESHOLD MISALIGNMENT; κ̂α SHOWS ONLY CIC × G EMMA -2 ACTUALLY IS . Dataset

Model

κ̂

κ̂α

A(τ̂ )

Regime

RT-IoT2022 Gemma-2 9B 0.803 0.122 0.111 Risk-constrained incapacity CIC-IDS-2018 Mistral 7B 1.000 0.380 0.380 Risk-constrained incapacity CIC-IDS-2018 Gemma-2 9B 0.915 0.637 0.552 Threshold misalignment (remaining 15 configurations: κ̂α ≥ 0.923, within 0.08 of A(τ̂ ) in all)

IoT2022, and 26 for HIKARI-2021, whose small test set gives a correspondingly wide margin εm ≈ 0.21). Certifying (α, ρ)-actionability at α = 0.05, ρ = 0.5 against Aδ (not the point estimate) still passes 16 of 18 configurations, and the two that fail—CIC × Mistral (Aδ = 0.350) and RT-IoT × Gemma-2 (Aδ = 0.084)—are exactly the two Table V configurations whose risk-feasible capacity κ̂α is itself below ρ = 0.5 (Table V’s risk-constrained-incapacity cases): no calibration choice at α = 0.05 could have certified these regardless of sample size, which is a stronger and more specific statement than a failed empirical check. For the 16 certified configurations, Theorem 8 with Aδ in place of ρ gives U ≥ Aδ − 0.05 and Q ≤ 0.05/Aδ on the 0.9-probability event that the action-rate bound holds, combined with the risk side’s standard marginal conformal guarantee (Theorem 1)—a stated confidence level on the action-rate side, rather than an unqualified point estimate, though not a single joint confidence level across both sides (Proposition 5). D. RQ3: CRC vs. Baselines a) CRC vs. Argmax across all configurations.: B1Argmax (no abstention) uses the LLM’s top-1 prediction directly, so FAR equals the base misclassification risk regardless of α. Across all 18 configurations at α = 0.05, B1-Argmax violates the FAR target in 6 of 18 cases (where the base error rate exceeds 0.05), while B3-CRC satisfies it in 16 of 18. In the 12 configurations where B1-Argmax happens to satisfy α = 0.05 (because the model is already accurate enough), CRC achieves comparable utility (mean 0.94 vs. B1’s 0.97) while adapting the decision rule to the target risk level. b) Detailed three-way comparison.: Table VI compares all three baselines on CIC-IDS-2018 × Qwen-3 8B, where the base error rate (0.033) makes the FAR–utility tradeoff non-trivial. B2-Threshold here acts on the predicted-class confidence p(1) (x) only (Definition of gτ , Section IV-D): the threshold τ is swept on the calibration split to find the smallest τ (highest action rate) whose calibration-set FAR satisfies the target, then frozen and applied to the test split. An earlier version of this baseline selected τ using the true test label directly, which is not a deployable decision rule; we report the corrected, label-free version throughout. At α = 0.01, B2 and B3 are statistically indistinguishable (τ = 0.839 vs. 1 − q̂ = 0.836): both restrict action to a comparably high-confidence region and reach nearly identical FAR and utility. At α ∈ {0.05, 0.10, 0.20}, the cal-split FAR of the unrestricted argmax rule (0.033) already satisfies the

TABLE VI BASELINE COMPARISON ON CIC-IDS-2018 × Q WEN -3 8B. U TILITY IS THE CORRECT AUTOMATED ATTRIBUTION RATE . B2-T HRESHOLD ’ S τ IS CALIBRATED ON THE CAL SPLIT ONLY AND FROZEN BEFORE EVALUATION ON TEST.

α

Method

FAR

Utility

Coverage

0.01

B1-Argmax B2-Threshold B3-CRC

0.033 0.001 0.001

0.967 0.858 0.858

— — 0.999

0.05

B1-Argmax B2-Threshold B3-CRC

0.033 0.033 0.030

0.967 1.000 0.959

— — 0.959

0.10

B1-Argmax B2-Threshold B3-CRC

0.033 0.033 0.016

0.967 1.000 0.933

— — 0.933

0.20

B1-Argmax B2-Threshold B3-CRC

0.033 0.033 0.000

0.967 1.000 0.780

— — 0.780

target, so the calibrated τ collapses to 0 and B2 degenerates to B1-Argmax exactly—both act on every alert, and both happen to realize test-set FAR below α in this particular split. On this single configuration, CRC does not dominate a correctly calibrated global threshold in raw utility; the earlier reported 25.7% utility gap at α = 0.05 was an artifact of a baseline that selected τ using held-out true labels and is not reproduced here. The distinction that survives is the one identified in Section IV-C and Section IV-D: CRC’s threshold q̂ is the calibration order statistic of Theorem 1 and inherits its finite-sample, distribution-free coverage guarantee under exchangeability, whereas B2’s τ is selected by unconstrained grid search against the calibration-set empirical FAR and carries no comparable guarantee—it can overfit a small or unlucky calibration split, a risk that does not appear in this single lownoise configuration but is not bounded by any theorem for B2. We quantify the operational cost of using CRC’s conformal miscoverage as a surrogate for wrong-automation risk next. c) Surrogate slack.: Theorem 7 decomposes conformal miscoverage C(τ ) = P(y ∈ / Γτ (x)) into the operational risk R(τ ) actually incurred by automated decisions and the deferred-miscoverage slack S(τ ) = C(τ ) − R(τ ) spent on alerts that are abstained anyway. Table VII reports this decomposition for B3-CRC on CIC-IDS-2018 × Qwen-3 8B, using C(τ ) = 1 − coverage from Table VI. At α = 0.20, the identity is most visible: all of the nominal 0.22 miscoverage budget is spent on samples that are already abstained (S = 0.2197, R = 0), so the coverage-calibrated

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

12

1.0

TABLE VII S URROGATE - SLACK DECOMPOSITION (T HEOREM 7) FOR B3-CRC ON CIC-IDS-2018 × Q WEN -3 8B. S(τ ) = C(τ ) − R(τ ) ≥ 0 IN EVERY ROW, CONFIRMING THE IDENTITY; S GROWS WITH α AS MORE OF THE MISCOVERAGE BUDGET IS SPENT ON ALREADY- DEFERRED ALERTS . C(τ )

R(τ )

S(τ )

AR

0.0008 0.0411 0.0668 0.2197

0.0008 0.0295 0.0163 0.0000

0.0000 0.0116 0.0505 0.2197

0.141 0.012 0.050 0.220

threshold is fully conservative relative to the operational objective at this operating point—conformal miscoverage control is valid here but tells us nothing about wrong-automation risk, which is exactly zero. At α = 0.05, roughly 28% of the miscoverage budget (0.0116/0.0411) is similarly nonoperational. This slack is the quantitative reason a practitioner should report R(τ ) (or FAR) directly rather than treating conformal coverage as self-evidently the quantity of interest. d) ML+CRC baseline (B4).: To isolate the contribution of LLM backbones from the CRC framework itself, we apply the same CRC abstention layer to XGBoost and LightGBM classifiers trained on the same attack-class features. Table VIII reports the results at α = 0.05. On all three datasets, ML+CRC is competitive with the best LLM+CRC (utility within 1–3 percentage points), indicating that CRC’s value is largely orthogonal to backbone choice when the base classifier’s probability estimates are usable. The largest gap is on HIKARI-2021, where the best LLM (LLaMA-3) retains a 2.5-point advantage over LightGBM (0.985 vs. 0.960). This result clarifies the contribution of the backbone: the CRC framework is model-agnostic and delivers comparable utility with ML classifiers when their probability estimates are useful. The observed advantage of the LLM backbones is consistency across these three datasets, not uniform dominance over every ML classifier. e) A retracted structural-incapacity claim, and what replaced it.: An earlier version of this analysis reported that LightGBM+CRC on RT-IoT2022 abstains on 100% of samples (FAR = 0, Utility = 0) at α ≤ 0.10, and interpreted this as a structural-incapacity instance under Theorem 6—the regime κ̂ < ρ, distinct from both risk-constrained incapacity and threshold misalignment in Table V, neither of which requires κ̂ itself to be low. This did not survive a trainingstability check. Retraining LightGBM on the identical data, feature pipeline, and calibration procedure with 20 different random seeds ({0, . . . , 19}, holding the CRC calibration seeds fixed) produced a degenerate (Utility < 0.1) model in 0 of 20 runs; every retrained model achieves κ̂ = 1.000 and Utility ≈ 0.948–0.950, matching XGBoost. The original collapse is therefore best explained as a one-off training artifact— plausibly floating-point nondeterminism in multi-threaded histogram construction producing, on that single run, a true-label probability distribution with a rare but non-negligible highnonconformity cluster large enough to saturate the conformal calibration quantile at q̂ = 1—rather than a reproducible property of LightGBM or of this dataset. We report this

Utility

α 0.01 0.05 0.10 0.20

0.9 0.8 0.7

Qwen-3 8B LLaMA-3 8B Qwen-3 32B

0.6 0.00

0.01

0.02 FAR

0.03

0.04

Fig. 2. FAR–Utility tradeoff on CIC-IDS-2018 for three representative LLMs, swept over α ∈ [0.001, 0.20]. Markers trace the empirical operating points; the elbow near α = 0.01–0.05 marks the practical transition from strict risk control to the main operating point.

as a negative result: no structural-incapacity instance occurs anywhere in the 24 LLM+CRC and ML+CRC configurations evaluated in this paper. Theorem 6 and Proposition 3 remain applicable as a pre-deployment diagnostic for the regime, and the retracted case is retained here as a caution against drawing structural conclusions from a single training run of a stochastic learner, a risk the label-free κ̂ certificate does not remove by itself—it must be paired with a stability check across retrainings when the base learner’s fit is not itself guaranteed to be stable. f) Pareto frontier analysis.: Figure 2 traces the FAR– Utility tradeoff for three representative models (Qwen-3 8B, LLaMA-3 8B, Qwen-3 32B) on CIC-IDS-2018 by sweeping α ∈ [0.001, 0.20]. All three models exhibit a sharp elbow between α = 0.01 and α = 0.05: utility jumps from ∼0.62– 0.86 to ∼0.96, after which further relaxation of α yields diminishing returns. LLaMA-3 8B achieves the earliest saturation (utility > 0.96 at α = 0.05), while Qwen-3 32B exhibits a delayed elbow due to its bimodal NC score distribution (nc var = 0.032).

E. RQ4: ATT&CK Hierarchy Consistency Because ϕ is bijective, technique-to-tactic propagation of the CRC guarantee is a mathematical consequence of the label structure (Section III), not an empirical finding; as a sanity check, FAR, utility, and coverage are numerically identical at both levels across all seeds and α (e.g., seed 42, α = 0.01: FAR= 0.00078, utility= 0.858, coverage= 0.999 at both). Under a genuinely many-to-one mapping the guarantee would still propagate but would no longer be tight, since some technique-level errors become correct at the tactic level— exactly the coarsening identity of Theorem 4, verified directly under a real many-to-one map in Section VII-C.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

13

TABLE VIII ML+CRC BASELINE (B4) VS . BEST LLM+CRC (B3) AT α = 0.05, AFTER THE TRAINING - STABILITY CORRECTION OF S ECTION VII-D0 D . U TILITY IS THE CORRECT AUTOMATED ATTRIBUTION RATE . FAR (mean ± std)

Utility

Guarantee

XGBoost+CRC LightGBM+CRC Best LLM+CRC

0.000 ± 0.000 0.000 ± 0.000 0.031 ± 0.005

0.961 0.959 0.961

✓ ✓ ✓

HIKARI-2021

XGBoost+CRC LightGBM+CRC Best LLM+CRC

0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000

0.956 0.960 0.985

✓ ✓ ✓

RT-IoT2022

XGBoost+CRC LightGBM+CRC Best LLM+CRC

0.000 ± 0.000 0.0001 ± 0.0000 0.001 ± 0.000

0.952 0.953 0.959

✓ ✓ ✓

Dataset

Model

CIC-IDS-2018

TABLE IX C ROSS - DATASET GENERALIZATION AT α = 0.05. “C AL” = CALIBRATION SOURCE , “T EST ” = TEST TARGET. FAR REMAINS BELOW α IN THESE 8 CONFIGURATIONS , BUT UTILITY VARIES WIDELY. Model

Cal

Test

FAR

Utility

Guarantee

Qwen-3 8B

CIC CIC HIKARI RT-IoT

HIKARI RT-IoT CIC CIC

0.000 0.011 0.000 0.000

1.000 0.962 0.043 0.836

✓ ✓ ✓ ✓

LLaMA-3 8B

CIC CIC HIKARI RT-IoT

HIKARI RT-IoT CIC CIC

0.000 0.003 0.000 0.000

1.000 0.992 0.611 0.580

✓ ✓ ✓ ✓

TABLE X E FFECT OF CALIBRATION SET SIZE nCAL ON FAR AND U TILITY (CIC-IDS-2018, Q WEN -3 8B, α = 0.05). U TILITY SATURATES AT nCAL ≈ 200; INTERMEDIATE VALUES (400, 600) FALL WITHIN 0.01 OF THE 800 ENDPOINT. ncal

FAR

AR

Utility

50 100 200 800

0.008 0.024 0.024 0.024

0.097 0.022 0.024 0.017

0.895 0.954 0.952 0.959

VIII. D ISCUSSION A. Failure Case Analysis

F. RQ5: Cross-Dataset Robustness Table IX tests whether a CRC threshold calibrated on one dataset transfers to another, for Qwen-3 8B and LLaMA-3 8B at α = 0.05. Empirical FAR remains below α in all 8 configurations despite exchangeability not being guaranteed across datasets— a robustness observation, not a formal transfer guarantee— but utility degrades substantially in several cases, down to 4.3% (95.7% deferred) for HIKARI→CIC with Qwen-3. The asymmetry is informative: CIC→HIKARI transfers well (utility 1.0) because CIC’s richer label space produces a threshold generous enough for HIKARI’s easier 2-class problem, while HIKARI→CIC fails because HIKARI’s 2-class calibration is overly conservative for CIC’s 3-class task. In-domain calibration is therefore essential for deployment; adaptive or non-exchangeable conformal methods [30], [31] are natural extensions. G. RQ6: Calibration Set Size Ablation Table X shows the effect of calibration set size on FAR and utility (CIC-IDS-2018 × Qwen-3 8B, α = 0.05). Utility rises sharply from ncal = 50 (0.895) to 100 (0.954), then saturates within 0.01 through 800, while FAR stays well below α = 0.05 throughout; at ncal = 50 the threshold is overly conservative (FAR= 0.008 ≪ 0.05), a finite-sample artifact where an imprecise q̂ makes CRC default to caution. Consequently, 200 labeled attack samples suffice for nearoptimal CRC deployment—a modest requirement for most SOC environments.

An earlier version of this analysis diagnosed low-utility configurations using nonconformity-score variance combined with base accuracy. Proposition 4 shows this heuristic is not sound in general: two classifiers can share the same truelabel score mean, variance, and top-one accuracy while having opposite singleton capacity (κ = 1 vs. κ = 0). We replace it with the label-free diagnostic developed in Section IV-D and verified in Section VII-C. Every low-utility configuration in the LLM matrix (Table V) has high singleton capacity (κ̂ ≥ 0.80): RT-IoT × Gemma2 (κ̂ = 0.803), CIC × Mistral (κ̂ = 1.000), and CIC × Gemma-2 (κ̂ = 0.915). By Theorem 6, this rules out structural incapacity for all three at ρ = 0.5: the base classifier’s top-1 and top-2 attack probabilities are separable enough, on most inputs, to support a much higher unconstrained automation rate than CRC’s calibrated τ̂ achieves. It does not follow that this unused capacity is reachable under the risk target, and here the three cases diverge once κ̂α (Definition 4) is checked directly against the true labels. For CIC × Mistral and RT-IoT × Gemma-2, κ̂α (0.380 and 0.122) sits almost exactly at the realized A(τ̂ ) (0.380 and 0.111): CRC’s calibrated threshold is already the best any threshold can do without exceeding α = 0.05 on this split, and the large unconstrained κ̂ was, in these two cases, not informative about what recalibration could achieve—these are risk-constrained incapacity, and the correct remedy is a less stringent α or a more competent base classifier, not a different threshold at the same α. CIC × Gemma-2 is the one case where the oracle κ̂α = 0.637 remains well above A(τ̂ ) = 0.552, and here the recoverability is confirmed without test-label selection (Section VII-C): a threshold search

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

restricted to the calibration split alone finds an alternative τ̂ that reaches A = 0.605 at empirical risk 0.036 ≤ α on the disjoint held-out test split, genuinely recoverable threshold misalignment for this configuration specifically. The complementary failure mode—genuine structural incapacity, where κ̂ itself is near zero and no threshold can raise the action rate at any risk level—is not observed anywhere in this study; Section VII-D0d reports an ML+CRC configuration that initially appeared to be such a case but did not survive a 20-seed training-stability check, and we retract that reading rather than retrofit a diagnosis to an artifact. The practical predeployment check that follows from Theorem 6 and Proposition 3 is therefore staged, not a single number: estimate κ̂ on an unlabeled readiness sample before calibration, and re-estimate it across a small number of retrainings when the base learner’s fit is not itself guaranteed to be stable. If κ̂ is already below the required action-rate floor ρ, no amount of recalibration, larger ncal , or α relaxation will make the configuration deployable, and the correct remedy is to change the base classifier. If κ̂ ≥ ρ, this only rules out the geometric obstruction; a shortfall at A(τ̂ ) < ρ still requires checking κ̂α (which needs labels) before concluding that recalibration—rather than a different α or a different classifier—is the right remedy. B. Practical Implications Section VII-A’s FAR check alone is not a deployment criterion: Proposition 1 and the retracted case in Section VII-D0d show that a low FAR is equally consistent with a wellfunctioning tiered workflow and with a system that automates almost nothing. The framework instead supports a four-stage deployment procedure, each stage gating the next: 1) Capacity check: before calibration, estimate κ̂ on an unlabeled readiness sample (Proposition 3). If κ̂ falls below the required action-rate floor ρ, no calibration choice can reach ρ (Theorem 6); replace the base classifier rather than proceeding. 2) Calibration: fix τ̂ on a labeled calibration split via Algorithm 1, certifying R(Cτ̂ ) ≤ α (Theorem 1). 3) Actionability certification: on a disjoint certification split, compute the action-rate lower bound Aδ (Proposition 5). Only if Aδ ≥ ρ does the contract satisfy (α, ρ)-actionability with confidence 1 − δ (Theorem 8); if it does not, check κ̂α against the same certification data (Definition 4) before choosing a remedy: κ̂α ≥ ρ means a different threshold at the same α can recover the shortfall (recalibration), while κ̂α < ρ means the current α itself is the binding constraint, and only relaxing α or replacing the base classifier will help (Section VIII-A). 4) Deployment: alerts with gτ̂ (x) = 1 are logged with the singleton ATT&CK technique and routed to automated playbooks; alerts with gτ̂ (x) = 0 are forwarded to analysts with the top-k candidate techniques and their scores as decision support. Skipping stages 1 and 3 and deploying on the FAR check alone is exactly the failure mode this paper opened with: RTIoT2022 × Gemma-2 (Table V) passes stage 2 with FAR = 0.036 ± 0.004 ≤ α but has Aδ = 0.084, and would clear a

14

FAR-only gate while automating under 10% of alerts. In the configurations that pass all four stages, the workflow handles roughly 90–95% of attack alerts correctly and automatically with the certified error bounds of Theorem 8. C. NC Score Design The ablation in E7 compares normalized NC scores (Eq. (1)) against raw softmax probabilities. At α = 0.05, the normalized design achieves FAR = 0.030 with utility = 0.959, while the raw design yields FAR = 0.032 with utility = 0.976 but higher FAR variance across seeds. Normalization produces tighter FAR control at a modest utility cost, which is the preferred tradeoff for safety-critical applications. D. Limitations a) Label granularity and score ranking.: Our LLM+CRC evaluation uses 3–4 ATT&CK techniques per dataset, whereas the full ATT&CK matrix contains over 600 techniques. The CRC framework itself is agnostic to the number of classes, but κ(f ) (Theorem 6) may degrade with finer-grained technique taxonomies where LLM confusion between similar techniques increases; as a supplementary check on the current 4-technique mapping, the Spearman rank correlation between fk (x) and the groundtruth indicator averages ρ = 0.672 (std 0.210, minimum −0.188) across 48 model–dataset–class configurations, with 43.75% of configurations exceeding ρ = 0.8—consistent with κ(f ) ≥ 0.80 holding everywhere in Table V but not a guarantee that it will continue to at finer granularities. Section IV-C shows that real fine-grained subtype labels exist in at least one benchmark (CIC-IDS-2018’s raw attack labels, pre-dating the dataset’s own 4-category collapse) and uses them to test the coarsening-transfer identity; extending the full LLM+CRC pipeline to this or a finer ATT&CK sub-technique resolution, rather than the single-classifier XGBoost instantiation used there, is left for future work. b) Input modality.: Our system operates on structured network flow features, not unstructured text. It is not comparable to CTI text-to-ATT&CK systems (rcATT, TRAM, LADDER), which solve a fundamentally different task. The two approaches are complementary: ours handles real-time IDS alerts, while CTI mappers process post-incident reports. c) Model dependence.: CRC utility is bounded by the base model’s classification accuracy and score separation. As shown in Table IV, poorly performing models (Gemma-2 on RT-IoT) result in near-total abstention. A single training run of the ML+CRC baseline initially appeared to reinforce this with a complete-abstention case (LightGBM on RT-IoT, utility = 0); Section VII-D0d shows this does not reproduce across 20 retrainings and should not be read as a stable property of either the classifier or the framework. CRC does not improve model accuracy; it provides a principled mechanism to expose and manage model uncertainty. In these experiments, LLM backbones offer more stable probability rankings across the three traffic distributions, even though ML classifiers are competitive on individual datasets.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

d) Conditional error among automated outputs.: The controlled FAR is an unconditional risk over all attack alerts. It should not be read as the conditional error rate among the subset of alerts that receive automated attributions. Remote recomputation from the raw logprob files shows that, at α = 0.05, the mean conditional automated-attribution error is 3.6%, but RT-IoT2022 × Gemma-2 reaches 36.3% because the model automates only 9.9% of alerts. This is why conditional error should be monitored as a deployment diagnostic even though the formal risk target is unconditional. e) Exchangeability assumption.: The conformal guarantee requires calibration and test samples to be exchangeable. In practice, network traffic is non-stationary (new attack types emerge, traffic patterns shift). Our cross-dataset experiments (Section VII-F) show low empirical FAR under the tested shifts, but they do not provide a formal guarantee once exchangeability is broken. Utility degrades substantially under several shifts. Adaptive conformal methods [30], [32] that reweight or refresh calibration data over time are a natural extension for production SOC deployments, as are nonexchangeable risk-control formulations [31] that relax the exchangeability requirement directly rather than tracking a drifting threshold; either would extend Theorem 1’s guarantee without changing the decision-contract accounting built on top of it. IX. C ONCLUSION This paper began from an operational puzzle—a calibrated conformal risk bound satisfied by a classifier that automates nothing—and generalized it: a risk certificate is a property of a decision contract, not of a predictor alone, and can be satisfied by weakening that contract through abstention or semantic coarsening rather than genuine predictive competence. We formalized this as an error-conservation law (B(h) = R(C)+D(C)+M (C), Theorem 3), an exact geometric characterization of singleton automation and the resulting capacity limit κ(f ) (Theorem 6), and a non-degenerate (α, ρ)actionable certificate whose action-rate side carries an explicit finite-sample lower confidence bound (Proposition 5), distinct in kind from the risk side’s standard conformal guarantee rather than resting on a single realized measurement. Instantiated on ATT&CK-aligned alert triage for LLMbased intrusion detection (3 datasets, 6 LLMs, 4 error-rate thresholds), the theory separates three regimes that a scalar risk number cannot distinguish. The singleton-capacity diagnostic rules out structural incapacity for all three low-utility LLM configurations at deployment floor ρ = 0.5, correcting what a variance-based heuristic would have wrongly permitted (Proposition 4); but κ(f ) alone is not sufficient to call the shortfall recoverable—the risk-feasible capacity κα (f ) shows two of the three are risk-constrained incapacity, not threshold misalignment, and only one is genuinely fixable, verified by exhibiting the alternative threshold rather than inferred from κ(f )’s unconstrained bound. A 20-seed training-stability rerun of the ML+CRC baseline shows an initially observed zeroutility collapse (LightGBM on RT-IoT2022) does not reproduce, so this study finds no confirmed instance of the third

15

regime, genuine structural incapacity; we report this as an open question rather than overstate a single-run anomaly. Using real fine-grained attack-subtype labels from CIC-IDS-2018’s own data pipeline, we confirmed the coarsening-transfer identity (Rϕ = Rfine −Mϕ , Theorem 4) exactly under a genuine manyto-one map, with small but non-zero masking mass, showing the identity’s practical bite is taxonomy- and model-dependent rather than universal. Calibration efficiency analysis shows 200 labeled attack samples suffice for near-optimal performance. The practical implication is that a security automation pipeline should certify capacity, risk, and action rate jointly, not risk alone. Future work includes searching for a confirmed structural-incapacity instance, extending the coarsening analysis to the full LLM matrix, handling distribution shift via adaptive or non-exchangeable conformal methods, and integrating actionability certification into multi-step incident response pipelines with trajectory-level risk guarantees. R EFERENCES [1] B. A. Alahmadi, L. Axon, and I. Martinovic, “99% false positives: A qualitative study of soc analysts’ perspectives on security alarms,” in USENIX Security Symposium, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:245130092 [2] M. A. Ferrag, M. Ndhlovu, N. Tihanyi, L. C. Cordeiro, M. Debbah, T. Lestable, and N. S. Thandi, “Revolutionizing cyber threat detection with large language models: A privacy-preserving bert-based lightweight model for iot/iiot devices,” 2024. [Online]. Available: https://arxiv.org/abs/2306.14263 [3] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=33XGfHLtZg [4] L. Li, C. Huang, and J. Chen, “Automated discovery and mapping att&ck tactics and techniques for unstructured cyber threat intelligence,” Comput. Secur., vol. 140, no. C, May 2024. [Online]. Available: https://doi.org/10.1016/j.cose.2024.103815 [5] V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World, 01 2005. [6] A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei, “Learn then test: Calibrating predictive algorithms to achieve risk control,” The Annals of Applied Statistics, vol. 19, no. 2, pp. 1641 – 1662, 2025. [Online]. Available: https://doi.org/10.1214/24-AOAS1998 [7] C. Lu, A. Lemay, K. Chang, K. Hoebel, and J. Kalpathy-Cramer, “Fair conformal predictors for applications in medical imaging,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 12 008– 12 016, 06 2022. [8] L. Lindemann, M. Cleaveland, G. Shim, and G. Pappas, “Safe planning in dynamic environments using conformal prediction,” IEEE Robotics and Automation Letters, vol. 8, pp. 5116–5123, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252992582 [9] V. Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. Jaakkola, and R. Barzilay, “Conformal language modeling,” ArXiv, vol. abs/2306.10193, 2023. [Online]. Available: https://api.semanticscholar. org/CorpusID:259203582 [10] R. Jordaney, K. Sharad, S. K. Dash, Z. Wang, D. Papini, I. Nouretdinov, and L. Cavallaro, “Transcend: detecting concept drift in malware classification models,” in Proceedings of the 26th USENIX Conference on Security Symposium, ser. SEC’17. USA: USENIX Association, 2017, p. 625–642. [11] R. Smith, N. Japkowicz, M. Dondo, and P. Mason, Using Unsupervised Learning for Network Alert Correlation, 05 2008, vol. 5032, pp. 308– 319. [12] Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 4885–4894. [13] R. El-Yaniv and Y. Wiener, “On the foundations of noise-free selective classification,” J. Mach. Learn. Res., vol. 11, p. 1605–1641, Aug. 2010. [14] Y. Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” in International Conference on Machine Learning, 2019.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

[15] A. Gangrade, A. Kag, and V. Saligrama, “Selective classification via one-sided prediction,” 2021. [Online]. Available: https://arxiv.org/abs/ 2010.07853 [16] D. Wei, “Decision-making under selective labels: Optimal finite-domain policies and beyond,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 11 035–11 046. [Online]. Available: https://proceedings.mlr.press/v139/wei21a.html [17] E. Phillips, F. K. Gustafsson, S. Wu, A. Thakur, and D. A. Clifton, “Entropy alone is insufficient for safe selective prediction in llms,” 2026. [Online]. Available: https://arxiv.org/abs/2603.21172 [18] V. Kotte, “When can conformal risk control certify llm outputs? bounds, impossibility, and adaptation for structured generation,” 2026. [Online]. Available: https://arxiv.org/abs/2606.29054 [19] H. Khosravi and X. Huo, “Conformal selective acting: Anytimevalid risk control for rlvr-trained llms,” ArXiv, vol. abs/2605.20270, 2026. [Online]. Available: https://api.semanticscholar.org/CorpusID: 288654956 [20] T. Ding, A. Angelopoulos, S. Bates, M. Jordan, and R. J. Tibshirani, “Class-conditional conformal prediction with many classes,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 64 555–64 576. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2023/file/cb931eddd563f8d473c355518ce8601c-Paper-Conference.pdf [21] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in International Conference on Information Systems Security and Privacy, 2018. [22] A. Ferriyan, A. H. Thamrin, K. Takeda, and J. Murai, “Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,” Applied Sciences, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:239692344 [23] B. S. Sharmila and R. Nagapadma, “Quantized autoencoder (qae) intrusion detection system for anomaly detection in resource-constrained iot devices using rt-iot2022 dataset,” Cybersecurity, vol. 6, pp. 1– 15, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID: 261516162 [24] G. T. M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram’e, J. Ferret, P. Liu, P. D. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stańczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. A. Welty, C. A. Choquette-Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozi’nska, D. Herbison, E. Bandy, E. Wang, E. Noland, E. Moreira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak-Pluci’nska, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernandez, J. R. van Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J. Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. cia Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Gorner, M. C. N. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. shad Bardoliwalla, N. Devanathan, N. Dumai, N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. chong Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. M. Carthy, S. Perrin, S. M. R. Arnold, S. bastian Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kociský, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Z. Wei, V. Cotruta, P. Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev, “Gemma 2: Improving open language models at a practical size,” ArXiv, vol. abs/2408.00118, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:270843326

16

[25] A. Dubey et al., “The llama 3 herd of models,” 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271571434 [26] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825 [27] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388 [28] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 785–794. [29] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: a highly efficient gradient boosting decision tree,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 3149–3157. [30] I. Gibbs and E. J. Candès, “Adaptive conformal inference under distribution shift,” in Proceedings of the 35th International Conference on Neural Information Processing Systems, ser. NIPS ’21. Red Hook, NY, USA: Curran Associates Inc., 2021. [31] A. Farinhas, C. Zerva, D. T. Ulmer, and A. Martins, “Nonexchangeable conformal risk control,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=j511LaqEeP [32] M. Zaffran, A. Dieuleveut, O. F’eron, Y. Goude, and J. Josse, “Adaptive conformal predictions for time series,” in International Conference on Machine Learning, 2022. [Online]. Available: https: //api.semanticscholar.org/CorpusID:246863519

Related documents

• A practical treatise on the law of contracts not under seal : and upon the usual defences to actions thereon#334191
• Risk management for e-procurement – realization and implementation of risks in IT projects#956664
• Security Protocols and Threat Models : Security and Privacy via The Applied π-Calculus / by Reynaldo Gil-Pons, Ross Horne, Sjouke Mauw, Felix Stutz, Semen Yurkov#995905
• Contract Theory and the Limits of Contract Law#269948
• Bridging Theory and Practice: An Executable Taxonomy of Security Properties for ProVerif and Tamarin#238520