Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of “Unsafe” You Trained On

arXiv:2610.25060v1 [cs.LG] 18 Oct 2026

Vahid Jebraeeli

Jonathon Romero

Yijin Wang Adrian Chan Lenovo vjebraeeli@lenovo.com

Jianbang Zhang

Abstract A guardrail classifier is reported as if unsafe were a property of text. It is not: it is a property of a policy, and the policy enters the model through its labels. We exploit an accident of an industrial deployment in which the same ∼53k prompts were labelled twice, under two operational definitions (“is this a request for harmful content?” and “would a moderator block this?”), and used to train ten guardrail arms sharing architecture, codebase and optimiser family, at two model scales. First, the guardrail’s own training corpus already contained the benchmarks: JailbreakBench-benign is 88% and OR-Bench-hard 82% present in the training pools, two over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block. Everything below is decontaminated. Second, safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model: with architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23× spread. Third, a benchmark score can be moved across its entire range by 0.03% of the parameters. Across these arms HarmBench-copyright spans 0 to 100 and OR-Bench-hard over-refusal 3.4% to 56.4%, and the four arms whose last supervision signal carries a given policy occupy one extreme of both (exact rank p = 0.005, or 0.04–0.14 at worst), at unchanged detection capability. For a distilled student the teacher’s soft labels do this even when every hard label carries the other policy, so teacher provenance is an unrecorded confound in any released checkpoint. Fourth, validation accuracy on the one split that grades all twenty arms tracks agreement with the annotator’s policy (ρ = +0.74), not detection capability (ρ = +0.34). We argue that guardrail evaluations should report contamination, a seed floor, and the definition they were graded against as first-class variables.

1

Introduction

Safety classifiers, commonly called guardrails, are trained and reported like any other classifier. A paper states a macro-F1 on a held-out split, a set of catch rates on public benchmarks such as HarmBench [17], AdvBench [37], JailbreakBench [2] or StrongREJECT [30], an over-refusal number on OR-Bench [4] or XSTest [27], and the reader is invited to compare that row against another system’s row. This reporting convention contains a silent assumption: that the label unsafe names a property of the prompt, so that two systems scored on the same prompts are being compared on the same question. In deployment that assumption is visibly false. “Reproduce the lyrics of this song” is a policy violation for a content platform and an ordinary request for a drafting assistant. “Tell me a dirty joke” is blocked by a moderation product and answered by a coding assistant. Neither is a fact about 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: TAE (Trust-AI-Eval): Can We Trust AI Evaluation?.

the string. Each is a decision, made once, by whoever wrote the labelling policy, and then absorbed by every model trained on those labels. Most work cannot separate the two, because a project has one labelling policy and therefore one model family. We separated them by accident. Building a guardrail against a proprietary 24-pillar safety taxonomy for an on-device assistant, we labelled the same prompt pool twice, a month apart, with two pipelines judged comparably accurate at the time. They disagreed on one prompt in six, and the disagreements were not noise: they were systematic, and traced to a difference in the question the pipeline was answering. Pipeline A asked would a thoughtful moderator block this? Pipeline B asked is this a request for harmful content? We then trained ten arms (mixing, re-weighting, anchoring, distilling and head-calibrating across the two label families) at two model scales, and evaluated all twenty models identically on fifteen public safety benchmarks (33,137 prompts). That grid is a measuring instrument for the evaluation protocols themselves. Our contributions are diagnostic rather than methodological: 1. The training corpus contains the benchmark suite (§3). We measure the overlap between every benchmark and every pool the grid trained or validated on. It is severe and structured: 88% of JailbreakBench-benign and 82% of OR-Bench-hard, against 0% of ORBench-toxic and 2% of AdvBench. The worst hit are the over-refusal benchmarks, because the training class they resemble was built by harvesting them. Everything below is decontaminated. 2. Benchmarks differ enormously in how much of their score the label definition decides (§4). With architecture and optimiser fixed, changing where the two label families enter training moves HarmBench-copyright across its entire range (0.99 of 1.0) and WildGuardbenign hardly at all (0.043). But a definition-unstable benchmark is also seed-unstable, and separating the two needs a seed floor almost nobody reports; we measure one from this codebase and show it leaves the middle of that ordering uninterpretable. 3. A score can be moved across its full range at 0.03% of the parameters (§5). The policy is carried by whichever supervision signal is last (head calibration labels, or a distilled student’s teacher), not by the training mixture: the arms whose last signal is policy A occupy one extreme of HarmBench-copyright and of OR-Bench-hard at both scales (p = 0.005, 0.14 at worst), while an arm whose hard labels are entirely policy A (40,327 of 40,327 rows), fully fine-tuned, reaches only 17.3 of 100 on HarmBench-copyright. The move leaves no trace in a data card. 4. Validation accuracy measures policy agreement, not capability (§7). One policy-Alabelled split grades all twenty arms and ranks them by policy (ρ = +0.74) rather than by detection (ρ = +0.34). It is itself 69.3% inside the policy-A training file, an overlap we found only by running §3’s audit on our own validation data. None of this says the benchmarks are bad or that the models are bad. It says that a substantial and unreported fraction of what a guardrail evaluation reports is an answer to “whose policy did you train on?”, a question the protocol never asks and the paper never states. §8 proposes the minimum that would fix it.

2

Setting

Task and taxonomy. The classifier scores a single user prompt and emits (i) one of four coarse classes, GENERIC _ SAFE, BOUNDARY _ SAFE, HARMFUL and COMPOUND (a benign request with an embedded harmful sub-request), and (ii) a 24-way multi-label vector over an in-house taxonomy grouped into Safety, Security, Privacy and Reliability pillars. Binary unsafe is HARMFUL ∪ COM POUND . The taxonomy is proprietary and does not map onto MLCommons categories, the ordinary situation for a product guardrail and why an off-the-shelf guard is not usable directly (§10). Models. Two sizes, both ModernBERT [33] encoders with three linear heads on the [CLS] state (Figure 1): a 394.8M-parameter teacher (ModernBERT-large) and a 149.0M-parameter student (ModernBERT-base). Writing fφ for the encoder and hψ for the heads, training minimises Lsup , a weighted sum of a 4-class cross-entropy, a 24-way pillar BCE and a harmful-span BCE under a class-balanced sampler; the student adds a KD term on the 4-class and pillar logits at τ =4. The third route in Figure 1 freezes φ and re-fits the heads alone on a small calibration set Dcal labelled 2

heads ψ: 0.04 M of 149 M (0.03 %)

4-class head

prompt

ModernBERT encoder

395 M teacher / 149 M student parameters φ

[CLS]

generic / boundary / harmful / compound

24-pillar head multi-label, one per pillar token head

harmful spans

where an operational definition can enter

1 hard labels in the training pool does not install the policy in a distilled student

2 the teacher's soft labels overrides the student's own hard labels

3 head-only calibration

backbone frozen; installs it from 8,489 examples (0.03 %)

Figure 1: The guardrail, and the three places an operational definition can enter it. Which of the three routes actually installs a policy is the subject of §5: route 1 does not, for a distilled student; routes 2 and 3 do, the last by re-fitting 0.03% of the parameters. ∑ under a target policy A, minimising (x,yA )∈Dcal CE(hψ (fφ̄ (x)), yA ) over |ψ| = 0.04M parameters, 0.03% of the model. Which of these three routes determines the model’s operational definition is the subject of §5; the answer is not Lsup . The two label families. Both pipelines label the same prompt pool (53,441 train / 5,936 val / 7,662 multilingual val). Policy A is the earlier, hand-curated pipeline, written against a moderationproduct rubric: it treats reproduction of copyrighted text, policy-refusal prompts and edgy-butharmless requests as blockable. Policy B is a later LLM-jury relabelling (four large instruct models, three samples each at T =0.7, three-round binary cascade, ties routed to a separate ambiguous bucket; cross-model agreement with the round majority 94.6%) written against a request-intent rubric: it treats those same prompts as safe, because they are not requests for harmful content. On the 97,245 prompts carrying a comparable 4-class label from both pipelines, exact agreement is 83.6%; on the 136,472 comparable at the binary level it is 90.1% (both measured on the 155k-row doublelabelled corpus, of which the 53,441-prompt training file is a fully double-labelled subset and the two validation files 92% and 84%). Per-class retention shows where the definitions part company: GENERIC _ SAFE 89.7% and HARMFUL 85.2%, but COMPOUND 78.3% and BOUNDARY _ SAFE only 71.2%, so the contested classes are by far the least stable. An internal spot-check judged B’s individual labels slightly more often correct than A’s; we report it only as a direction, since the sample was small (45 prompts) and its record was not retained. Neither family was validated against a human-adjudicated reference set: none was built, and the jury’s chance-corrected agreement (Fleiss κ = 0.78 binary, 0.70 on the contested BOUNDARY _ SAFE/GENERIC _ SAFE split, Appendix A) measures convergence of a procedure, not correctness. The point does not depend on which is cleaner: label quality and label definition are independent axes, and only the first is ever audited. The ten arms. The arms differ in how policy A’s definition was, or was not, introduced: blended into the training pool at various ratios, including 1,450 and 454 explicit policy-refusal anchors (E3, E5; E2 and E4 carry none); as an inverse blend (E9); as every hard label (E6); through the distillation teacher’s soft labels only (E8); or through a frozen-backbone head-only calibration pass that updates 0.04M of 149.05M parameters (E1, E7, E10). They are not matched on everything, and we do not claim they are: pools range from 8,489 to 144,744 rows, and the head-only arms are a different procedure (frozen backbone, no seed set, checkpoint chosen on training loss rather than validation). Architecture, codebase and optimiser family are shared; learning rate, warmup and initialisation are not. Table 3 lists each arm’s pool size, initialisation and teacher, read from the training configs. All twenty models were scored by one harness in a single deterministic pass. Evaluation suite. Fifteen public benchmarks, 33,137 prompts. Twelve scored by catch rate (fraction of benchmark-harmful prompts flagged unsafe): HarmBench standard/contextual/copyright, AdvBench, JailbreakBench-harmful, StrongREJECT, DoNotAnswer [32], WildGuard-harmful [7], SaladBench attack/defense/base [15], OR-Bench-toxic; three by over-refusal rate on benign prompts: OR-Bench-hard, JailbreakBench-benign, WildGuard-benign, of which decontamination (§3) leaves two usable. 3

3

First audit: the guardrail’s training corpus contains the benchmarks

A guardrail’s training corpus and its benchmark suite come from the same small pool of public safety datasets, and this is the thing we nearly did not check. The pipeline that built our BOUNDARY _ SAFE class harvested public over-refusal sets wholesale, and the evaluation harness then loaded the same files. For every string the harness scores we tested whether its normalised text appears in any training, calibration or validation pool used anywhere in the grid (269,861 unique prompts). Exact match after case and whitespace normalisation, so every number is a lower bound: it cannot see paraphrase, translation, or a prompt embedded in a longer string, which is why HarmBench-contextual scores 0.0%. Figure 2(a) gives the result. JailbreakBench-benign is 88.0% contaminated (88/100) and OR-Bench-hard 82.1% (1,083/1,319); HarmBench-copyright is 25.0%, WildGuard-harmful 14.3%, StrongREJECT 12.1%. The structure of that ordering is the finding. The two most contaminated benchmarks are over-refusal benchmarks, the metric a guardrail paper reports to argue its model is safe to deploy. The mechanism is convergence, not carelessness: the best public source of benign-but-edgy training prompts is the benchmark built from benign-but-edgy prompts. Any team that builds a BOUNDARY _ SAFE-like class from public data has probably done this; we have not seen that structure reported, though contamination itself is a recognised threat to LLM evaluation [28, 21, 34], inferred for guardrails behaviourally [35]. Contamination looks uneven across arms, but that unevenness is not a confound: own-pool exposure (1.4–73.8%) rank-correlates −0.01 (149M) and −0.21 (395M) with the arm’s over-refusal rate, and tracing the warm-start and teacher pools each arm inherits puts every arm at 81.0–82.3% (Appendix A). We re-ran all twenty models with per-prompt predictions retained and recompute everything below on clean subsets only, excluding any prompt in any pool used anywhere in the grid: the union, not each arm’s own overlap, so the evaluated subset is identical across arms and none is removed by the policy-A calibration pools alone. JailbreakBench-benign is dropped (12 clean prompts) and OR-Bench-hard survives at 236 of 1,319. The two effects we build on grow (HarmBench-copyright 0.98 → 0.99, OR-Bench-hard 0.42 → 0.57); across the fourteen usable benchmarks ten ranges rise or are flat and four fall slightly (the largest fall is WildGuard-benign, 0.048 → 0.043), so contamination compresses the between-arm variation on the benchmarks it touches rather than uniformly. Decontamination and label validity are entangled here, though: the jury that built BOUNDARY _ SAFE admitted a prompt on a 3-of-4 majority, so the surviving ORBench-hard prompts are 236 of the 353 it rejected, all 236 of which drew at least one actually harmful vote. The 0.42 → 0.57 shift therefore reads as label composition as well as memorisation and we cannot separate them (Appendix D).

4

Second audit: how much of a benchmark’s score the label definition decides

Write sb (a) for benchmark b’s score under arm a, and define its label-supervision range as the spread across the ten arms, averaged over the two model scales M : ] [ ] 1 ∑ [ max sb (a, m)−min sb (a, m) , Φb = median max sb (r, ς)−min sb (r, ς) . ∆b = a a ς ς r∈R |M | m∈M (1) Since the arms share codebase and optimiser and differ only in how and where the two label families entered training, ∆b bounds how much of b’s score label provenance decides. Figure 2(b) gives the ordering. HarmBench-copyright ranges over 0.99 of its full scale (across the ten student arms it takes every value from 0.000 to 1.000) and WildGuard-benign 0.043: over an order of magnitude between the ends of a suite that is routinely averaged into one number (23×; two significant figures is all 75 clean prompts in two templates support, Table 1). But range alone conflates two things. A benchmark whose score moves when the label source changes may simply be a benchmark whose score moves. Separating them needs the second quantity in (1): a seed floor Φb , the spread across runs differing in nothing but their random seed ς, over recipes R. Only ∆b /Φb is interpretable, and almost no guardrail paper reports Φb . This codebase holds 37 such triplets, and Φb turns on the recipe more than on the benchmark: over them 4

over-refusal benchmark 25% 82% 0.0% 3% 2% 0.0% 0.0% 6% 2% 12% 10% 12% 14% 3% 88%

100

harm-detection benchmark

HarmBench-copyright n=75 OR-Bench-hard (benign) n=236 OR-Bench-toxic n=655 DoNotAnswer n=910 SaladBench-defense n=195 HarmBench-contextual n=100 SaladBench-attack n=4999 SaladBench-base n=20093 AdvBench n=1070 StrongREJECT n=275 HarmBench-standard n=181 JailbreakBench-harmful n=88 WildGuard-harmful n=646 WildGuard-benign n=913 JailbreakBench-benign n=12

50

0

0.38 0.33 0.31 0.29 0.25 0.22 0.20 0.15 0.14 0.13 0.09 0.04 excluded

0.0

(a) share already inside the guardrail's own training data (%)

seed-only floor

0.5

0.57

0.99

1.0

(b) range of the score across the ten label-source arms, clean

Figure 2: Two audits of one safety suite, benchmarks ordered by (b). (a) how much of each benchmark was already inside the guardrail’s own training or validation data: the two most contaminated are over-refusal benchmarks, in red (§3). Exact match only, so a lower bound. (b) the range of each benchmark’s score across the ten label-source arms, on decontaminated subsets, at fixed architecture, averaged over the two model scales; n is the clean prompt count. The grey band is the seed-only floor Φb of §4, from the median over all 37 seed triplets to the median over the four of §4; Φb is measured on the full benchmarks, ∆b on the clean subsets (Appendix C). WildGuard-benign and SaladBench-attack do not clear their own upper edge; all fourteen clear twice the lower one. JailbreakBench-benign has 12 clean prompts and is excluded from (b) and from every aggregate. HarmBench-copyright spans 0–71 points (median 4), OR-Bench-hard 0.2–58 (median 2.8). We quote the four closest in configuration, which supply the widest of the 37 on the four benchmarks our argument needs, so our floor is a pessimistic tail: HarmBench-copyright 54–71 points, OR-Benchhard 21–58, while in-distribution accuracy moves under 1.1. They differ from the 395M arms in one config key (binary_mode) and their training pool, and at 149M also in class-loss weight and backbone, so we read them as this grid’s seed band at 395M and an off-scale estimate at 149M (Appendix C). Two effects clear that floor, both separations in level rather than differences of means: over the twelve runs of those four recipes HarmBench-copyright never exceeded 73, while every policy-Alast arm reaches 100 at 149M and 74–100 at 395M; OR-Bench-hard never fell below 11.2%, while those arms sit at 1.6–3.3% and 2.5–3.8%. Both are read on the full benchmarks, as the floor is; the 395M margins are narrow and do not survive decontaminating the arms. The group gaps of §5 are inside the band, which is why it rests on a rank statistic. The qualitative split survives: the top of the ordering turns on contested cases (copyright reproduction, refusal-eliciting questions), the bottom on explicit requests for weapons and violence. Its middle does not, and neither does the ordering by ∆b /Φb (HarmBench-copyright 1.7× its floor, OR-Bench-hard 1.5×), uncorrelated with this one under the median floor of the other 33 triplets. We claim only the split. A system reporting HarmBench-copyright = 100 and another reporting 0 are therefore not one better and one worse: on our grid those numbers come from one architecture and codebase differing in whose rubric supervised them last, and §6 shows two released guards doing the same. Stability across our two scales is partial (ρ = 0.71), and the scales are not independent.

5

Third audit: a benchmark score is steerable at 0.03% of the parameters

A score movable across its entire range by 0.03% of a model’s parameters, at unchanged detection capability and with no trace in a data card, is not measuring the model. HarmBench-copyright is such a score. If the definition travelled with the data, mixing enough policy-A data into a distilled student’s pool would install policy A. It does not. Figure 4 (Appendix A) holds the 149M student fixed and varies only the route. Mixing policy-A examples into the training pool and fully finetuning 149M parameters (Table 1) leaves HarmBench-copyright at 0.0–1.3 across E2–E5, with overrefusal on OR-Bench-hard between 37.7% and 56.4%. Making the hard labels entirely policy A 5

Table 1: All ten arms at both scales, on decontaminated subsets. VAL: accuracy on the one policy-A split that grades every arm, itself 69.3% leaked (§7); C OPY: HarmBench-copyright (n=75); OR- H: OR-Bench-hard over-refusal (n=236), lower better; C AP: mean catch rate over the six capabilityloaded benchmarks. The policy-A-last arms take one extreme of C OPY and OR- H at both scales; the row blocks are the 149M grouping. Column definitions, effective n, pool sizes and initialisations: Appendix A. 149M student

395M teacher

Policy-A route

Arm

VAL

C OPY

OR- H

C AP

VAL

C OPY

OR- H

C AP

blend blend + anchors filter only blend + anchors inverse blend all policy-A hard labels

E2 E3 E4 E5 E9 E6

88.8 85.9 85.7 88.6 78.5 90.1

0 1 0 0 31 17

37.7 56.4 53.4 40.3 35.6 31.4

88.1 88.3 88.0 86.8 82.6 80.0

88.9 86.6 86.7 89.8 94.4 97.2

1 7 15 5 67 80

24.2 70.8 71.2 38.6 11.4 10.6

83.2 88.4 84.9 91.3 80.0 90.1

policy-A KD teacher

E8

97.3

100

6.4

91.1

87.5

8

69.1

89.6

head-cal, 8.5k head-cal, 53k head-cal, 8.5k (A base)

E1 E7 E10

87.9 96.0 97.1

100 100 100

4.7 6.8 3.4

81.0 89.1 83.5

90.3 96.5 97.2

87 73 100

9.3 12.7 10.2

85.6 91.3 89.8

(E6) reaches only 17.3 and 31.4%, and the inverse blend (E9) 30.7 and 35.6%, because both are still distilled from a policy-B teacher whose soft labels override their own data. Two routes work, and both put the policy in the last signal before the output. Swapping the distillation teacher for a policy-A one while leaving every hard label policy B (E8) gives 100.0 and 6.4%; that teacher matches the policy-B one on backbone, lineage, seed and schedule but saw half as much data (Appendix A). Freezing the backbone and retraining only the three linear heads (0.04M of 149.05M parameters, 0.03%, on 8,489 calibration examples: 5,489 label-disagreement cases plus 3,000 GENERIC _ SAFE anchors) gives 100.0 and 4.7% (E1); extending it to 53,395 examples (E7) changes little (100.0, 6.8%), and running the same pass on a policy-A backbone (E10) gives 100.0 and 3.4%. A rank test, because one seed per arm does not license a difference of means. Given §4’s floor we test the grouping, not the gap. Call an arm policy-A-last if the supervision signal nearest its output carries policy A. At 149M every arm is a distilled student, so that set is {E1, E7, E10} (head calibration) plus E8 (policy-A teacher). At 395M no arm is distilled, so nothing overrides the hard labels and the set gains E6 and E9, whose labels are policy-A-dominant. That 395M assignment was revised after the 149M definition was applied there and failed; the revision follows from the configs, so we read the 395M p as confirmatory and not as pre-specified. Let A be that set, k = |A|. Under the null that the route is irrelevant, arm ranks are exchangeable, so ( )−1 [ ] 10 Pr A occupies the k extreme ranks of 10 = , (2) k which is 0.005 (k=4, 149M) and 0.004 (k=5, 395M). They do, on HarmBench-copyright and on OR-Bench-hard at both scales, with no arm out of place, separating by 69.3 and 52.0 points and by 24.6 and 11.4, though only the copyright half is mapping-independent (§6). The 395M column is the positive control: there policy-A hard labels do install policy A (E6 80 and 10.6%, E9 67 and 11.4%) and E8 inverts, falling back to its policy-B data (8, 69.1%). The claim is not that data cannot carry a definition, but that a teacher’s soft labels override the student’s hard labels, and a head-only pass does the same at 0.03% of the cost. Equation (2) is its loosest reading, though: E1, E7 and E10 are one unit, the statistic is two-sided in practice, and the two benchmarks are one test. Clustered and two-sided, p = 0.071 and 0.036; dropping the head-calibration arms leaves all four separations intact at p = 0.14 and 0.048, which 395M survives and 149M does not (Appendix A). The move is close to free on capability: 91.1% mean catch rate over the six capability-loaded benchmarks for E8 and 89.1% for E7 against 88.3–86.8% for the data-mixing arms, with E1 the exception at 81.0% (smallest calibration set, weakest starting representation). Why this matters for a released checkpoint. A team that changes its labelling policy, regenerates its data, and distils from a checkpoint trained under the old policy ships the old policy while believing otherwise, and no in-distribution metric will tell them (§7). Teacher provenance is a confound in 6

any distilled guardrail, and appears in no model card. We hit this failure, and an early calibration pass also erased GENERIC _ SAFE outright (Appendix A): the same lever that makes a policy cheap to swap makes it cheap to destroy.

6

The same disagreement, between independently built systems

If the argument holds it should be visible between labs, so we ran four current open guardrails (Qwen3Guard-Gen at 0.6/4/8B and Granite Guardian 3.1-2B [25, 22]) over the identical decontaminated prompts in each system’s own verdict format (33,137 each, zero unparsed). The comparison (Table 2, Figure 5). The two “Ours” rows are single arms, E8 at 149M and E10 at 395M, both selected on these same benchmarks. On capability-loaded benchmarks E8 is level with the 8B guard in its loose mode (91.1 vs 91.8) and behind it in strict (96.7), at 1/55 of the parameters; that parity is E8’s alone, as our four policy-A-last 149M arms span 81.0–91.1. The benign side does not depend on the arm: all four sit at 4.5–6.3% over-refusal against 29.7–30.5% for the Qwen3Guard sizes and 65.2% for Granite Guardian, and 6.4% against 55.9% and 97.0% on OR-Bench-hard alone, though on its 25 clean prompts our jury left unclear Qwen3Guard falls to 4.0–16.0% against our 4.0% while Granite stays at 84.0%. Tail latency separates us from the Qwen3Guard decoders (p95 16.1 against 94.0 ms) but not from Granite (15.6/16.4). Two caveats beyond §9: the policy-loaded column (90.9 vs 56.1–72.4) mostly records that these benchmarks encode rulings close to our policy A, this paper’s point rather than a result about detection; and deploy size and latency are not like-for-like. The disagreement. Read the other way, this replicates §4 between labs. On the 75 clean HarmBench-copyright prompts Granite Guardian scores 0 and Qwen3Guard-4B scores 99 on identical inputs. Neither is broken: Granite Guardian’s shipped harm risk definition, the one our runs invoked, asks whether a message is “harmful by common-sense” [22]; Qwen3Guard’s category list names Copyright Violation. The benchmark measures which taxonomy the vendor wrote down. Qwen3Guard makes the point again, in a released model: it emits three verdicts (Safe, Unsafe and Controversial), closely analogous to the contested middle class our taxonomy calls BOUNDARY _ SAFE [25] and the closest prior art on the mapping half, which already reports much of it (Appendix B). Where a deployer maps that third class is a policy choice made after training, with no retraining and no data. The vendor names both conventions, strict (Controversial counted unsafe) and loose, prints both and declares no default; the rows we quote are loose, and under strict the 8B model moves from 29.7% to 48.5% over-refusal, 72.4 to 86.8 policy and 91.8 to 96.7 capability. Our own binary is the same lever and a larger one: counting BOUNDARY _ SAFE unsafe takes over-refusal from 5.7% to 96.0% at 149M and 8.5% to 97.6% at 395M, while HarmBench-copyright is exactly invariant and OR-Bench-hard saturates: §5’s rank test survives on both, but only copyright keeps its margin (Table 2).

7

Fourth audit: validation accuracy tracks policy, not capability

The standard defence is the held-out split, and it fails here. One fixed policy-A-labelled split, the multilingual validation file, grades all twenty arms, including the fourteen trained on policy B, and still ranks them by policy. It is also not clean: §3’s audit run on our own validation data finds 69.3% of it verbatim inside the policy-A training file, per-arm exposure 10.9–69.3% (Appendix A). Exposure does not itself produce the ordering (rank correlation with VAL is −0.15 and +0.16), but VAL is a policy-agreement number measured with leakage. A clean split existed and was not used: the monolingual validation file is 0.2% inside the training file. Across the twenty models (Figure 3, Table 1) validation accuracy correlates ρ = +0.74 with the policy-loaded benchmarks (HarmBench-copyright, DoNotAnswer, SaladBench-defense) and ρ = +0.34 with the capability-loaded ones (AdvBench, HarmBench-standard, JailbreakBenchharmful, StrongREJECT, SaladBench-attack, WildGuard-harmful; criterion in Appendix A), holding within each scale (+0.64/+0.32 student, +0.83/+0.37 teacher); dropping each set’s most influential member gives +0.53 and +0.07. At n=20 the intervals overlap, so we claim the ordering, not the ratio. The contrast is between blocks, which is the mechanism and not a confound: VAL tracks the route (+0.78), which tracks the policy-loaded set (+0.86) and not the capability-loaded one (+0.13); inside either block it favours neither (Appendix A). 7

8

What we think should change

The benchmarks are fine; what is missing are three columns, and all three are cheap. (1) Run the overlap check and print it. An exact-match hash is an afternoon; near-duplicate matching is the honest version. Ours changed which results we were willing to state, and we had to run it twice: the first pass covered the benchmarks but not our own validation split. Assume overlap until measured. (2) Publish a seed floor. A benchmark delta is uninterpretable without one, and ours (54–71 points on HarmBench-copyright) invalidated half of what we first wrote down. Run three seeds of more than one recipe, and report the spread. (3) State the operational definition, and say which split you are quoting. A category list does not determine the decisions: both our pipelines used the same 24 pillars and disagreed on one prompt in six, and Qwen3Guard and Granite Guardian disagree by 99 points on one benchmark (§6). What determines them is the rubric’s rulings on contested cases (copyright reproduction, refusal-eliciting questions, fiction depicting harm, already-defended prompts) plus, for any guard with a contested class, ours included, where that verdict is mapped. A split drawn from your own labelling pipeline measures label fit, not capability, and a differentlysourced one is necessary but not sufficient: ours is differently sourced for fourteen of twenty arms and still ranks by policy. Guardrail evaluation needs those three columns more than it needs new benchmarks: prior exposure, the seed floor, and the definition a model was graded against.

9

Limitations

Variance is the main weakness: each arm is a single seed, and the seed floor of §4 comes from neighbouring recipes rather than the arms themselves, in a direction we cannot test, so the middle of Figure 2(b) is unresolved; we rest §5 on a rank statistic and on the two separations in level that clear it. The head-calibration arms were trained with no seed set and selected on training loss (Appendix A). Scope: twenty ModernBERT encoders, one proprietary 24-pillar taxonomy, two rubrics (a two-point sample) differing in annotation modality as well as in rubric, so the specific ordering may not transfer, though the qualitative claim should and §6 reproduces it with neither of our pipelines involved. The arms are not matched on pool size (8,489–144,744) or checkpointselection rule, and at 395M pool size and route are collinear. The overlap audit is a lower bound (§3). Metric conventions: a benchmark-harmful prompt counts as caught if we flag it, so catch rate measures agreement with that benchmark’s rubric, which is the phenomenon under study and why our numbers are not comparable to published ones; our binary also counts BOUNDARY _ SAFE safe, worth 90 points of over-refusal (§6). HarmBench-copyright is two templates: its 75 clean prompts are 39 generate the lyrics of X and 36 generate a long passage from X, so its 0.99 range is one contested ruling read at n=75, not a broad construct. The SOTA comparison is asymmetric: our arms were developed against these benchmarks, the baselines were not, each system is scored under its own verdict convention, and decontamination is defined by our pools only. Neither baseline vendor publishes a train–test overlap check for these benchmarks, so their own exposure is unknown to us and the comparison is one-sided in a direction we cannot bound, which is itself an instance of what recommendation (1) asks releases to report. Taxonomy: pillar macro-F1 enters none of our claims. We do not fix generalisation: pooled at its deployment threshold the shipped model runs at precision 0.968 / recall 0.496.

10

Related work

Guardrails. LlamaGuard [12], ShieldGemma [36], WildGuard [7] and Aegis [5] established the fine-tuned-classifier-as-guardrail pattern; all ship a fixed taxonomy, and LlamaGuard’s customcategory parameter is cosmetic in independent reports [19], which we did not re-run. Closest to our concern, the Granite Guardian team reports that a definitional distinction, between using and mentioning harmful content, was the main source of error in early versions of its social-bias detector, and encodes it through taxonomy-driven synthetic data [18]; we isolate such a choice by training on two labellings of the same prompts. Annotator disagreement and the cost of aggregating it away are well established [23, 24, 31], including for toxicity [29, 6]; we add its behaviour under training, as a separable component installed by the final supervision signal. Construct validity is rarely established [26, 13, 1, 16]: for guardrails the assumption that unsafe is annotator-independent is loadbearing, false and never stated, and we give a quantitative instance. What transfers: distillation 8

carries bias and calibration [8, 20, 9] and other behavioural traits [3]; we add the teacher’s policy, overriding the student’s own hard labels. That small parameter subsets steer behaviour underlies adapters and LoRA, and head-only calibration is last-layer retraining on frozen features [10, 11, 14].

References [1] Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACLHLT), pages 4843–4855. Association for Computational Linguistics, June 2021. doi: 10.18653/ v1/2021.naacl-main.385. URL https://aclanthology.org/2021.naacl-main.385/. [2] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. JailbreakBench: An open robustness benchmark for jailbreaking large language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Datasets and Benchmarks Track, 2024. URL https://arxiv.org/ abs/2404.01318. [3] Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Sören Mindermann, Jacob Hilton, Samuel Marks, and Owain Evans. Language models transmit behavioural traits through hidden signals in data. Nature, 652(8110):615–621, April 2026. doi: 10.1038/s41586-026-10319-8. URL https://www.nature.com/articles/ s41586-026-10319-8. [4] Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An over-refusal benchmark for large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML). PMLR, 2025. URL https://arxiv.org/abs/2405.20947. [5] Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AEGIS: Online adaptive AI content safety moderation with ensemble of LLM experts. arXiv preprint arXiv:2404.05993, 2024. URL https://arxiv.org/abs/2404.05993. [6] Nitesh Goyal, Ian D. Kivlichan, Rachel Rosen, and Lucy Vasserman. Is your toxicity my toxicity? Exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–28, November 2022. doi: 10.1145/3555088. URL https://doi.org/10.1145/3555088. [7] Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Datasets and Benchmarks Track, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 0f69b4b96a46f284b726fbd70f74fb3b-Abstract-Datasets_and_Benchmarks_Track. html. [8] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531. Presented at the NIPS 2014 Deep Learning Workshop. [9] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models. arXiv preprint arXiv:2010.03058, 2020. URL https: //arxiv.org/abs/2010.03058. [10] Neil Houlsby, Andrei Giurgiu, Stanisław Jastrz˛ebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR, 2019. URL https://proceedings.mlr.press/v97/houlsby19a.html. [11] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9. 9

[12] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674, 2023. URL https://arxiv.org/abs/2312.06674. [13] Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), pages 375– 385. Association for Computing Machinery, 2021. doi: 10.1145/3442188.3445901. URL https://dl.acm.org/doi/10.1145/3442188.3445901. [14] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk. Oral presentation. [15] Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. SALAD-Bench: A hierarchical and comprehensive safety benchmark for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3923–3954, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-acl.235/. [16] Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? A meta review of evaluation failures across machine learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ 757b505cfd34c64c85ca5b5690ee5293-Abstract-round2.html. [17] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 35181–35224. PMLR, 2024. URL https://proceedings.mlr.press/v235/mazeika24a.html. [18] Manish Nagireddy, Inkit Padhi, Soumya Ghosh, and Prasanna Sattigeri. When in doubt, cascade: Towards building efficient and capable guardrails. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 1812–1821, 2025. doi: 10.1609/aies. v8i2.36676. URL https://doi.org/10.1609/aies.v8i2.36676. [19] Mahesh Kumar Nandwana, Youngwan Lim, Joseph Liu, Alex Yang, Varun Notibala, and Nishchaie Khanna. Taxonomy-adaptive moderation model with robust guardrails for large language models. arXiv preprint arXiv:2512.05339, 2025. URL https://arxiv.org/abs/ 2512.05339. To be presented at the AAAI-26 PerFM Workshop. [20] Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 2433fec2144ccf5fea1c9c5ebdbc3924-Abstract-Conference.html. [21] Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. Proving test set contamination in black-box language models. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id= KS8mIvetg2. Oral presentation. [22] Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, and Prasanna Sattigeri. Granite Guardian: Comprehensive LLM safeguarding. In 10

Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 607–615, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.naacl-industry.49. URL https://aclanthology. org/2025.naacl-industry.49/. Extended technical report: arXiv:2412.07724. [23] Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694, 2019. doi: 10. 1162/tacl_a_00293. URL https://aclanthology.org/Q19-1043/. [24] Barbara Plank. The “Problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 10671–10682, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.731. URL https://aclanthology.org/2022.emnlp-main.731/. [25] Qwen Team. Qwen3Guard technical report. arXiv preprint arXiv:2510.14276, October 2025. URL https://arxiv.org/abs/2510.14276. [26] Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ 084b6fbb10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html. [27] Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5377–5400, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.301. URL https://aclanthology.org/ 2024.naacl-long.301/. [28] Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.722. URL https://aclanthology.org/ 2023.findings-emnlp.722/. [29] Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 5884–5906, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.431. URL https://aclanthology.org/2022. naacl-main.431/. [30] Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id= KZLE5BaaOH. [31] Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72: 1385–1470, 2021. doi: 10.1613/jair.1.12752. URL https://jair.org/index.php/jair/ article/view/12752. [32] Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-Not-Answer: Evaluating safeguards in LLMs. In Findings of the Association for Computational Linguistics: 11

EACL 2024, pages 896–911, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-eacl.61/. [33] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.127. URL https://aclanthology.org/2025.acl-long.127/. [34] Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244, June 2024. URL https://arxiv.org/abs/2406.04244. [35] Richard J. Young. Evaluating the robustness of large language model safety guardrails against adversarial attacks. arXiv preprint arXiv:2511.22047, November 2025. URL https://arxiv. org/abs/2511.22047. [36] Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. ShieldGemma: Generative AI content moderation based on Gemma. arXiv preprint arXiv:2407.21772, 2024. URL https://arxiv.org/abs/2407.21772. [37] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. URL https://arxiv.org/abs/2307.15043.

A

Evaluation suite, aggregate membership, and arm provenance

Suite. Fifteen benchmarks, 33,137 prompts. Sizes: SaladBench-base 21,318; SaladBench-attack 5,000; OR-Bench-hard 1,319; AdvBench 1,094; WildGuard-benign 945; DoNotAnswer 939; WildGuard-harmful 754; OR-Bench-toxic 655; StrongREJECT 313; HarmBench-standard 200; SaladBench-defense 200; HarmBench-contextual, HarmBench-copyright, JailbreakBench-harmful, JailbreakBench-benign 100 each. Clean counts after decontamination are printed in Figure 2. HarmBench-copyright’s 75 clean prompts are two prompt templates, and at 149M seven of the ten arms of Table 1 score exactly 0/75 or 75/75, so its effective n there is near two. The VAL column of that table is the multilingual validation file, 7,662 rows and 7,658 unique after normalisation. Aggregates. Policy-loaded (used for the ρ = +0.74 column): HarmBench-copyright, DoNotAnswer and SaladBench-defense, the three whose scoring turns on a contested ruling (copyright reproduction, refusal-eliciting questions, already-defended prompts). Capability-loaded (ρ = +0.34): AdvBench, HarmBench-standard, JailbreakBench-harmful, StrongREJECT, SaladBench-attack and WildGuard-harmful, all explicit requests for weapons, malware or violence. Membership was chosen by that criterion, not by a threshold on Figure 2(b). Over-refusal (the OVER - REF column of Table 2): OR-Bench-hard and WildGuard-benign, the two benign benchmarks that survive decontamination. JailbreakBench-benign would be the third and is excluded at 12 clean prompts; including it would narrow the headline over-refusal gap over the 8B guard from 24.0 to 16.0 points. ORBench-toxic fails the criterion in both directions: it is scored as a catch rate, but its prompts are the toxic half of an over-refusal benchmark and the two rubrics rule on them differently, so it is neither an uncontested request for harm nor a benign prompt. The exclusion is load-bearing and we report both figures: including it in the policy-loaded set moves ρ = +0.74 to +0.61. SaladBench-base and HarmBench-contextual are in neither. The policy-loaded set is a construct-validity grouping, not a claim that all three of its members separate the arms: on decontaminated subsets HarmBenchcopyright separates the policy-A-last arms by 69.3 points at 149M and 52.0 at 395M, DoNotAnswer by 6.2 at 149M and not at all at 395M, and SaladBench-defense at neither scale. The body therefore names HarmBench-copyright and OR-Bench-hard individually rather than by that aggregate label. 12

Per-benchmark correlation with VAL. Spearman across the twenty arms: DoNotAnswer +0.74, AdvBench +0.74, HarmBench-contextual +0.71, HarmBench-copyright +0.66, SaladBench-attack +0.57, WildGuard-benign +0.41, SaladBench-defense +0.38, SaladBench-base +0.29, WildGuardharmful +0.16, JailbreakBench-harmful +0.11, HarmBench-standard −0.25, StrongREJECT −0.43, OR-Bench-toxic −0.71, OR-Bench-hard −0.74. The dichotomy is coarser than the data, and its coarseness runs against us: AdvBench, a capability member, is tied with the best policy member, which inflates the capability aggregate and shrinks the contrast we report; HarmBench-standard and StrongREJECT are what hold that aggregate down.

0.900

ρ = +0.34

policy-loaded benchmarks, score

capability-loaded benchmarks, catch rate

The contrast is between route blocks. Within the eleven policy-B-last arms VAL correlates −0.03 with the capability aggregate and −0.04 with the policy aggregate; within the nine policyA-last arms, +0.68 and +0.77. The block difference in the policy correlation is supported at about p = 0.045 (bootstrap, 4.5% of draws at or below zero). Partialling the route indicator out of the pooled correlations leaves both associations between +0.2 and +0.4, neither distinguishable from zero, and their ordering undetermined at n=20: one estimator (rank-transform, then residualise on the route) puts capability above policy, another (rank within each block, then pool) puts them level, and the bootstrap interval on the difference spans zero. We therefore claim the block effect and not an ordering within blocks.

0.875 0.850 0.825

0.9

ρ = +0.74

0.8 0.7 0.6 0.5 0.4

0.800 0.80

0.85 0.90 0.95 validation accuracy on the one policy-A split that grades all arms

0.80

0.85 0.90 0.95 validation accuracy on the one policy-A split that grades all arms

policy-A supervision is last

policy-B supervision is last

Figure 3: Accuracy on the one policy-A split that grades all 20 arms, against the two things it might be measuring. It tracks agreement with the annotator’s policy more closely than detection capability, though at n=20 neither association is resolved on its own (Appendix A). Marker shape gives the route block, which is most of what the correlation is. Policy-B jury. Four instruct models (two Gemma-family, two Qwen-family), each sampled three times at T =0.7, top-p=0.95, so every round carries exactly 12 votes. The deployed output holds 144,727 records and 404,167 round instances, with 12 votes in every one of them and 1,212,501 votes per model; 0.06% of votes failed to parse. The cascade is three binary rounds: LEGITIMATE/TRUNCATED, then SAFE/UNSAFE, then either BOUNDARY _ SAFE/GENERIC _ SAFE or COMPOUND / HARMFUL . Cross-model agreement with the round majority is 94.6% over all valid votes, 95.7% over rounds with an unambiguous majority, and it splits by round the same way the per-class retention in §2 does: 98.2% on the UNSAFE branch against 91.6% on the SAFE branch, which is where BOUNDARY _ SAFE is decided. Rather than force a label, 9,431 tied rounds route 5,632 records (3.9%) to a separate ambiguous bucket and a further 12,191 are dropped as truncated, so 12.3% of records never receive a policy-B label. (An earlier draft quoted 85.6% here. That figure is top-1 accuracy for a different, five-model ensemble at the 24-pillar labelling stage, measured against a 2,400-sample gold set, and is not jury agreement.) Arm provenance. Read from the training configs. All arms use seed 42 where a seed is set; finetune_head_only.py sets none, never reads the validation split, and selects on training loss. Its checkpoints nonetheless record seed: 42 in their saved config, because the script writes the base checkpoint’s config dictionary through unchanged; the calibration pass itself is unseeded. E1 and E10 train on the same file (calib_E1_balanced.jsonl, 8,489 rows: 5,489 label-disagreement 13

One 149 M student, four routes to install the same policy (decontaminated) policy-A examples mixed into the training data E3 E4 E2 E5

policy A adopted, to the right 1.3 0.0 0.0 0.0

hard labels are mostly or all policy-A; teacher is not E6 E9

17.3 30.7

teacher soft labels are policy-A E8

100.0

head-only calibration, backbone frozen E1 E7 E10 0

50

fewer false blocks, to the left

E3 E4 E2 E5

56.4 53.4 37.7 40.3

E6 E9

31.4 35.6

E8

6.4

100.0 E1 100.0 E7 100.0 E10

4.7 6.8 3.4

100

HarmBench-copyright catch (%), n=75

0

50

100

OR-Bench-hard over-refusal (%), n=236

Figure 4: One 149M student, four routes to install the same policy, on decontaminated subsets. Datalevel interventions (orange) do not install policy A even when every hard label carries it; a policy-A distillation teacher or a frozen-backbone head-only pass over 0.03% of the parameters (blue) does. cases plus 3,000 GENERIC _ SAFE anchors) and differ only in which checkpoint they initialise from. The explicit policy-refusal anchors are 1,450 rows of E3’s 144,744 (1.0%) and 454 of E5’s 47,287 (1.0%); E2 and E4 carry none. All 1,904 are drawn from that same 5,489-row disagreement set and all carry policy A’s unsafe label. The 3,000 anchors were added after an earlier calibration set of disagreement cases only, containing zero GENERIC _ SAFE examples, erased that class entirely (526 of 526 re-routed to BOUNDARY _ SAFE) and cut 4-class validation accuracy from 87% to 57.9%; the anchors restored it to 87.9%. At 395M no arm has a KD teacher, which is why E6 and E9 join the policy-A-last set there and E8 leaves it (§5). The seven full fine-tune arms train for 5 epochs at 149M and 2 at 395M; the three head-only arms train for 5 epochs at both scales, because the orchestrator hardcodes --epochs 5 --lr 5e-5 for every head-only entry. Learning rate is 5e-5 at 149M and for every head-only arm, and 3e-6 for the 395M full arms; the full arms use warmup fraction 0.06 and the head-only arms no warmup. finetune_head_only.py takes no validation file and saves on the epoch-average training loss, which in five of the six head-only runs selected the final epoch, that is, selected nothing. The sixth, E1 at 149M, stopped after 2 of its 5 epochs and was never re-run (its best.pt records epoch_at_best=2); it is therefore the least-trained head-only arm, and it still sits at the policy-A extreme of both HarmBench-copyright and OR-Bench-hard (100.0 and 4.7%), so the shortfall runs against the claim rather than for it. The rank test, in full. Equation (2) treats the ten arms as exchangeable and fixes the extreme end in (advance. It does neither well, so( we) give the ladder. (i) As printed, one-sided, arms exchangeable: ) 1/ 10 = 0.0048 (149M) and 1/ 10 = 0.0040 (395M). (ii) Two-sided, since A takes the top of 4 5 HarmBench-copyright and the bottom of OR-Bench-hard and which end was never fixed: 0.0095 and( 0.0079. (iii) Clustering ) ( ) E1, E7 and E10 into one unit and permuting the eight resulting units: 1/ 82 = 0.036 and 1/ 83 = 0.018 one-sided, 0.071 and 0.036 two-sided. The clustering is not arbitrary: those three arms are three cells of one 2 × 2, E1 and E7 sharing a base checkpoint and differing only in the calibration file, E1 and E10 sharing the calibration file and differing only in the base checkpoint, all three run by one script with no seed and no validation. (iv) Dropping the three head-calibration arms altogether, which removes the frozen-backbone and checkpoint-selection differences with them: A = {E8} at 149M (gaps +69.3 and +25.0 points, p = 1/7 = 0.14) and A = {E6, E9} at 395M (gaps +52.0 and +12.7, p = 1/21 = 0.048). The same values follow from clustering under a size-respecting label permutation, so two defensible clustered conventions differ by a factor of four and we name the one we quote. The 395M column survives (iv) and the 149M column does not. 14

catch rate, capability-loaded benchmarks (%), better upward

ours, 'boundary_safe' Qwen3Guard 8 B, vendor strict mode counted unsafe 100

Ours 149 M (E8) 95

85

80

Ours 395 M

100

Ours 149 M

100

Qwen3Guard 4 B Granite Guardian 2.5 B

99

Qwen3Guard 8 B

95

90

Ours 395 M (E10)

(b)

Qwen3Guard 0.6 / 4 / 8 B, loose mode Qwen3Guard 0.6 B the other nine 149 M arms

87

0

Granite Guardian 2.5 B

(a) 0

25

50

75

100

over-refusal on benign prompts (%), worse to the right

0

50

100

HarmBench-copyright catch rate (%) same 75 prompts, every system

Figure 5: (a) the deployment frontier: over-refusal against detection, bubble area proportional to parameters. Ours are two single arms selected on these same benchmarks, E8 at 149M and E10 at 395M; the grey points are the other nine 149M arms (catch 80.0 to 89.1, over-refusal 4.5 to 29.8). Both dashed arrows move one contested class and retrain nothing: in orange, the 8B guard’s Controversial verdict counted unsafe, which is that vendor’s strict mode, 29.7 to 48.5; in blue, our own BOUNDARY _ SAFE counted unsafe, 5.7 to 96.0, with catch 91.1 to 97.2. Our convention is worth 90.3 points where the one we criticise is worth 18.8. The Qwen3Guard points are its loose mode. (b) one benchmark, six systems, 75 identical prompts: whether reproducing copyrighted text counts as harm is a taxonomy decision, and these systems made different ones. The four reported cells are not four tests. Spearman across the ten arms between HarmBenchcopyright and OR-Bench-hard is −0.868 (149M) and −0.745 (395M), and between the two scales +0.716 and +0.758. More sharply, the exact joint permutation (does A occupy the extreme ranks on both benchmarks?) returns exactly the marginal p at every setting above, because exactly one subset of the right size qualifies on each benchmark and it is the same subset: the second benchmark contributes no independent evidence. A Bonferroni correction over four cells (0.019 and 0.016) would therefore claim more tests than were run. The 395M grouping was revised after the 149M definition was applied there and failed: with A = {E1, E7, E8, E10} the 395M arms hit 3/4 on HarmBench-copyright and 2/4 on OR-Bench-hard. The revision follows from the configs, since no 395M training config carries a teacher_ckpt key and all seven 149M ones do, so we regard it as principled; it was not pre-specified, and we read the 395M p as confirmatory. The two distillation teachers (E8). Both are ModernBERT-large warm-started from the round3 checkpoint of the same recipe (r3b_warm_augmented_drift_s42, each at its own data version) at seed 42, 2 epochs, learning rate 3 × 10−6 , so lineage, backbone and schedule are matched. Data volume is not. The policy-A teacher (r4b_warm_r3b_s42_v3_5, on round3_train_r4_augmented_clean.jsonl) saw 67,352 unique prompts after normalisation; the policy-B teacher (r4b_warm_r3b_s42_v10, on train_v10_balanced.jsonl) saw 138,805, a factor of 2.06, and 107,044 of those (77.1% of that pool, 32,715 of them harmful-class) never appear in the policy-A teacher’s pool. E8’s contrast is a teacher swap, not a single-variable one, though the confound runs against the effect: the teacher that installs policy A is the one trained on less data. Pool size and route. At 395M the two are collinear: Spearman ρ(pool, route) = −0.738 over the ten arms and −0.805 among the seven full fine-tunes, giving ρ(pool, OR-Bench-hard) = +0.917 and ρ(pool, HarmBench-copyright) = −0.654. At 149M it is absent (−0.215; +0.416 among the 15

Table 2: Our guardrail against current open guards [25, 22], on identical decontaminated prompts, same host. The two “Ours” rows are single arms, E8 at 149M and E10 at 395M, each selected on these same benchmarks; across our four policy-A-last 149M arms C AP spans 81.0–91.1 and OVER - REF 4.5–6.3. C AP and P OLICY are means over the capability-loaded and policy-loaded sets (Appendix A); OVER - REF is the mean over the two usable benign benchmarks (JailbreakBenchbenign is excluded at 12 clean prompts); C OPY is HarmBench-copyright (n=75). Bold is best in column among the main systems; C OPY is deliberately unbolded, because which end of it is better is the policy choice under study. D EPLOY is on-disk size in decimal GB and mixes precisions: ours is a measured INT8 ONNX export (150.1 MB, taken from the shipped 149M checkpoint, which shares E8’s architecture and 149.05M parameters), the baselines are the vendors’ released bf16 weights, and the 397 MB teacher cell is extrapolated at 1.007 bytes per parameter because no teacher export exists. Latency is median over 180 single-prompt requests on one H100 and is not like-for-like either: ours is eager PyTorch on the fp32 checkpoint padded to 512 tokens, the baselines run under vLLM with CUDA graphs, and the mismatch favours the baselines. The same checkpoint under a CUDA graph runs at 6.4 ms at that sequence length and 2.6 ms at a 64-token bucket (the prompts median 14 tokens). Decontamination is defined by our pools only. The retained subset is harder for every system, raising OR-Bench-hard over-refusal by 3.0 points for our student, 7.4 for our teacher and 7.3–19.1 for the baselines, so the clean OR-Bench-hard gap against Qwen3Guard-8B (49.6 points) is wider than the full-benchmark gap (34.3), which is itself inflated in our favour by that benchmark’s 82% training overlap. Size-matched random subsets of the same benchmark (2,000 draws) move that gap by +0.1 points (sd 2.9), so the widening is a property of which prompts our pools contain and not of subsetting. The lower block applies one lever to every system that emits a contested verdict, with no retraining and no data: Qwen3Guard’s Controversial counted unsafe (the vendor’s strict mode, against its loose mode above) and our BOUNDARY _ SAFE counted unsafe. Granite Guardian emits no third class and so has one row. C OPY does not move for our arms because no arm assigns any of the 75 prompts to BOUNDARY _ SAFE: 0.00% at all twenty arms, and at all 100 prompts before decontamination. OVER - REF saturates instead: under the remap all ten arms land between 97.9% and 100.0% on OR-Bench-hard at both scales, against 3.4–56.4 and 9.3–71.2 under the shipped mapping, and the policy-A-last group of §5 moves from 24.6 and 11.4 points below the rest to 0.4 points above it. System

Params

Deploy ms (b=1)

C AP P OLICY OVER - REF C OPY

contested verdict counted safe: our shipped binary, Qwen3Guard loose mode Ours, 149 M student 149 M 150 MB 16.0 91.1 90.9 Ours, 395 M teacher 395 M 397 MB 19.9 89.8 92.4 Qwen3Guard-Gen 0.6 B 750 M 1.5 GB 33.2 91.4 69.6 Qwen3Guard-Gen 4 B 4.4 B 8.8 GB 53.7 91.5 70.7 Qwen3Guard-Gen 8 B 8.2 B 16.4 GB 78.6 91.8 72.4 Granite Guardian 3.1 2.5 B 5.1 GB 15.6 95.1 56.1

5.7 8.5 30.5 30.5 29.7 65.2

contested verdict counted unsafe: BOUNDARY _ SAFE → unsafe, Qwen3Guard strict mode Ours, 149 M student 149 M 150 MB 16.0 97.2 93.8 96.0 Ours, 395 M teacher 395 M 397 MB 19.9 97.6 96.2 97.6 Qwen3Guard-Gen 0.6 B 750 M 1.5 GB 33.2 96.2 83.4 44.2 Qwen3Guard-Gen 4 B 4.4 B 8.8 GB 53.7 96.6 86.2 49.1 Qwen3Guard-Gen 8 B 8.2 B 16.4 GB 78.6 96.7 86.8 48.5

100 100 87 99 95 0 100 100 93 100 100

full fine-tunes). Route reproduces the extreme-k grouping exactly at both scales and pool size does not: the five smallest pools are {E1, E10, E2, E6, E9} against an observed extreme five of {E1, E10, E6, E7, E9}, E7 (53,395 rows) sits on the policy-A side, and at 149M the four policy-A-last arms span 8,489–138,823 rows inside a 3.4-point over-refusal band. The honest caveat: within the mixing arms pool size does track over-refusal (ρ = +0.971 at 149M, n=6; +0.912 at 395M), so the OR-Bench-hard magnitudes, unlike the grouping, are partly a pool effect. Jury agreement. The relabelling jury’s per-model votes were retained, so a chance-corrected statistic is computable and we compute one. Over each jury model’s modal vote, Fleiss κ = 0.78 on the binary safe/unsafe round (N =132,470 items; 0.82 treating every vote as a separate rater), 0.70 on the BOUNDARY _ SAFE versus GENERIC _ SAFE refinement (N =55,381) and 0.87 on COMPOUND versus HARMFUL (N =71,507). Within-model self-consistency across three samples is 93.6–99.7% and pairwise raw agreement between models 84.1–94.4%, so the residual disagreement is between 16

Table 3: Arm provenance, read from the training configs. Arm

Policy-A route

Pool rows

Initialisation

KD teacher (149M only)

E1 E10 E7 E2 E5 E3 E4 E6 E9 E8

head-only calibration head-only calibration head-only calibration blend blend + anchors blend + anchors filter only all hard labels inverse blend teacher only

8,489 8,489 53,395 47,287 47,287 144,744 138,638 40,327 40,327 138,823

policy-B ckpt policy-A ckpt policy-B ckpt v2 warm-start v2 warm-start v2 warm-start v2 warm-start v2 warm-start v2 warm-start v2 warm-start

n/a (backbone frozen) n/a (backbone frozen) n/a (backbone frozen) policy-B policy-B policy-B policy-B policy-B policy-B policy-A

models rather than sampling noise. Agreement is lowest on exactly the contested class this paper is about, and none of it speaks to correctness. Prompt-sampling dispersion in ∆b . Resampling prompts with replacement inside each benchmark and rescoring all ten arms at both scales on the same resample (B=2,000) gives SD(∆b ) between 0.26 (SaladBench-base) and 3.39 points (HarmBench-contextual), with 95% intervals of 99.3 [98.0, 100.0] on HarmBench-copyright, 57.4 [52.8, 62.9] on OR-Bench-hard, 33.4 [30.6, 36.4] on DoNotAnswer and 4.3 [3.5, 5.4] on WildGuard-benign. What makes the middle of Figure 2(b) unreadable is therefore seed variance, not prompt sampling. Two cautions: max − min is biased upward under resampling, by 0.0–2.0 points here and worst on the 100-prompt subsets; and the arms are scored on byte-identical prompt lists, so an independent-binomial floor would be the wrong null. Under the matched test every one of the 28 benchmark × scale extreme-arm gaps is significant, worst case exact McNemar p = 0.008. Per-arm benchmark exposure. Share of OR-Bench-hard’s 1,319 prompts present in each arm’s own training or calibration pool: E9 1.4%, E8 37.0%, E4 38.7%, E3 39.3%, E1/E10 40.3%, E6 63.2%, E2/E5/E7 73.8%. Rank correlation with the OR- H column of Table 1 is −0.01 (149M) and −0.21 (395M), computed on the percentages as printed here; the E2/E5/E7 tie is an artefact of that rounding, and with the unrounded overlaps (974, 973, 973 prompts) the 149M figure is 0.00. Effective exposure is far flatter than these ownpool numbers. Following init_from and teacher_ckpt through every training config, each arm also inherits the pools of the runs it was warm-started or distilled from, and all twenty lineages reach r2a_t6_everything_s42 (train_p3c_balanced_filt500.jsonl) and then p2_c_pillar_only (the original train.jsonl/val.jsonl split); those three files alone cover 81.0% of OR-Bench-hard. Over the whole traced lineage, effective exposure is 82.3% for seventeen of the twenty models, 81.7% for E8 at 149M (no warm start, only a v3_5 KD teacher) and 81.0% for the two E10 arms (calibrated on v3_5 bases), against the 82.1% of the benchmark that the union of all pools contaminates. Every other arm inherits through the v2 relabelling runs, so it is the shared ancestry, not any single intermediate pool, that flattens exposure. Union decontamination also costs nothing on this benchmark: no OR-Bench-hard prompt is removed only because of the policy-A head-calibration pools, and total contamination is 1,083/1,319 either way. Validation-split exposure. Share of the 7,658-prompt validation split present in each arm’s own training pool: E1/E10 10.9%, E2 23.2%, E5 24.2%, E8 35.9%, E3 36.8%, E4 37.0%, E9 40.0%, E6 52.8%, E7 69.3%. Rank correlation with the VAL column is −0.15 (149M) and +0.16 (395M).

B

Prior art on the contested middle class, and on contamination

Qwen3Guard [25] is the closest prior work to §6. Its §3.4.1 reports that labelling Controversial samples unsafe better matches Aegis while labelling them safe better matches OpenAIModeration; its §3.4.2 (Table 8, Qwen3Guard-Gen-4B only) ablates the class; and the captions of its results tables declare a per-benchmark optimal mode. Its §3.3 also reports a training-time effect: relative to a 1:1 Safe-to-Unsafe ratio, doubling the proportion of Safe examples makes the model more permissive, and it uses two such re-weighted models’ disagreement to construct the Controversial label. Two things separate that from our claim. First, there the lever is the class ratio inside one fixed taxonomy, used as label-construction machinery, and its effect is never measured on public safety 17

benchmarks; here the lever is the definition, and we measure it across fifteen decontaminated benchmarks and twenty models. Second, its distillation stage is motivated as annotation-noise reduction and makes no policy-transfer claim, whereas we find distillation carries the teacher’s policy over contrary hard labels. On contamination, the closest guardrail-specific work is Young [35], which evaluates ten released guards on 1,445 prompts across 21 attack categories and finds every one degrades on novel attacks, the largest gap being Qwen3Guard-8B’s fall from 91.0% to 33.8% and the smallest Granite-Guardian-3.2-5B’s 6.5 points; that inference is behavioural, where §3 measures corpus overlap directly. On taxonomy adaptation, given a 25-category in-house taxonomy at inference LlamaGuard3-8B scores 3.5% F1 against 84.9% on HarmBench [19].

C

Seed floors: all 37 triplets

A triplet is one recipe trained at seeds 42/43/44 with configs identical apart from run_id and seed (with ten exceptions noted below), scored on the fifteen-benchmark suite of Appendix A. The training tree holds 37 complete triplets, 111 runs; they are nine lineages at up to six data versions each, not 37 independent recipes. The four used in §4 are the v10x family, whose configs differ from arm E3’s in exactly one key, binary_mode (a 2-class rather than 4-class head), plus the training pool: initialisation, learning rate, epochs, backbone, class and pillar loss weights (0.01 and 1.0) and checkpoint-selection metric (pillar macro-F1, the default, which no config in the grid or the triplets overrides) are those of the 395M arms. In both families the benchmark verdict is read off the class head (eval_external_benchmarks.py), so at 395M the floor recipes’ head is supervised and selected exactly as the arms’ is. At 149M the arms use a class-loss weight of 1.0 and a smaller backbone, which is why we read the floor as off-scale there. Table 4 gives Φb over all 37 triplets against the value used in §4, with ∆b /Φb under each. Three things follow. Table 4: Seed floors on the suite of Appendix A, in points. ∆b is the label-supervision range of (1) on decontaminated subsets. The three Φb columns are the minimum, median and maximum seed spread over all 37 triplets; USED is the median over the four v10x recipes quoted in §4, the family that supplies the maximum of the 37 on the first two rows and on SaladBench-attack and WildGuard-benign. The last two columns give ∆b /Φb under that floor and under the median of all 37, both computed before rounding. Φb over all 37 triplets

∆b /Φb

Benchmark

∆b

min

median

max

USED

used

all-37

HarmBench-copyright OR-Bench-hard OR-Bench-toxic DoNotAnswer SaladBench-defense HarmBench-contextual SaladBench-attack SaladBench-base AdvBench StrongREJECT HarmBench-standard JailbreakBench-harmful WildGuard-harmful WildGuard-benign

99.3 57.4 38.5 33.4 31.0 29.0 25.2 21.7 19.8 15.5 14.1 13.1 8.9 4.3

0.0 0.2 0.6 0.1 0.5 1.0 0.3 0.2 0.2 0.3 0.0 0.0 0.4 0.1

4.0 2.8 3.1 2.0 4.5 5.0 3.4 1.6 1.2 1.0 1.5 2.0 1.5 0.6

71.0 58.1 28.9 18.8 39.5 14.0 42.2 32.4 50.2 44.7 53.0 49.0 21.1 10.7

58.5 38.6 18.2 3.9 7.0 10.0 37.0 11.1 12.6 6.1 5.8 7.0 5.9 5.0

1.70 1.49 2.12 8.46 4.43 2.90 0.68 1.96 1.57 2.55 2.45 1.87 1.51 0.85

24.83 20.47 12.60 16.48 6.89 5.80 7.51 13.58 16.63 16.12 9.39 6.53 6.10 6.73

The floor is a recipe-level choice, not a property of the benchmark. On HarmBench-copyright the median spread is 4.0 points over all 37 triplets and 3.0 over the 33 that are not v10x, against 58.5 over the four we use: a factor of 19.5 between the family we chose and the rest, and 24.2 on OR-Bench-hard (38.6 against 1.6). The choice runs against our own claims. The widest of the 37 triplets on HarmBench-copyright, OR-Bench-hard, SaladBench-attack and WildGuard-benign is in every case a v10x recipe, and those are exactly the four benchmarks §4 and §5 turn on. 18

The ordering by ∆b /Φb is not identified by the triplets available. Under the floor we use, SaladBench-attack (0.68×) and WildGuard-benign (0.85×) fall below 1. Under the median of all 37, of the other 33, or of the two on-path triplets named at the end of this appendix, nothing falls below 1, and those three orderings correlate with the printed one at Spearman −0.03, −0.08 and +0.26. The ratio also stops re-ranking: its correlation with raw ∆b rises from +0.24 under our floor to +0.58 and +0.59 under the first two. We claim the top/bottom split, not the order inside it. Levels, and what the level claim is scoped to. §4 compares arm levels to what the twelve v10x runs ever reached, on the full benchmarks, since that is where Φb is measured. On HarmBenchcopyright those runs span 2–73 and the policy-A-last arms sit at 100 (149M, all four) and 74–100 (395M); on OR-Bench-hard the runs span 11.2–72.3% and the arms 1.6–3.3% and 2.5–3.8%. At 395M the copyright margin is one point (74 against 73), and replacing the arm levels by their decontaminated values removes it (66.7 for E9); the same substitution puts two 395M arms above the OR-Bench-hard minimum (12.7% for E7, 11.4% for E9). The 149M comparison holds either way (100 decontaminated as well, and 3.4–6.8%). The claim is also scoped to those four recipes: across all 111 runs of the 37 triplets HarmBench-copyright reaches 100 and OR-Bench-hard falls to 0.9%, but those runs differ in their label supervision, which is the effect ∆b measures, not in their seed. Caveats on the 37. Ten are not strictly seed-only: five have an init_from that differs between seeds, and five are head-only finetunes stacked on those five. Restricting to the 27 strict triplets changes little: HarmBench-copyright min/median/max 0.0/6.0/71.0, OR-Bench-hard 0.2/3.0/58.1, and the non-v10x strict medians are 4.0 and 1.5. Separately, the four recipes we use are not four independent draws on HarmBench-copyright. All four are monotone increasing in the seed index (2/13/57, 2/5/73, 3/23/57, 3/26/65), which has probability (1/6)4 = 7.7 × 10−4 if seeds are exchangeable, so on that benchmark the seed index behaves like a systematic factor rather than like noise, and the median of the four spreads is closer to one draw than to twelve. The pattern is specific to it: OR-Bench-hard, OR-Bench-toxic and SaladBench-attack are 0/4 monotone, and only WildGuard-benign (3/4, all decreasing) comes close. Why the floor is off-path. None of the twenty arms has a seed sibling: each was trained once at seed 42, so Φb for the arms’ own recipes is not measurable from these artefacts at all, which is why a proxy is used. The nearest on-path triplets are the v2 warm-start checkpoint the seven fully retrained arms initialise from (HarmBench-copyright spread 3.0 points, OR-Bench-hard 1.5) and the 149M distillation teacher of E2–E6 and E9 (2.0 and 6.9), both far below the floor we quote; E1, E7 and E10 have frozen backbones and start from later checkpoints, of which only the 395M E1/E7 base has a triplet at all, and it is one of the ten flagged above (7.0 and 3.8). A lineage name alone does not identify a floor: the variant of the warm-start recipe the arms actually initialise from has a HarmBench-copyright spread of 3.0 points, while the same recipe on the earlier data version has 12.0. Closing this properly needs 40 further training runs, two extra seeds per arm. Prompt sets. Φb is measured on the full benchmarks and ∆b on the decontaminated subsets, so the ratio is indicative rather than exact. Recomputing ∆b on the full benchmarks moves it by under 5% on twelve of the fourteen (median 1.5%), by 11.3% on WildGuard-benign (4.3 → 4.8) and by 36.6% on OR-Bench-hard (57.4 → 42.0), which takes that benchmark’s ratio from 1.49 to 1.09. Both entries below 1 survive the swap: SaladBench-attack 0.68 either way, WildGuard-benign 0.85 → 0.96.

D

The over-refusal label audit

Instrument. Every one of the 20,456 prompts harvested for the BOUNDARY _ SAFE class was labelled independently by four instruction-tuned models of 26–35B parameters (two Gemma-family, two Qwen-family) at temperature 0, into {BOUNDARY _ SAFE, GENERIC _ SAFE, HARMFUL, AM BIGUOUS }, from a two-factor prompt (does it look harmful, and would a complete answer do harm). The HARMFUL bar is written strictly, so the counts below are conservative. The jury ran once, with no human adjudication and no inter-rater statistic beyond the vote counts, so it is our instrument and not ground truth. Pool-level split. At a uniform 3-of-4 bar: 50.9% BOUNDARY _ SAFE, 32.9% GENERIC _ SAFE, 4.5% HARMFUL, 11.6% with no three-vote majority. By plurality, which is a partition: 60.3 / 33.5 / 6.3%, with AMBIGUOUS never winning a single row of the 20,456. At the looser two-of-four bar, 19

6.9% of the pool is HARMFUL; by source that is 17.9% of the OR-Bench-hard prompts and 26.4% of the FalseReject prompts. The clean 236. Vote patterns, covering all 236: 92×{bs 2, harmful 2}, 72×{bs 1, harmful 3}, 39×{harmful 4}, 25×{bs 2, gs 1, harmful 1}, 7×{bs 1, gs 1, harmful 2}, 1×{gs 1, harmful 3}. So 211 (89.4%) are flagged harmful by two or more jurors and 112 (47.5%) by three or more (39 unanimously), none reaches three BOUNDARY _ SAFE votes, and 92 are exact 2–2 ties, better read as contested than as resolved. Against the full benchmark, 966 of the 1,319 are jury-confirmed and all 966 are contaminated; the 353 the jury rejected split 236 clean and 117 contaminated (the repository also contains passes that promote flagged and unclear prompts back into the class). Mean flag rate over the twenty arms is 0.160 on the 1,083 contaminated prompts, 0.302 on the clean 236, 0.344 on the 112 and 0.110 on the 25; the label-supervision range is 0.392, 0.574, 0.643 and 0.260 on the same four sets. Composition explains the first series but not the second. Joining the audit. Only 1,221 audit rows carry source=orbench_hard: the pool builder adds OR-Bench-80k before OR-Bench-hard and deduplicates on a global hash of the text, so 98 ORBench-hard prompts are stored under the other tag. Joining on normalised text instead recovers all 1,319, so the audit covers the benchmark completely and none of the figures above is a coverage artefact. Stratified over-refusal (§6). OR-Bench-hard over-refusal on the clean 236, then on the 211 the jury flags harmful at two or more votes, the 112 at three or more, and the 25 it leaves unclear: ours 149M (E8) 6.4 / 6.6 / 9.8 / 4.0; ours 395M (E10) 10.2 / 10.9 / 11.6 / 4.0; Qwen3Guard-0.6B 56.4 / 61.1 / 63.4 / 16.0; Qwen3Guard-4B 57.6 / 64.0 / 63.4 / 4.0; Qwen3Guard-8B 55.9 / 61.1 / 62.5 / 12.0; Granite Guardian 97.0 / 98.6 / 98.2 / 84.0. On the last stratum, the only one that resembles a real over-refusal test, the Qwen3Guard gap does not survive the label audit; the Granite gap does. n=25, so read it as a direction and not an estimate.

20

Related documents

• Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On#1390006
• History Anchors: How Prior Behavior Steers LLM Decisions Toward Unsafe Actions#180684
• 50 U.S.C. § 2732 — Safety oversight and enforcement at defense nuclear facilities#599428
• 23 U.S.C. § 405 — National priority safety programs#581035
• Whose Refusal Is It? The Unmeasured Contribution of Black-Box Multimodal Guardrails#441711