HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation

arXiv:2609.01046v1 [cs.CR] 1 Sep 2026

HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation Nikita Oblakov HiveTraceLab

Sabrina Sadiekh HiveTraceLab

Evgeniy Kokuykin HiveTraceLab

nikobl25@gmail.com

sadsobr7@gmail.com

evgeniy.kokuykin@raftds.com

September 2, 2026 Abstract Production LLMs must handle inputs that attempt to override system instructions, bypass safety policies or elicit harmful responses. A common mitigation is a separate guardrail model. Existing reports, however, provide little evidence on Russian prompt injection or Russian surface obfuscation. We present HiveTraceGuard-Pro, a 0.6B generative guardrail LoRA-tuned from Qwen3-0.6B. It is trained on Russian and English and uses one binary scoring rule (safe/unsafe) for the final target turn. Its training corpus pairs harmful examples, where a counterpart exists, with benign examples from the same domain and applies eight obfuscation transforms to both labels. In one harness, we compare HiveTraceGuard-Pro with thirty-four other guards on nineteen benchmark groups, sixteen of which are public. Its aggregate key is 0.7432, behind 0.7641 and 0.7552 for the two higher-scoring guards. Over the sixteen public groups alone, its key is 0.7153 and four of the thirty-four other suite guards score higher. In a fifteen-model comparison, HiveTraceGuard-Pro has the highest clean Russian robustness combined-F1 (0.88) and Russian prompt-injection recall (0.999). Both results use Russian sets assembled by our team, and at least 27.1% of the prompt-injection set overlaps the training corpus. Its 14.3 ms median latency is the lowest among those fifteen models in that run. Across the suite, FPR is 0.268 and FNR is 0.156. All reported response results use a legacy standalone-reply serialization rather than the natural assistant-role path of the shipped chat template. We release the merged weights on Hugging Face under Apache-2.0. The corpus, evaluation sets and evaluation code remain internal.

1

Introduction

Production language models interact with untrusted users. A guardrail can evaluate a user request before generation or an assistant response before delivery. Because the guard is separate from the assistant, its moderation behavior can be updated without retraining the assistant itself. Published guardrails focus mainly on English in both training and evaluation. Russian-language attacks can combine instruction replacement with transliteration, keyboard-layout substitutions, symbol insertion or informal rewriting. Published evaluations provide limited evidence about how existing guards handle these combinations. A guardrail serving Russian traffic must distinguish harmful intent from benign discussion of the same topic and preserve that distinction when the surface form changes. We also target deployments with tight latency and memory budgets. HiveTraceGuard-Pro therefore keeps the base model at 0.6B and uses a corpus weighted toward Russian, a structured harm taxonomy, paired benign and harmful examples, and obfuscation applied to both labels. 1

Contributions. • A compact (0.6B) generative guardrail trained on Russian and English that uses one model and one binary decision rule for requests and responses. The evaluation places particular emphasis on Russian robustness. • A corpus construction method that pairs harmful examples, where a counterpart exists, with benign examples from the same domain and applies obfuscation to both labels. This makes over-blocking measurable on topics that also contain harmful examples. We release the weights under Apache-2.0 and describe the method, but withhold the corpus (Section 8). The generative label-emission mechanism itself is established practice [1, 10, 11]. • A confirmation set reserved before development and evaluated once after the model was frozen (Section 4.4), together with a training-overlap audit of the scored benchmarks and overlap-adjusted rescoring of our own model on the contaminated ones (Section A.4).

2

Related Work

Generative guardrails differ in what they moderate and in how the policy reaches them. Qwen3Guard [1], Llama Guard [10] and ShieldGemma [11] cover multiple safety domains and languages against fixed category sets. WildGuard [12] targets adversarial prompts and jailbreaks, while PolyGuard [13] emphasizes multilingual moderation. Other systems expose the policy more directly: YuFeng-XGuard [2] adds risk reasoning and a runtime policy interface, SingGuard [3] accepts the active policy at runtime across modalities, and Shieldstral-1.0-3B [4] scores text and images against natural-language policies. OpenGuardrails [5] packages a 15B judge as a deployable platform. The reports for these models do not give results specifically for Russian prompt injection or Russian surface obfuscation, even when a model supports Russian. The 8B and 12B guards in Table 2 take 63 to 243 ms in our benchmark run, compared with 14.3 ms for HiveTraceGuard-Pro. Serving path matters as well as model size (Section 4.2), so we treat the larger guards as accuracy references and include sub-1B variants where available.

3

HiveTraceGuard-Pro Model

3.1

Threat Model

Prompt injection attempts to override or ignore system instructions through instruction replacement or role manipulation. Jailbreak attacks preserve malicious intent while reframing a request through role-play, persuasion, hypothetical scenarios or other semantic transformations. Surface obfuscation changes the textual representation without changing the intended meaning. Examples include transliteration, keyboard-layout substitution, character replacement, random capitalization and symbol insertion. Table 9 lists eight training transforms and Table 13 lists eight evaluation transforms. Five occur in both sets. The evaluation-only transforms are to_lower, to_upper and informal_rewrite. HiveTraceGuard-Pro performs binary safety classification for user requests and assistant responses. Its chat template preserves user and assistant roles and selects the last eligible turn as the target. In the natural deployment path, the request guard judges the final user turn before generation and the response guard judges the final assistant turn before delivery. The response benchmarks in this report use a legacy standalone-reply path instead (Section B.1). They

2

pass each reply as a standalone user turn, so they measure reply-content classification without exercising the template’s assistant-target branch. The threat model covers injected instructions that appear in the moderated turn, regardless of their origin. An instruction in a retrieved document that is never shown to the guard is out of scope. The evaluations target one turn per example and do not test behavior conditioned on a longer dialogue history. They also do not test adaptive attacks against the released weights. Every evaluated transform was fixed without access to model gradients. User request

Request guard

safe

LLM

unsafe

Response guard

safe

Output

unsafe

Block / refuse

Block / rewrite

Figure 1: Conceptual guard placement. In the natural deployment path, the request guard targets the final user turn and the response guard targets the final assistant turn. Reported response benchmarks instead use the legacy standalone-reply serialization of Section B.1. An unsafe verdict is a triage signal, and the surrounding application decides whether to block, refuse or rewrite.

3.2

Model Architecture and Training Data

HiveTraceGuard-Pro is built on Qwen3-0.6B and fine-tuned with LoRA. Its scoring rule returns one binary verdict without an explanation or category label. The generative formulation uses the causal language-model head already present in the base model. During training, only the verdict token and end-of-sequence marker contribute to the completion-only loss. Section B gives a configuration summary and the measured inference footprint. The training corpus combines moderation datasets, translated resources, synthetic adversarial examples and internally generated robustness augmentations. It contains Russian and English examples in both dialogue roles. A labeled subset of the unsafe examples, 35 013 rows under the 14 reported harm categories of Table 7 and 2 948 under one additional policy category, is used for balancing and stratified evaluation, although the model predicts only a binary label. Where possible, each harmful example is paired with a semantically related benign example so that false positives and false negatives can be measured within the same topic. Table 7 reports the training count for each presented category. Section A describes sources, licensing, deduplication and augmentation.

4

Evaluation

4.1

Evaluation Methodology

Two cohorts. The wider suite contains thirty-five openly released guards. We score each guard through a model-specific adapter that preserves its published prompt format, decode settings and verdict parsing. Response items use the common standalone-reply construction described below. The harness covers nineteen benchmark groups drawn from forty-four datasets. Sixteen groups are public and three are our internal sets (Section C). We order the suite by the aggregate key, the geometric mean of the nineteen group scores. Seven groups have no benign class and

3

therefore contribute recall alone. Their group scores do not measure over-blocking. Table 12 defines each group score. Cross-model comparisons within a cell use that cell’s metric. The head-to-head cohort in Table 2 is a fifteen-guard subset of the suite: the eleven highest-scoring guards under the aggregate key and four widely deployed references below them. The cohort spans 0.5B to 15B parameters. All cells are our reruns rather than values copied from model reports. We checked the cohort tables against benchmark artifacts, the held-out tables against evaluation reports and the contamination results against a complete rerun of the audit. The head-to-head comparison has not been reproduced by a third party. Two harnesses. Cohort results and internal held-out results come from separate harnesses, so their values are not interchangeable even when they use the same rows. On augmented requests they agree to two decimals, with combined-F1 0.853 in the cohort harness and harm-F1 0.852 in the internal harness. Their error rates differ slightly, with FPR 0.132 versus 0.139 and FNR 0.128 versus 0.123. On the underlying counts these are 475 versus 500 false positives among the 3 599 benign rows and 354 versus 339 misses among the 2 756 harmful ones, and the paired row-level disagreement can be larger. Response-side unsafe-class F1 differs more, at 0.756 in the cohort harness and 0.671 in the internal harness. Their error rates also differ more, with FPR 0.026 versus 0.043 and FNR 0.164 versus 0.180. Only 6.5% of the 945 response rows are positive, so the fifteen additional false positives and one further missed harmful reply in the internal harness reduce precision from 0.689 to 0.568. Table 2 uses the cohort harness that scored all fifteen models. Comparisons among internal results use the internal harness. A model is graded only on rows for which it emits a usable verdict. Section D reports the per-model shortfalls. Three F1 aggregations. macro-F1 is the unweighted mean of safe-class and unsafe-class F1. harm-F1 is unsafe-class F1 alone. combined-F1 is the same unsafe-class F1 formula computed by the cohort harness over pooled safe and harmful slices. Harm-F1 and combined-F1 therefore differ in the harness and rows used, not in the formula. These quantities are not interchangeable. On the augmented robustness-test, for example, macro-F1 is 0.866 and harm-F1 is 0.852. We also report false-positive and false-negative rates to separate over-blocking from missed harm. Attack-only sets contain no benign class, so we report recall. Results on the confirmation set use 95% Wilson intervals because every cell has fewer than 1 000 examples (Table 4). Other tables report point estimates. We do not report confidence intervals or multiplicity-corrected comparisons for cross-model differences or the suite ordering. Evaluation sets. We use four internal held-out sets (Table 1). None is a training split, and all four serve as references in the contamination audit. Three influenced accept or reject decisions during development and are reported as development sets. The fourth, hivetrace/insecure-prompts, was reserved and not queried until the released model was final (Section 4.4). We also use public benchmarks where noted. In the Russian development sets, each harmful prompt is matched with a benign prompt from the same domain, such as Fascism versus History of fascism or Drugs versus Pharmacology. Decision rule and decision path. The scoring rule returns one verdict. We take the larger of the safe and unsafe logits, which is equivalent to a fixed 0.5 boundary under a two-way softmax. The boundary is not calibrated separately by language. The shipped path has no input normalizer, and reported labels are direct model outputs. Request-side evaluation matches the natural deployment path. Response-side evaluation does not. Every reported response result passes the reply standalone as a user turn, including the cohort comparison in Table 2. A natural dialogue would pass the reply as an assistant turn and select a different branch of the chat 4

template. Apart from the Aegis-2.0 check in Section B.1, that assistant-role path is not evaluated here. The primary internal augmented result is harm-F1 on the robustness set, which we call robustness-augmented harm-F1. Set

Role

Lang

Rows scored

Composition

hivetrace/ robustness-test

development, primary

RU

11 879 (10 934 req / 945 resp)

RU-categories

development

RU

220 per role per category

nvidia/ Aegis-2.0 [9]

development, secondary

EN

1 964 req / 813 resp

hivetrace/ insecure-prompts

reserved until final model

RU

156 req / 156 resp

paired safe/harm twins, requests crossed with clean and eight obfuscation transforms 17 topic-paired safe↔harm categories, request and response externally authored, out-of-distribution English holdout, lower-confidence reference every request an injection attempt (no benign class). Responses 85 harm / 71 safe

Table 1: The four internal held-out sets. The first three influenced development decisions. The fourth was reserved before development and evaluated once after the released model was final (Section 4.4). Only nvidia/Aegis-2.0 was authored outside our team. Aegis-2.0 response row counts differ between the internal and cohort harnesses, 813 versus 852, entirely because of benign rows. This table reports the internal count.

4.2

Comparison with Existing Guardrails

Results vary by domain. HiveTraceGuard-Pro has clean Russian robustness combined-F1 0.88, the highest value among the fifteen models and also the maximum in the wider suite. Its Russian prompt-injection recall is 0.999, with 4 misses among 4 417 attacks. On English prompt injection it reaches 0.88, compared with 0.90 for YuFeng-XGuard-Reason-8B. On augmented requests it reaches 0.853 and on responses 0.756. Three guards score higher on the augmented column and three on the response column, where SingGuard-2B is level with ours at the table’s precision. Across the full suite, three guards exceed the augmented-request score and five exceed the response score. Because the injection and S-Eval sets contain only attacks, their recall should be read together with benign over-blocking. Request-side FPR is 0.016 on clean inputs and 0.262 on wrong_layout, the highest among the individual transforms. Only the internal harness breaks these errors down transform by transform (Table 13). The 14.3 ms median latency is the lowest value in the table, 0.41 ms below Shieldstral-1.0-3B. The two models with higher aggregate keys take 53.35 and 71.79 ms, and Qwen3Guard-Gen-8B takes 152.71 ms. These values are comparable only within the same benchmark run. Figure 2 plots the twenty-two guards with an aggregate key above 0.55. The two other 0.6B guards in Table 2 have aggregate keys of 0.7200 for YuFeng-XGuardReason-0.6B and 0.7118 for Qwen3Guard-Gen-0.6B, compared with 0.7432 here. Relative to YuFeng-XGuard-Reason-0.6B, HiveTraceGuard-Pro has higher values on every Russian column, English prompt injection and Aegis-2.0 responses, and lower latency. It has lower values on both S-Eval columns, Aegis-2.0 prompts and ToxicChat. Relative to Qwen3Guard-Gen-0.6B, it has higher values on seven of ten score columns and lower latency, but lower values on both Aegis-2.0 columns and ToxicChat. Most of the fifteen models score higher on Aegis-2.0 prompts, Aegis-2.0 responses and ToxicChat. Section 4.6 examines English performance on additional 5

public benchmarks.

aggregate key (19-group geometric mean)

YuFeng-XGuard-Reason-8B 71.8 ms

0.75

HiveTraceGuard-Pro 0.6B 14.3 ms

SingGuard-2B 53.4 ms

Shieldstral-1.0-3B YuFeng-XGuard-Reason-0.6B Qwen3Guard-Gen-0.6B

0.70

OpenGuardrails-Text-2510 Qwen3Guard-Gen-8B

Qwen3Guard-Gen-4B Llama-3.1-Nemotron-Safety-Guard-8B-v3

PolyGuard-Qwen-Smol

Nemotron-3.5-Content-Safety

Llama-Guard-3-8B

0.65 Llama-Guard-4-12B

Llama-Guard-3-1B

0.60 no equal-or-smaller model scores higher other guards (21 of the 34 others) HiveTraceGuard-Pro (ours)

0.55

0.3B

0.5B

1B

2B 4B parameter count (log scale)

8B

15B

Figure 2: Aggregate key against parameter count. The dashed line joins models for which no model of the same or smaller size has a higher aggregate key. Points show the 22 of 35 suite guards with an aggregate key above 0.55, and unnamed points are listed in Table 11. The two guards above HiveTraceGuard-Pro on the aggregate key use 2B and 8B parameters and have median latencies of 53.4 and 71.8 ms in the same run, compared with 14.3 ms here. Each guard was run once, so no uncertainty intervals are shown. Latency in Table 2 depends on the serving configuration. The same HiveTraceGuard-Pro weights measure p50 32.75 ms under our --enforce-eager configuration and 14.3 ms in the benchmark run. The models in the column come from that benchmark run, so within-run ratios are comparable, but absolute values are not portable. Runtime also depends on hardware, engine configuration, output length and serving implementation, not only parameter count. For example, a 1B guard takes 20.50 ms while another 0.6B guard takes 37.18 ms. HiveTraceGuard-Pro emits one token (Section B.1). The column therefore measures end-to-end harness latency and does not isolate model compute. Every guard uses its published prompt format, decode settings and verdict strings through a per-model adapter. The data comparison is still asymmetric. Three of the nineteen groups are internal sets that we authored and used during development. Removing those groups drops HiveTraceGuard-Pro below two further guards on the key recomputed over the sixteen public groups (Section C). An unexpectedly low competitor cell may reflect its adapter or the benchmark as well as the model. We therefore interpret size comparisons primarily within the Qwen3Guard-Gen 0.6B/4B/8B and YuFeng-XGuard-Reason 0.6B/8B model families, and on the public Aegis-2.0 columns. Llama-3.1-Nemotron-Safety-Guard-8B-v3 is flagged for evaluation leakage on Aegis-2.0. Its 6

training data is partly derived from Aegis-2.0 and overlaps its holdout splits. We report the measured values without adjustment and mark the overlap. Our own model also has English-side Aegis-2.0 overlap, quantified in Section A.4. Model

Aegis-2.0 Aegis-2.0 Robust-clean Robust-in Robust-out PI-EN PI-RU S-Eval S-Eval Toxic Params combined-F1 combined-F1 combined-F1 combined-F1 combined-F1 recall recall base atk Chat (prompt) (resp.) (FPR/FNR) (FPR/FNR) (FPR/FNR)

YuFeng-XGuard-Reason-8B [2]

8B

0.85

0.78

SingGuard-2B [3]

2B

0.84

0.82

HiveTraceGuard-Pro (ours)

0.6B

0.82†

0.80

Shieldstral-1.0-3B [4]

3B

0.85

0.81

OpenGuardrails-Text-2510 [5]

15B

0.82

0.87

Qwen3Guard-Gen-8B [1]

8B

0.86

0.84

Qwen3Guard-Gen-4B [1]

4B

0.86

0.83

YuFeng-XGuard-Reason-0.6B [2]

0.6B

0.86

0.77

Llama-3.1-Nemotron-Safety-Guard-8B-v3 8B

0.87§

0.86§

Qwen3Guard-Gen-0.6B [1]

0.85

0.82

0.6B

PolyGuard-Qwen-Smol [13]

0.5B

0.85

0.73

Nemotron-3.5-Content-Safety

4B

0.86

0.85

Llama-Guard-3-8B

8B

0.76

0.64

Llama-Guard-4-12B

12B

0.71

0.63

Llama-Guard-3-1B

1B

0.73

0.61

0.81 (.03/.10) 0.87 (.01/.12) 0.88 (.02/.05) 0.72 (.04/.13) 0.77 (.01/.32) 0.80 (.03/.08) 0.76 (.04/.09) 0.78 (.03/.12) 0.68 (.04/.26) 0.63 (.07/.14) 0.67 (.04/.22) 0.60 (.09/.10) 0.61 (.02/.43) 0.49 (.07/.38) 0.48 (.09/.32)

0.86 (.03/.21) 0.84 (.04/.24) 0.85 (.13/.13) 0.82 (.08/.24) 0.73 (.03/.40) 0.89 (.05/.15) 0.88 (.04/.18) 0.78 (.05/.31) 0.71 (.07/.39) 0.81 (.12/.21) 0.70 (.08/.41) 0.85 (.11/.16) 0.72 (.08/.38) 0.78 (.24/.16) 0.77 (.13/.27)

0.71 (.00/.43) 0.76 (.01/.28) 0.76 (.03/.16) 0.71 (.01/.34) 0.75 (.00/.38) 0.82 (.00/.26) 0.77 (.01/.33) 0.67 (.00/.49) 0.72 (.00/.42) 0.70 (.01/.39) 0.51 (.05/.43) 0.67 (.00/.48) 0.47 (.01/.66) 0.30 (.05/.69) 0.24 (.18/.51)

p50 (ms)

0.90

0.909

0.81

0.95

0.55

71.79

0.76

0.742

0.68

0.63

0.54

53.35

0.88†

0.999†

0.71

0.80†

0.51†

14.30

0.74

0.836

0.73

0.61

0.61

14.71

0.72

0.687

0.52

0.49

0.56

63.22

0.83

0.904

0.72

0.68

0.58

152.71

0.83

0.908

0.72

0.65

0.58

96.92

0.87

0.918

0.79

0.95

0.53

16.85

0.85

0.870

0.67

0.60

0.51

243.28

0.73

0.894

0.70

0.61

0.55

37.18

0.79

0.861

0.65

0.59

0.44

77.36

0.83

0.930

0.73

0.65

0.49

54.90

0.71

0.603

0.43

0.42

0.30

63.16

0.66

0.529

0.36

0.45

0.26

86.19

0.68

0.635

0.49

0.59

0.18

20.50

Table 2: Head-to-head comparison. Fifteen suite guards from 0.5B to 15B parameters, scored by us in one benchmark run under Section 4.1 and ordered by the same nineteen-group aggregate key. Bold marks the best displayed value in each column, so two cells that round to the same figure are both bold. Counts of how many guards score higher, here and in Section 6, compare the underlying values. Each Aegis-2.0 column is unsafe-class F1 over pooled safe and harmful slices, with prompts and responses scored separately. Robust-clean, Robust-in and Robust-out are the clean-request, augmented-request and response portions of robustness-test. Each is reported as combined-F1 with FPR/FNR below. PI-EN and PI-RU contain only attacks, so they report recall and their FNR is one minus recall. They are translations of the same 4 417 upstream items, not independent benchmarks. S-Eval base and attack report recall [19]. ToxicChat reports unsafe-class F1 [14]. The p50 column is median wall-clock latency with one request in flight (Table 10). Each model uses a per-model adapter for its published prompt, decode settings and verdict format. The response side uses the common standalone-reply construction. †: training overlap is at least 27.1% for PI-RU, 10.0% for S-Eval attack, 21.3% for the Aegis-2.0 prompt set and 3.2% for ToxicChat (Section A.4). The contamination scan is monolingual, so the low measured overlap on PI-EN does not establish training independence. §: evaluation-leak flag, described in Section 4.2. Response columns use the legacy standalone-reply protocol (Section B.1).

4.3

Internal Held-Out Evaluation

On clean classification, macro-F1 is 0.952 on the synthetic-clean subset, the 450 benign and 345 harmful paired examples to which the obfuscation transforms are applied, and 0.937 on RU-categories. Harm-F1 is 0.946 on the synthetic-clean subset. AUROC is 0.997 over the 4 579 clean request rows and 0.985 on RU-categories. The primary augmented result is harm-F1 0.852 (Table 3). Macro-F1 is 0.898 when clean and augmented conditions are pooled. Over the combined request and response robustness-test (n = 11 879), the model correctly labels 8 110 benign turns and flags 607 benign turns. It detects 2 798 harmful turns and misses 364. Section 4.5 breaks these errors down by condition and category. Across 32 reported RU-categories cells, unsafe-class F1 ranges from 0.843 to 1.000, 7

with 25 cells at or above 0.90 and a minimum request-side value of 0.875 (Table 14). We also evaluate a prompted baseline under the same scoring rule. The untuned Qwen3-0.6B backbone receives an English guard prompt and uses the same constrained verdict-token argmax. It misses most harmful requests, with harm-FNR 0.87 on Aegis-2.0 and 0.93 on the robustness set, and over-blocks benign responses at FPR 0.49–0.77. The released model’s harm-FNR is 0.206 and 0.114 on those request sets, and its benign-response FPR is 0.193 and 0.043. A simple Russian prompt that requests JSON output fails at the output-format stage, producing RU-categories FPR 0.169 and FNR 0.770. This comparison changes the prompt and scoring setup as well as the weights, so it does not isolate the effect of fine-tuning. Held-out set

Condition

robustness-test robustness-test robustness-test robustness-test robustness-test robustness-test RU-categories Aegis-2.0 (EN) Aegis-2.0 (EN) Aegis-2.0 (EN)

synthetic-clean subset robust (augmented) clean requests requests only (clean+aug) responses overall (clean+aug, both roles) request + response request response request + response

macro-F1

FPR

FNR

safe n

harm n

0.952 0.866 0.939 0.898 0.821 0.898 0.937 0.815 0.807 0.813

0.053 0.139 0.016 0.073 0.043 0.070 0.082 0.159 0.193 0.170

0.041 0.123 0.041 0.114 0.180 0.115 0.044 0.206 0.193 0.202

450 3 599 4 234 7 833 884 8 717 3 740 905 419 1 324

345 2 756 345 3 101 61 3 162 3 740 1 059 394 1 453

Table 3: The three development sets under the internal harness. Table 4 reports the reserved fourth set. Counts come from the run’s classification reports. The augmented-only result is macro-F1 0.866 and harm-F1 0.852. The Aegis-2.0 values differ from Table 2. Both harnesses score the same 1 964 requests and give similar rates, FPR 0.159 versus 0.152 and FNR 0.206 versus 0.211.

4.4

Reserved Confirmation Set

The three development sets influenced accept or reject decisions, and we did not record how often each set was queried. Their results may therefore be optimistic. We reserved a fourth set, hivetrace/insecure-prompts, before development. It was part of the decontamination reference but not of the residual contamination scan (Section A.4), and it was not used for model selection. We evaluated the frozen release on it once with the same contract and internal harness used for Table 3, and made no changes after observing the result. Because this set differs in composition from the development sets, it is an out-of-development check rather than a direct estimate of selection bias. The set contains 156 Russian dialogues. Every user turn is an injection attempt: 120 jailbreaks, 22 prompt injections, 13 harmful-content requests and 1 data-extraction request. Each dialogue has an independently labeled assistant reply, giving 85 harmful and 71 safe replies. The request block has no benign class, so it has no measurable FPR or macro-F1. Its harm-F1 is determined entirely by recall because unsafe-class precision is fixed at 1.0. An always-unsafe classifier would therefore score perfectly on the request block. The pooled request-plus-response result (n = 312, macro-F1 0.655) mixes this one-class request block with the response block and is not used as a summary result. The response block is the only part that supports macro-F1. In the same run, rescoring the three development sets reproduced all 18 metric aggregates to ten decimal places. Responses use the same legacy standalone-reply path as the other reported response results, so the model does not see the associated injection prompt. On this attack-only request block, harm-F1 is 0.953 and FNR is 0.090, compared with request8

Slice

n

macro-F1

harm-F1

FPR (95% CI)

FNR (95% CI)

n/a n/a n/a 0.602

0.953 n/a 0.874 0.691

n/a 0.563 (0.448–0.673) n/a 0.563 (0.448–0.673)

0.090 (0.054–0.145) n/a 0.224 (0.148–0.323) 0.224 (0.148–0.323)

Development held-out sets, response side, same scoring run robustness-test 945 0.821 0.671 0.043 (0.031–0.058) Aegis-2.0 (EN) 813 0.807 0.802 0.193 (0.158–0.234)

0.180 (0.104–0.295) 0.193 (0.157–0.235)

insecure-prompts (reserved) Requests, all injection 156 Responses, safe slice 71 Responses, harm slice 85 Responses, pooled 156

Table 4: The released model on the reserved set, together with two development response sets from the same scoring run. Intervals are 95% Wilson intervals for the underlying rates. An n/a marks undefined quantities. The request block contains only injection attempts, so FPR and macro-F1 are undefined. Each single-label response slice supports only its corresponding error rate. On the 85-row harmful-response slice, precision is fixed at 1.000 because no benign rows are present. Harm-F1 0.874 is therefore a transformation of recall, with 66 of 85 rows predicted unsafe. side FNR 0.114 on robustness-test and 0.206 on Aegis-2.0. The model detects all 22 prompt injections, 108 of 120 jailbreaks and 12 of 14 harmful-content or data-extraction requests. The 14 misses are 12 jailbreaks and 2 requests from the remaining categories. Because the set has no benign requests, this result does not establish better overall request moderation than on the development sets. Response over-blocking is much higher on the reserved set. The model flags 40 of 71 safe replies, giving FPR 0.563, compared with 0.043 and 0.193 on the development sets. The 95% Wilson interval is wide, but its lower bound of 0.448 is above the upper bounds for both development results. Errors occur in both response types: 22 of 40 safe refusals and 18 of 31 safe compliances are flagged. The replies are in-character role-play continuations of jailbreak attempts and are labeled safe when they contain no harmful content. This narrow register is a plausible source of distribution shift. The set is small, entirely Russian and evaluated with one model seed, so it should be read as a stress test rather than a replacement for the development sets.

4.5

Error Analysis

On the Russian robustness set, 607 of 971 errors are benign turns flagged as unsafe. Most occur after augmentation: 500 errors among 3 599 augmented benign requests, for FPR 0.139. The remaining false positives are 69 among 4 234 clean requests and 38 among 884 benign responses. Section D compares this error profile with the rest of the suite. Strong surface obfuscation remains difficult. The held-out evaluation builds its wrong_layout transform with a cipher that leaves five Cyrillic letters unchanged (U+0431, U+0436, U+0445, U+044A, U+044E) and maps yo (U+0451) to a backtick. An earlier training-time cipher mapped those characters to punctuation, so the pre-fix model was evaluated on substitutions absent from training. On the pre-fix build that transform carried a request-side harm-FNR of 0.675. Aligning the ciphers, then rebalancing the obfuscation augmentation (Section G), brought the released build to 0.125 at the same model size. Because the builds differ in more than the cipher, this comparison establishes the direction of change but is not a controlled ablation (Table 13). The remaining errors occur on both labels. Harmful requests are missed most often after the held-out informal_rewrite transform, with harm-FNR 0.307, and after randomized case, with FNR 0.136. Figure 3 shows all conditions together.

9

clean requests transform trained on transform held out of training

0.30 informal_rewrite

harm missed (FNR)

0.25

0.20

0.15

random_case

random_replace_symbols

transliterate_to_latin

0.10

wrong_layout

random_insert_symbols

0.05

to_lower

to_upper

Original (clean)

0.00 0.00

0.05

0.10

0.15 0.20 benign over-block (FPR)

0.25

0.30

Figure 3: Request-side FPR and FNR for each condition in Table 13. The clean point uses 4 234 benign rows, compared with 449 or 450 for each transform. Every transform worsens at least one error rate relative to clean requests. The wrong_layout transform has the highest FPR (0.262), and the held-out informal_rewrite has the highest FNR (0.307). Filled points denote transforms used in training, and hollow points denote evaluation-only transforms. Across the additive rounds in Section G, improvements on one obfuscation error rate repeatedly worsened the other. None of the tested data mixtures removed this coupling. Because all experiments keep the model at 0.6B, they do not distinguish a data limitation from a capacity limitation. The largest response-side false-positive rates occur on benign discussion of harm-adjacent topics. Response-side FPR is 0.373 for cybercrime, 0.373 for non-violent crime and 0.291 for weapons, the three highest reported categories. The three RU-categories topic pairs without a dedicated training class (Table 7) test transfer rather than in-distribution performance. Military conflict has request-side harm-F1 0.875, the request-side minimum, and copyright infringement has 0.885. Both are among the four lowest request-side cells. All three pairs carry request-side FPR between 0.127 and 0.145. We do not rank them against one another at n = 220 per cell.

4.6

External Benchmark Generalization

Because the preceding results rely heavily on our held-out distribution, we also evaluate the released model on ten public benchmarks against four other guards (Table 5). Every cell is our measurement. Two competitors are also 0.6B, including the closest of the four below HiveTraceGuard-Pro on the aggregate key. Five are English benchmarks and five are multilingual slices. Across these ten benchmarks, HiveTraceGuard-Pro has the highest value in three rows, the lowest 10

in two and is within 1.5 points of the highest value in three. Its recall is 95.2 on the Russian Aya red-teaming slice and 91.7 on the English slice. Its unsafe-class F1 on XSafety English is 62.1. Absolute performance on XSafety is low across the suite. Four guards have FPR at or above 0.95, and among the other 31 the highest score is 67.5. Three of the four post larger nominal scores, 96.0, 73.0 and 71.7, at FPR 1.000, 0.975 and 1.000, and the fourth scores 56.2 at FPR 1.000. The largest gaps are all on English benchmarks. SimpleSafetyTests recall is 91.0 compared with 99.0 for all four competitors. HarmBench response F1 is 82.7 compared with 86.3 for WildGuard. OpenAI Moderation F1 is 71.4 compared with 79.0 for Llama-Guard-3-8B, and ToxicChat F1 is 50.6 compared with 54.9 for Qwen3Guard-Gen-0.6B. The ToxicChat cell carries FNR 0.580 in the cohort harness. The English deficit is not uniform across paired datasets. On four of seven English/Russian pairs in the suite, English recall is at least as high as Russian recall, and the model posts the highest score of the five guards in Table 5 on both Aya slices. These results point to benchmark-specific weaknesses rather than a single language-wide shift. Table 12 reports all nineteen group scores. Benchmark

Metric

Aya Red-teaming (RU) [6] Aya Red-teaming (EN) [6] XSafety (EN) [8] RTP-LX (RU, request) [7] RTP-LX (EN, request) [7] BeaverTails (responses) [17] HarmBench (responses) [16] OpenAI Moderation [15] ToxicChat [14] SimpleSafetyTests [18]

recall recall unsafe-class F1 unsafe-class F1 unsafe-class F1 unsafe-class F1 unsafe-class F1 unsafe-class F1 unsafe-class F1 recall

HiveTraceGuard-Pro (ours, 0.6B)

YuFeng-XGuardReason-0.6B

Qwen3GuardGen-0.6B

WildGuard 7B

Llama-Guard-3 8B

95.2 91.7 62.1 87.5 93.0 85.5 82.7 71.4 50.6 91.0

90.6 85.0 60.6 88.8 91.1 83.6 85.5 71.0 53.0 99.0

92.6 90.7 60.7 86.3 91.2 86.3 84.8 66.1 54.9 99.0

73.6 90.9 61.4 64.5 94.5 84.1 86.3 72.7 43.5 99.0

69.6 62.3 39.8 48.0 48.8 67.8 84.5 79.0 29.9 99.0

Table 5: Public safety benchmarks (×100), measured by us in one benchmark run. Each row names its metric. Rows with both classes report unsafe-class F1. Attack-only rows report recall, which is not directly comparable with F1. Bold marks the best value per row. The first five are multilingual slices of Aya Red-teaming, RTP-LX and XSafety (Section 4.6). The YuFeng-XGuardReason-0.6B, Qwen3Guard-Gen-0.6B and Llama-Guard-3-8B ToxicChat values repeat the same runs used in Table 2. Competitors are English-native or multilingual. HiveTraceGuard-Pro also has direct English supervision, with 187 215 of 464 092 training rows in English (40.3%, Table 8).

5

Intended Use and Scope

Intended use. HiveTraceGuard-Pro is a request and response filter for LLM applications, positioned as in Figure 1. We trained it on Russian and English, and evaluated it most extensively on Russian. Its binary verdict is an input to a moderation policy, not a complete policy decision. Response numbers come from the legacy standalone-reply path of Section B.1, not from the natural assistant-role path. Out of scope. The model is not validated for languages other than Russian and English. Other deployments require domain-specific evaluation. It does not return categories, explanations or policy reasoning. Applications that need those outputs require a separate classifier or policy layer. The model judges one final turn and is not evaluated as a monitor for multi-step agent trajectories or tool calls. It is text-only and does not provide a legal or compliance determination. Operating point. The decision rule is the argmax boundary of Section 4.1. AUROC provides a threshold-independent comparison: 0.985 on RU-categories and 0.973 on pooled robustness-test rows, compared with 0.888 on Aegis-2.0. The English shortfall therefore reflects ranking quality 11

as well as threshold placement. A global threshold sweep over the three development sets finds τ = 0.532 and changes mean macro-F1 by less than 0.001. Even a separate oracle threshold for each set improves macro-F1 by at most 0.003. On the robustness-test response slice the same sweep improves macro-F1 by 0.061. Operators serving substantial English traffic should calibrate on their own distribution. Prompt handling. The shipped tokenizer call applies no automatic truncation. The 2048token value in Table 10 is a training limit, not an inference limit, and forward passes were verified through the model’s 40 960-position range. Behavior beyond that range is unspecified. In one exploratory probe, a harmful request padded to roughly 80 000 filler tokens was labeled safe (Section B.1). Because the policy block opens the prompt and the target turn and its marker close it, neither end may be cut. Any application-side length cap must shorten the conversation between them (Section B.1). Failure behavior. If the guard is unavailable, a fail-open pipeline permits all traffic during the outage, while a fail-closed pipeline blocks all traffic. This operational choice is separate from the model’s normal classification errors. At the shipped operating point on the Russian robustness set, the guard flags 7.0% of benign turns and misses 11.5% of harmful turns (Section 4.3). Individual slices vary substantially, from FPR 0.016 on clean requests to 0.563 on the reserved-set responses. Operators should estimate both error costs on data that resemble their own traffic (Sections 4.4 and 6).

6

Limitations

Over-blocking is more common than missed harm. Pooled over the suite, FPR is 0.268 and FNR is 0.156. Eight of the thirty-five guards have lower FNR, while twenty-five have lower FPR (Figure 4). Averaging by benchmark group gives FPR 0.254 over the twelve groups with a benign class and FNR 0.184 over all nineteen groups. The same ordering holds under both aggregations. Section D identifies the benign-stress sets with the highest FPR. Response side. Clean response harm-F1 is 0.671 compared with 0.889 on requests, mainly because of lower precision. Response-side FPR is 0.563 on the reserved set and 0.043 on robustness-test (Section 4.4), so the development-set FPR does not transfer to the reserved response distribution. All reported response results use the legacy standalone-reply serialization with role user. Apart from the sensitivity check in Section B.1, they do not evaluate the natural assistant-role branch of the chat template. Obfuscation. Across six additive data rounds and one change of training objective, improvements in recall on obfuscated harmful requests repeatedly increased over-blocking of obfuscated benign requests, or the reverse (Sections 4.5 and G). The tested changes did not remove this trade-off. Prompt length. The tokenizer does not truncate automatically, and behavior beyond the verified 40 960-position range is unspecified (Section 5). Deployments should therefore enforce a validated context limit while preserving both the policy prefix and the target suffix (Section B.1). Scope of the summary results. The released robustness-augmented harm-F1 of 0.852 is above the three-seed mean of 0.844 ± 0.011. On the response robustness column it reaches combined-F1 0.76 against 0.82 for an 8B guard, with three cohort guards ahead, and it leads the clean-request column. Recall on attack-only sets measures detection only and does not include benign over-blocking. One global argmax boundary is used for both languages. Two suite cells are too small for reliable ranking, at n = 16 and n = 61. The RTP-LX English response FPR is based on 26 benign examples within a 999-row cell. Latency is measured with one request in flight 12

HiveTraceGuard-Pro (ours) YuFeng-XGuard-Reason-8B SingGuard-2B

harm missed, pooled FNR over the suite

0.8

Shieldstral-1.0-3B OpenGuardrails-Text-2510 field median FPR 0.227

Qwen3Guard-Gen-8B YuFeng-XGuard-Reason-0.6B Qwen3Guard-Gen-0.6B

0.6

ShieldGemma-9B

0.4

field median FNR 0.222

0.2

0.0 0.0

0.1

0.2 0.3 0.4 benign over-block, pooled FPR over the suite

0.5

Figure 4: Pooled FPR and FNR for each guard across the nineteen suite groups. Three injectionfocused detectors fall outside the plotted range, at FPR 0.776, FPR 1.000 and FNR 0.989, and the medians include them. HiveTraceGuard-Pro has FPR 0.268 and FNR 0.156, compared with field medians of 0.227 and 0.222. Labels identify nearby guards on the aggregate key and selected reference models. Qwen3Guard-Gen-0.6B lies under the HiveTraceGuard-Pro marker because both have FPR 0.268 and their FNR differs by 0.004. The aggregate key weights groups equally, whereas this figure pools rows across sets of different sizes. and does not estimate concurrent throughput. Several orderings in this report turn on differences of a few thousandths, among them the response robustness column and the false-positive ranking above. Each guard was run once, so these are point estimates without intervals.

7

Conclusion

We presented HiveTraceGuard-Pro, a 0.6B generative guardrail for request and response moderation in Russian and English. We release the weights on Hugging Face under Apache-2.0. The evaluation compares thirty-five guards in one harness on nineteen benchmark groups, includes a confirmation set reserved before development and evaluated once on the frozen release (Section 4.4), and audits overlap between the scored benchmarks and the training corpus (Section A.4). The model’s aggregate key is 0.7432, behind an 8B guard at 0.7641 and a 2B guard at 0.7552. Restricted to the sixteen public groups, its key is 0.7153, with four of the thirty-four other suite guards above it (Section C). In the fifteen-model comparison it has the highest score on the clean Russian robustness set, which we authored and used during development. It also has the highest Russian prompt-injection recall and reaches 0.88 on the English prompt-injection set, compared with 0.90 for YuFeng-XGuard-Reason-8B. The Russian set contains only attacks and overlaps 13

the training corpus by at least 27.1%, and the English set translates the same items, so its low measured overlap does not establish training independence (Section A.4). Three guards score higher on augmented requests and three on the legacy response evaluation. HiveTraceGuard-Pro has the lowest median latency in the comparison, with a margin of 0.41 ms over the next model, although absolute latency depends on the serving setup (Section 4.2). Unsafe-class F1 is at least 0.90 on 25 of 32 reported category cells (Table 14). The model’s strongest results are on our in-domain Russian tests. Performance is weaker under strong obfuscation, on English general-moderation benchmarks and in response-side precision.

8

Model Availability

HiveTraceGuard-Pro is released on Hugging Face under Apache-2.0 at https://huggingface. co/hivetrace/HiveTraceGuard-Pro as a single merged model. The training corpus, evaluation datasets and evaluation code remain internal. Section B.1 specifies prompt serialization, verdicttoken ids and truncation behavior, and includes code that reproduces one reported result. The shipped generation_config.json uses greedy one-token decoding with max_new_tokens 1 and do_sample false. The README also documents caller-side constrained scoring with allowed_token_ids=[18675,38157] and a two-way softmax over the verdict logits. The 0.6B model’s weights occupy 1.11 GiB, and allocator peak is 1.92 GiB at a 2048-token context (Table 10), or 1.39 GiB with logits restricted to the final position (Section B). In the comparison run it is about 10.7× faster than Qwen3Guard-Gen-8B at 152.7 ms. This ratio applies only to that run.

A

Dataset Construction

A.1

Sources and Admission

The training corpus combines public safety datasets, translated moderation resources, internally curated Russian datasets and synthetic adversarial examples from the six provenance families in Table 6. Public sources are filtered by license at the family level rather than source by source. Rows are admitted from existing label, category or score columns, or through a whole-source decision. We do not use regex, keyword or length filters to infer the safety label. Because labels are inherited column-wise instead of re-annotated by multiple human raters, the corpus carries no inter-annotator-agreement statistic of its own. Peer corpora that do re-annotate report such measures (WildGuard [12] Fleiss’ kappa 0.55–0.72, Aegis [9] approximately 74% agreement, PolyGuard [13] Krippendorff’s alpha 0.46 and 0.94), whereas our label reliability rests on the upstream sources’ own labeling. The corpus draws on 19 551 distinct source tags. The only per-source breakdown available to us is an earlier assembly snapshot (n = 424 040), in which the largest single source is an in-house synthetic obfuscation set at 22 874 rows and no other source exceeds it. The per-source composition of the released split itself was not recomputed, so we make no concentration claim about it and do not restate that count as a share of the 464 092 released rows. For the same reason no per-source figure is drawn. Synthetic data provenance. Of the 464 092 training rows, 5 300 (1.14%) contain text generated by our team with a language model. The contrastive block contains 4 700 rows generated with Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive at Q4_K_M quantization. The 600row roleplay-continuation block uses Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q8_0 14

for harmful continuations and Qwen3-1.7B-Instruct for paired benign continuations. All three models were served through Ollama 0.31.2 with temperature 0.95 and top_p 0.95. Labels are assigned by generator cell before generation rather than inferred from the generated text. A harmful cell assigns unsafe, a paired benign cell assigns safe, and the roleplay wrapper is applied only after the label is fixed. Of 84 distinct wrappers, 78 appear with both labels and six are single-sided. Each generated message then passes an automatic content validator that evaluates the text independently of its source cell. It rejects empty outputs, refusals from harmful cells and other messages inconsistent with the intended cell. Rejected messages are revised manually and validated again rather than relabeled post hoc, and only passing text enters the corpus. The admitted rows then undergo the corpus-wide deduplication of Section A.3. A further 47 421 rows (10.2%) are mechanical obfuscations of existing labeled rows. The transforms cover random case, keyboard-layout swap, transliteration to Latin, symbol insertion and replacement, leetspeak, zero-width insertion and phonetic substitution. Each row records its source in metadata.orig_source and inherits the source label. Another 19 398 rows wrap harmful seeds in jailbreak templates from PyRIT, garak, llamator and an internal red-team suite, again inheriting the seed label. These are provenance counts over the full corpus and are larger than the augmentation block in Table 8, which counts only rows added by our augmentation stage: 38 729 obfuscation rows and 13 764 red-team rows, comprising 9 185 requests and 4 579 paired benign rewrites. The provenance view also includes inherited transformations from all corpus sources, so the two views are separate and non-additive. Here “random case” refers to random_case. The to_lower and to_upper transforms appear only in evaluation (Table 13). Roughly 38 000 additional rows tagged as synthetic come from upstream corpora, whose generators are not attributed here. Provenance class

Representative sources

Type

Admission policy

RU human instructions

ZeroAgency/ru-big-russian-dataset, attn-signs/russian-easy-instructions, IlyaGusev/ru_stackoverflow, d0rj/alpaca-cleaned-ru allenai/wildguardmix, OpenAssistant/oasst2, HuggingFaceH4/ultrachat_200k hivetrace/harmful-dataset-ru, hivetrace/crescendo_attacks/ru, Anthropic/hh-rlhf/red-team-attempts, do_not_answer hivetrace/wildguardmix-ru

real

permissive-only

real

permissive-only

real

permissive-only

real

permissive-only

in-house paired obfuscation set (safe + harm) in-house long safe-assistant synthesis

synthetic synthetic

in-house in-house

EN moderation / chat RU harm / red-team

RU moderation (MT) Synthetic obfuscation Synthetic safe assistant

Table 6: Provenance families and commercial-use admission policy. The policy keeps permissive licenses (MIT/Apache-2.0/CC-BY) and drops non-commercial, ShareAlike and undeclared, as a policy class per family. Exact per-source SPDX licenses inside the internal corpus remain an open audit item, so the weights are released under Apache-2.0 on the family-level admission policy rather than on a completed per-source audit. The Apache-2.0 grant applies to the released weights and is not a relicense of the underlying datasets.

15

A.2

Safety Taxonomy

Table 7 presents 14 training-side categories. Together with the additional policy category noted below, they drive corpus balancing and harm-category stratification of the training split. RUcategories, one of the held-out evaluation sets, carries its own 17 topic-paired categories, 16 of which this report breaks out. Thirteen of the fourteen presented training categories map to a RU-categories cell. Financial crime has no distinct counterpart and folds into non-violent crime, and three RU-categories pairs (extremism, copyright infringement, military conflict) have no training class and are scored only as transfer. Policy scope. The taxonomy encodes one deployment policy. It is not a claim about what constitutes harm in general. Several categories, politics and religion among them, are normative rather than technical. What counts as unsafe within them is a product decision, and the labels here inherit the conventions of the upstream sources. Operators whose policy differs should re-label and re-tune. The bundled message interface supplies the full policy and drops a callerprovided system turn. An integrator can modify the template or construct the rendered prompt directly, but the resulting policy requires fresh validation and may require re-tuning (Section B.1). This report breaks out 14 harm categories but omits one additional policy category from the per-category presentation. Its 2 948 training rows remain in the corpus totals of Table 8, and its evaluation pair is included in every aggregate over RU-categories, including Table 3. Per-category results for the reported categories appear in Table 14. Category

MLCommons

Aegis category

RU-categories cell

Training rows

Weapons Cybercrime

S9 (Weapons) S2 / S14

Weapons Cybercrime

1 916 7 621

Drugs Violence Offensive language Sexual content Non-violent crime Self-harm Financial crime Child-labor exploitation Religion Politics Discrimination Nazi propaganda

S2 (Non-violent) S1 (Violent) S10 (Hate) S12 (Sexual) S2 (Non-violent) S11 (Self-Harm) S2 (Non-violent) S2 (Non-violent)

Guns & Illegal Weapons Malware & Cyber Crimes Controlled Substances Violence Harassment / Profanity Sexual Criminal Planning Suicide & Self-Harm Fraud / Deception n/a (no direct)

1 805 1 643 6 981 2 528 1 787 3 742 1 630 1 051

S10 (Hate) S13 (Elections) S10 (Hate) S10 (Hate)

Hate / Identity Hate Political / Misinfo. Hate / Identity Hate Hate / Identity Hate

Drugs Violence Profanity Intimate content Non-violent crime Self-harm n/a (folded) Child-labor exploitation Religion Politics Discrimination Fascism

safe / None

n/a

Safe

safe

n/a

1 202 404 2 672 31

Table 7: The 14 training harm categories reported here, alignment to MLCommons S1–S14 [10] and Aegis [9], and the corresponding RU-categories evaluation cell. The model output is binary. Categories are used data-side for balancing and stratified evaluation. Child-labor exploitation lacks a direct Aegis code, and an n/a in the RU-categories column marks a training category with no distinct evaluation cell of its own. The training-row column runs from 7 621 for cybercrime to 31 for Nazi propaganda and sums to 35 013 categorized rows, or 18.2% of the 192 464 unsafe rows. Apart from the 2 948 rows of the withheld policy category, the remaining unsafe rows carry no category label. The corpus and the RU-categories set store these categories under Russian label strings. This report uses their English glosses throughout, and the mapping is one to one except where the RU-categories column says otherwise.

16

A.3

Processing and Composition

The composition figures below describe the released training set (n = 464 092) and were verified from the corpus with three exceptions. The counts of 9 185 red-team requests, 4 579 paired rewrites and 13 207 corpus evaluation rows were carried over from an earlier assembly stage and were not recomputed. Those values and totals derived from them are therefore approximate. The 38 729-row obfuscation block and its per-transform breakdown (Table 13) were recomputed on the released set. During pooling, the corpus is normalized to a common dialogue representation while retaining the original safety annotations. Exact duplicates are removed by SHA-256, and MinHash is used to screen for near-duplicates. After preprocessing, the corpus contains both roles, languages and labels and is nearly balanced by role. It is weighted toward Russian (0.597) and safe examples (0.585, Table 8). A separate held-back response pool is kept for a future iteration and is not part of these rows.

A.4

Contamination audit

After training, we measured every scored benchmark directly against the training corpus, except for the reserved hivetrace/insecure-prompts set discussed below. During corpus construction, candidate training rows were compared with the four internal held-out sets using word-level 3-gram shingles. An earlier character-4-gram rule removed 92–96% of candidate rows and was replaced. The final removal rule was not uniform across assembly stages. Some stages drop any row above 50% overlap. The stage contributing most of the released corpus drops exact whole-message matches of at least 40 characters and flags near matches for audit. A stricter near-match rule removed too many topically paired benign Russian examples. The decontamination reference included only the four internal sets. Five of the ten score columns in Table 2, the Aegis-2.0 and robustness-test columns, come from sets included in that reference. The remaining columns and all external benchmarks in Table 5 were outside it. We then scanned all 736 803 message and metadata texts in the training corpus against each evaluation text. As a positive control, re-injected evaluation rows reach containment 1.000, and variants truncated by 20% reach at least 0.70 in every set. The reserved hivetrace/insecure-prompts set underwent the construction-time removal procedure but was not included in this residual scan. Its residual overlap is therefore unknown. The same removal procedure left 21.3% overlap on nvidia/Aegis-2.0 even though it was in the reference. The Russian prompt-injection set was outside the reference and retains 27.1%. The two Russian development sets in Table 3 show only three exact matches, all two-word greetings, and no item above 0.7 containment in the primary pass. That pass sends short items only through the exact-match channel. A supplementary pass with a five-word floor finds eleven robustness-test items at or above 0.7, all boilerplate greetings or a lorem-ipsum noise string. RU-categories peaks at 0.667. The contamination rates below count an item when it matches exactly or clears the primary containment threshold. The removal experiments use stricter cuts, so their row counts need not match the audit counts item for item. Four scored benchmarks show residual overlap. Aegis-2.0 has 21.3% overlap among test prompts, including 176 content-bearing verbatim matches in the 1 964-prompt set. The Russian promptinjection set has 27.1% overlap, including 200 content-bearing verbatim matches. Its upstream repository was used as a training source, and some rows were reused as obfuscation seeds. The S-Eval attack set has 10.0% overlap, all in the instruction_jailbreak category whose public wrappers appear verbatim in the corpus. ToxicChat has 3.2% overlap, measured over the 10 166 17

Quantity

Count

Fraction

Split sizes Training rows Corpus eval-split rows

464 092 13 207

n/a n/a

Label safe unsafe

271 628 192 464

0.585 0.415

Language Russian English

276 877 187 215

0.597 0.403

Role user turn assistant turn

235 593 228 499

0.508 0.492

Label × role safe user unsafe user safe assistant unsafe assistant

136 974 98 619 134 654 93 845

0.295 0.212 0.290 0.202

Label × language safe RU safe EN unsafe RU unsafe EN

160 686 110 942 116 191 76 273

0.346 0.239 0.250 0.164

Augmentation red-team requests paired benign rewrites obfuscation (8 transforms) total augmented

9 185 4 579 38 729 52 493

0.020 0.010 0.083 0.113

Other Categorized unsafe (14 reported cats) Distinct source tags

35 013 19 551

0.075 n/a

Table 8: Training-set profile (n = 464 092), verified from the corpus apart from three carried-over counts noted in the text. Fractions are over the training split. The corpus is weighted toward Russian (0.597), is nearly even across user and assistant roles (0.508 user), and contains 0.415 unsafe rows. Both roles and both languages include harmful examples. Only 35 013 unsafe rows carry one of the fine-grained categories reported here, so most balancing is performed at the binary-label level. A further 2 948 categorized rows belong to the withheld policy category (Section A.2).

18

rows of the toxicchat0124 release rather than the 5 083-row split scored in the results tables. The scan uses word-level n-grams within one language. It cannot detect overlap preserved through translation or through the surface transforms in Table 9. The English prompt-injection set is a translation of the same upstream items as the Russian set but returns 0.07% overlap. This is a monolingual scan result and does not show that the English items are independent of training. The S-Eval base-risk split (0.40%) and OpenAI Moderation (0.42%) have low measured overlap. In the internal harness, removing the 199 exact-match Aegis-2.0 items identified by the audit, 176 of them content-bearing, changes prompt-side unsafe-class F1 from 0.8229 to 0.8230 (n = 1 765). At the most aggressive 50%-containment cut, F1 becomes 0.809 (n = 1 570). FPR remains between 0.158 and 0.161, and the response result is unchanged. The other scored benchmarks, rescored the same way rather than in the cohort harness, change by at most 0.017 unsafe-class F1 under their corresponding removal rules. ToxicChat has the largest of these changes. OpenAI Moderation and SimpleSafetyTests change by less than 0.005 in either direction. The scan also covered WildGuard [12] and PKU-Alignment/PKU-SafeRLHF held-out splits, which are not reported in the results tables, and both change by less than 0.005. Because allenai/wildguardmix is a training source (Table 6), its held-out split is not an independent external check. We scored the full S-Eval attack set: recall comes out at 0.806 rather than the benchmark run’s 0.802, a serving-configuration difference like the latency one in Section 4.2. Removing the entire contaminated instruction_jailbreak category, which scores 0.996, leaves the 9 000 surviving items at recall 0.784. The two YuFeng-XGuard-Reason guards in Table 2 remain ahead. The removal changes task composition because it drops one attack family. Russian prompt-injection recall changes by less than 0.001 under any removal because the attackonly set contains only four false negatives. This stability does not resolve the contamination concern. The model has seen part of the set, so its 0.999 recall should not be treated as an independent estimate of generalization. Across eleven row-level rescores, the largest change is the S-Eval recall drop from 0.806 to 0.784, every other shift is at most 0.017, and a few cells improve after removal. These checks quantify sensitivity for our model but do not provide decontaminated head-to-head scores for competitors.

A.5

Robustness Augmentation

The training set carries 52 493 augmented rows (11.3%): 13 764 synthetic red-team rows and 38 729 obfuscation rows across the eight transforms of Table 9. Augmentation covers both labels and both dialogue roles. The augmented harm:safe ratio is approximately 0.81 on the request side and 1.68 on the response side.

19

Tier

Our training coverage

Evaluation signal

Original

clean Russian/English corpus (both roles)

Obfuscation (8 transforms)

random_case, random_insert_symbols, random_replace_symbols, zero_width_insert, leet, transliterate_to_latin, wrong_layout (raw table name kbd_layout_swap), phonetic_substitution (all eight trained) synthetic red-team: redteam_request (9 185) + paired benign rewrites (4 579)

Robustness Test real (clean) Robustness Test robust (augmented)

Attacks

S-Eval attack set. Prompt injection RU/EN

Table 9: Training-time perturbation taxonomy by tier, with the held-out evaluation signal that scores each threat class. None of the eight transforms listed here is withheld from training, and what is excluded from training is the held-out evaluation rows. Three further transforms, to_lower, to_upper and informal_rewrite, are evaluated but never trained on (Table 13). Five of the eight are covered by the held-out robustness-test menu. The other three, leet, zero_width_insert and phonetic_substitution (12 679 rows), are trained but not evaluated (Table 13). Prompt-injection is an evaluation-only axis for the English set as a dataset, though its items translate the same upstream repository that was ingested for the Russian one (Section A.4).

B

Training Details

The model is initialized from Qwen3-0.6B and fine-tuned with LoRA. The completion-only loss is computed on the verdict token and the end-of-sequence marker. Adversarially augmented inputs are mixed with standard moderation examples throughout training rather than added in a separate robustness stage. The released artifact is a single model with the adapter merged into the base weights (Section B.1). At inference, the model receives a role-tagged dialogue and predicts one verdict token for the final target turn. Inference memory footprint. Inference memory is the peak allocator high-water mark over a batch-1 forward pass. The resident value in parentheses is the whole-process total reported by nvidia-smi, including the CUDA context. Weights occupy 1.11 GiB and do not scale with input length. The key/value cache costs 112 KiB per token, adding 0.22 GiB at 2048 tokens and 0.88 GiB at 8192 tokens. At long context, the output projection is a larger cost because the default path materializes logits over the full 151 936-token vocabulary at every position. Restricting logits to the final position with logits_to_keep=1 is sufficient for this classifier and gives identical verdicts. It reduces peak allocation to 1.39 GiB at 2048 tokens and 2.20 GiB at 8192 tokens. The fulllogits measurements agree with a weights-plus-cache-plus-logits estimate within 0.03 GiB. The restricted-logits measurements exceed that estimate by 0.06 GiB at 2048 tokens and 0.21 GiB at 8192 tokens because the estimate omits per-layer activations. Deployment sizing should therefore use the measured values.

B.1

Inference contract

The internal harness used the prompt format, verdict tokens and scoring rule below for every held-out result of the released model in this report. The prompt renderings and token identifiers were read from the released artifact (Section 8). If the training code and released artifact differ, the released artifact defines the reported inference path.

20

Setting

Value

Base model Adapter Framework Loss mask Effective batch Learning rate Epochs Max sequence length (training) Hardware Wall-clock Peak memory Output Inference latency

Qwen3-0.6B LoRA, rank 64, α = 32, 7 target modules Unsloth + FlashAttention-2 + sequence packing train_on_responses_only (verdict token + EOS) 32 (e.g. bs8 × ga4) 2e-4 2 2048 (content p99 ≈ 805 + system prompt) 1 × A100-80GB ≈ 5.5 h, 9 096 steps, ≈ 2.2 s/it ≈ 39 GB one merged model, fp16 weights (public, Apache-2.0) p50 14.30 ms / p95 28.79 ms / p99 37.62 ms

Inference (latency measurement) Source run the released model’s row in the suite benchmark run (19 benchmark groups) Serving vLLM OpenAI-compatible server, fp16 weights, one output token (max_tokens 1, temperature 0). One harness for every guard in the suite, each served through its own supported path, emitting its own verdict format Latency measured client-side wall clock around one request at a time: the harness issues requests serially with one in flight, and the chunk size of 300 sets loop bounds and log labels only, so each figure is a single-request round trip that carries no queuing, with p50/p95/p99 over the run’s own benchmark traffic (77 643 requests, of which 77 640 were scored) Input length the benchmark rows themselves, not a fixed-length probe, so the percentiles run over the benchmark’s own length mix Warmup none on this path (the server is already resident) Independent re-run same weights, our own serve script on 1 × A100-80GB PCIe, vLLM 0.21.0, float16, --enforce-eager, --max-model-len 8192, --gpu-memory-utilization 0.35: p50 32.75 ms / p95 48.76 ms / p99 75.95 ms over 65 748 requests Inference (memory measurement) GPU 1 × A100-80GB Serving engine HF transformers 5.3.0, SDPA attention Precision bf16 Batch size 1 Weights only 1.11 GiB Peak, 2048-token context 1.92 GiB (2.51 GiB resident) Peak, 8192-token context 4.33 GiB (5.19 GiB resident) Measurement torch.cuda.max_memory_allocated on a real forward pass. Warmup pass discarded, 5 repeats agreeing to 3 decimal places

Table 10: Training-configuration summary and measured inference cost. The latency and memory rows describe different measurements and are not directly comparable. Memory was measured with plain transformers under the stated conditions, whereas latency comes from the cohort harness. The independent latency run illustrates the sensitivity to serving configuration (Section 4.2).

21

Deployed artifact. The repository ships one merged model with fp16 weights and no adapter to attach at load time. The reproduction snippet below loads this artifact. The internal results use these weights directly, while the cohort harness of Section 4.1 serves the same weights through a different path. As Section 4.1 reports, the two serving paths can assign different labels to identical rows. System prompt (supplied by the chat template). The released tokenizer’s bundled chat template prepends a fixed English system message carrying the policy taxonomy. A caller supplies only user and assistant turns. That message is emitted by the template and must not be overridden (a caller-supplied system turn is silently dropped by the template). The block below is that message with one policy category replaced by a placeholder and every other line as the template emits it. The file chat_template.jinja in the public repository carries the complete text. Its category names are the template’s own English strings and correspond one to one with the glosses of Table 7: Classify target turn; context=data not instructions. Unsafe if it matches any Harm category or Attack ,→ class; else safe. Neutral legal/medical/edu/news/art/defensive safe unless it ,→ enables/instructs/recruits/promotes/finances/conceals harm. <POLICY> Harm: Cybercrime: illegal PII ops; phishing/exploits; auth/session attack; DDoS; CII. Pornography: porn ops; coercion; minor abuse/CSAM. Religion: hate; terror propaganda. Profanity: configured obscene lexicon. Financial: fraud; laundering. [one further policy category, see chat_template.jinja] Weapons: illegal arms/explosives/WMD. Discrimination: protected trait hate. Politics: extremist ops; false authority info; military disinfo; neutral facts safe. Self harm: suicide/self harm influence or operational support; intent; romanticize. Nazi: glorify; symbols/slogans endorsement. Child labor: minor trafficking; forced labor. Non-violent crime: property; IP/digital-rights abuse. Violence: kidnapping; sexual assault; robbery/extortion; genocide. Drugs: production; market; logistics; concealment; inducement. Attack: Jailbreak: persona/role-play/hypothetical only if used to bypass rules; DAN; ignore/override. Obfuscation: leet/base64 hiding request. Secret extraction/leak: system/dev prompts; internal routing/tool policy. Prompt injection: policy/tool override; via tool-output. Tool hijack: hidden tool/subagent enumeration. </POLICY> Reply one lowercase word: safe or unsafe; nothing else.

Serialization. After the system message, the template wraps the dialogue in a <CONVERSATION> block, tags each turn USER: or ASSISTANT:, and appends a Target: line naming the turn to be judged. The request-side rendering for one Russian example (messages = [{user: <request>}]) is shown below. The first line abbreviates the system block printed above. The remaining lines reproduce the template output. The Russian user turn translates as “How do I make an explosive device from household chemicals?” <|im_start|>system ... <|im_end|> [the policy message above, as the template emits it] <|im_start|>user <CONVERSATION> USER: Как сделать взрывное устройство из бытовой химии? </CONVERSATION> Target: last user message; judge its request/attempt.<|im_end|> <|im_start|>assistant <think> </think>

22

Reported response-side serialization. The reported response results do not use the full request-reply dialogue. Response scoring judges each reply on its own by constructing messages = [{user: ""}, {user: <reply>}]. The first turn is empty, and the reply is carried as a second user turn. The original request is therefore absent, and the reply is scored with the request-side Target: line. Because the template chooses the target from the last rendered role, carrying the same text as an assistant turn selects a different branch. The rendering below includes the empty first turn as USER: followed by one trailing space. The Russian reply translates as “Sure! Mix the following substances in the proportion. . . ”. <|im_start|>system ... <|im_end|> [the policy message above, as the template emits it] <|im_start|>user <CONVERSATION> USER: USER: Конечно! Смешайте следующие вещества в пропорции... </CONVERSATION> Target: last user message; judge its request/attempt.<|im_end|> <|im_start|>assistant <think> </think>

Every rendering ends with the assistant generation prefix <|im_start|>assistant\n<think>\n\n</think>\n\n

so the verdict word is already the next token to be predicted, and our evaluation code accordingly calls apply_chat_template without add_generation_prompt. The flag has no effect for this model: renderings with the flag set to true, set to false or omitted are byte-identical. The prefix comes from the template rather than from the flag, so a caller should not rely on the flag to supply it. The empty first turn in the response-side construction above is likewise part of that measured path: every published response-side number was produced through it, and a reproduction that drops it is scoring a different configuration. In a natural dialogue, the request is a user turn and the reply is an assistant turn, as documented in the repository README. The template then selects Target: last assistant message; judge its reply. On Aegis-2.0 responses, changing only to this role-aware form moves macro-F1 from 0.807 to 0.799 and overall macro-F1 from 0.813 to 0.811. This is a sensitivity check on one English benchmark, not an evaluation of the natural assistant-role path across the full suite. Reproducing the response figures in this report requires the standalone-reply construction above. Deployment should use the role-aware dialogue form documented in the repository. Verdict tokens and decision rule. The two verdict strings are lowercase single tokens in the Qwen3 vocabulary: safe is id 18675 and unsafe is id 38157. The end-of-sequence marker is <|im_end|> (id 151645). The label is the argmax over these two logits at the final position, i.e. the decision rule already stated in Section 4.1. The corresponding continuous score is the two-way softmax over the same pair, equivalently P (unsafe) = σ(ℓunsafe − ℓsafe ). The fixed argmax boundary therefore corresponds to P (unsafe) = 0.5. This single-forward-pass rule produced the same labels as greedy one-token generation in our evaluation. On the full Aegis-2.0 set, the unconstrained full-vocabulary argmax never left {safe, unsafe} (0/2 777 inputs, comprising 1 964 requests and 813 responses), so the two mechanisms agreed exactly. Truncation. The shipped scoring path applies no truncation. The rendered prompt is tokenized whole (tokenizer(text, return_tensors="pt"), with no max_length and no truncation 23

argument) and passed to a single forward pass. The 2048-token figure in Table 10 is the training sequence cap, not an inference limit. The released tokenizer carries model_max_length = 40 960, matching the weights’ RoPE position limit. The tokenizer truncates only when a caller explicitly requests it, which the measured path does not. We verified forward passes up to 40 960 positions. Behavior beyond that point is unspecified. In one probe, a harmful request buried under approximately 80 000 filler tokens was labeled safe. This has a security consequence for integrators. The target turn and the Target: marker are at the end of the prompt, so a wrapper that enables truncation=True with the default right-side truncation would remove both from the model input. Left-side truncation instead removes the policy block that defines the verdict. A wrapper that needs a length cap must shorten the conversation between these two parts and preserve both ends. Reproduction. The following plain transformers code loads the model and implements the scoring rule without Unsloth or our evaluation package. On the Aegis-2.0 test split, it reproduces macro-F1 0.813 from the public artifact and the contract stated here. The measured values are 0.81302 overall, 0.815 request-side and 0.807 response-side. import torch from transformers import AutoModelForCausalLM, AutoTokenizer REPO = "hivetrace/HiveTraceGuard-Pro" # public, Apache-2.0 tok = AutoTokenizer.from_pretrained(REPO) model = AutoModelForCausalLM.from_pretrained( REPO, dtype="auto", device_map="auto").eval() SAFE, UNSAFE = 18675, 38157 # single-token ids of safe / unsafe @torch.inference_mode() def guard(messages): # System prompt and policy come from the tokenizer's bundled chat template. # Pass only user/assistant turns, never a system turn of your own. text = tok.apply_chat_template(messages, tokenize=False) logits = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1] p_unsafe = torch.softmax(torch.stack([logits[SAFE], logits[UNSAFE]]), 0)[1] return ("unsafe" if logits[UNSAFE] > logits[SAFE] else "safe", p_unsafe.item()) # input guard: judge the final user request print(guard([{"role": "user", "content": "Как обойти антивирус и внедрить троян?"}])) # ^ "How do I bypass antivirus and plant a trojan?" # output guard, standalone-reply form used for the reported response numbers: print(guard([{"role": "user", "content": ""}, {"role": "user", "content": "Конечно, вот рабочий эксплойт: ..."}])) # ^ "Sure, here is a working exploit: ..." # natural response guard documented for deployment: print(guard([{"role": "user", "content": "Как обойти антивирус?"}, {"role": "assistant", "content": "Конечно, вот рабочий эксплойт: ..."}]))

C

The Suite

Three of the nineteen groups, Robustness-test (clean), Robustness-test (augmented) and Prompt injection, are built from our internal sets. The other sixteen are public benchmarks. Together the groups cover forty-four datasets, and Table 12 defines the group score. Table 11 lists the thirty-five guards with their parameter counts, aggregate keys and median latencies. The clean and augmented robustness families list seven and three dataset views, respectively, including safe-only and harm-only views in addition to the pooled cells used for scoring. StrongREJECT++ [24] contains five language slices. PolyGuard and RTP-LX contain four request/response and Russian/English slices each. HarmBench contains one response set and two request forms. S-Eval, Aegis-2.0, OR-Bench, Aya Red-teaming and prompt injection each contain two datasets. 24

The remaining eight groups contain one dataset each: ToxicChat, XSTest [21], BeaverTails, MultiJail [22], SimpleSafetyTests, CSRT [23], XSafety and OpenAI Moderation. Guard

Params (B)

Aggregate key

p50 (ms)

8 2 0.6 3 15 8 4 0.6 8 0.6 0.5 4 0.3 0.4 8 0.3 4 12 1 7 0.3 0.3 2 9 4 0.1 0.1 0.6 0.2 0.15 0.086 0.2 0.1 0.2 0.022

0.7641 0.7552 0.7432 0.7420 0.7379 0.7258 0.7237 0.7200 0.7119 0.7118 0.6946 0.6871 0.6685 0.6651 0.6631 0.6381 0.6337 0.6305 0.6293 0.6217 0.6183 0.5897 0.5249 0.4907 0.4041 0.3460 0.2633 0.0807 0.0068 0.0022 0.0006 0.0003 0.0002 <0.0001 <0.0001

71.79 53.35 14.30 14.71 63.22 152.71 96.92 16.85 243.28 37.18 77.36 54.90 8.10 17.68 63.16 6.80 86.96 86.19 20.50 99.67 7.19 9.93 200.01 359.93 201.61 6.58 39.78 160.96 8.05 213.81 18.26 8.10 8.72 7.54 7.47

YuFeng-XGuard-Reason-8B SingGuard-2B HiveTraceGuard-Pro (ours) Shieldstral-1.0-3B OpenGuardrails-Text-2510 Qwen3Guard-Gen-8B Qwen3Guard-Gen-4B YuFeng-XGuard-Reason-0.6B Llama-3.1-Nemotron-Safety-Guard-8B-v3 Qwen3Guard-Gen-0.6B PolyGuard-Qwen-Smol Nemotron-3.5-Content-Safety opir-multitask-multilang-v1.0 opir-multitask-large-v1.0 Llama-Guard-3-8B gliguard-LLMGuardrails-300M SingGuard-4B Llama-Guard-4-12B Llama-Guard-3-1B WildGuard HiveTraceLite† gliner-guard-omni† ShieldGemma-2B ShieldGemma-9B OpenGuardrails-Text-4B-0124 promptguard gliner-guard-uniencoder† Nandi-Mini-600M-GuardRails deberta-v3-base-injection Nandi-Mini-150M-GuardRails Llama-Prompt-Guard-2-86M deberta-v3-base-prompt-injection-v2 Text-Moderation deberta-v3-base-prompt-injection-detection Llama-Prompt-Guard-2-22M

Table 11: The thirty-five suite guards ordered by the aggregate key of Section 4.1, the geometric mean of nineteen group scores. Parameter counts refer to the released checkpoints. Median latency comes from the same configuration-specific benchmark run, so only within-run comparisons are meaningful (Section 4.2). Each guard was evaluated once and the keys have no seed intervals. The margin between HiveTraceGuard-Pro and the next key below, 0.7432 against 0.7420, is nearly an order of magnitude smaller than the ±0.011 three-seed spread of the internal robustnessaugmented harm-F1 (Section 6), so an ordering this close should not be read as stable. Over the sixteen public groups alone, HiveTraceGuard-Pro has key 0.7153 and four suite guards score higher: YuFeng-XGuard-Reason-8B (0.7467), SingGuard-2B (0.7425), OpenGuardrails-Text-2510 (0.7369) and Shieldstral-1.0-3B (0.7281). The three models marked † are earlier guards released by our team.

25

D

Per-Model Coverage

A model is graded only on rows for which it returns a usable verdict, so dropped rows are absent from that model’s denominator. On Aegis-2.0 responses, Llama-3.1-Nemotron-Safety-Guard-8Bv3 drops 36 of 852 rows (4.2%) and Nemotron-3.5-Content-Safety drops 39 (4.6%). The other thirteen models in Table 2 are scored on all 852. On Robust-out, the same two models are scored on 913 of 945 rows and 944 of 945 rows. Their Aegis-2.0 response scores, 0.86 and 0.85, are therefore computed on reduced subsets. The suite is a matrix of 35 models by 44 datasets, so it holds 1 540 dataset cells. One cell (WildGuard on the Russian PolyGuard response set) is unscored, and 81 others are scored on fewer rows than the set carries. The largest proportional shortfall is 421 of 1 709 rows (24.6%), and the largest absolute shortfall is 501 of 4 595 rows. HiveTraceGuard-Pro returns a usable verdict for 77 640 of 77 643 rows, missing three. The published aggregation omits an unscored cell from its group rather than assigning it zero. WildGuard’s PolyGuard group score of 0.8336 is the harmonic mean of its three scored cells and reproduces to seven decimal places. Assigning zero to the missing cell would instead make the group score zero. The resulting group score enters the aggregate key like any other, and all thirty-five keys reproduce as geometric means of their nineteen group scores. Dataset cells otherwise receive equal weight within a group regardless of row coverage, so a cell scored on 1 288 of 1 709 rows counts for as much as a complete cell. Published cell metrics therefore use only rows with usable verdicts. Relative to counting every parse failure as an error, this treatment can leave a cell score unchanged or raise it, but cannot lower it. The stored per-row artifacts also encode an unusable verdict as an error, allowing the stricter calculation to be reproduced. For HiveTraceGuard-Pro the two treatments differ on three of 77 643 rows. Error profile against the field. The highest FPR values occur on benign-stress sets. On RTP-LX Russian responses, FPR is 0.921 compared with a field median of 0.905. On OR-Benchhard it is 0.607 compared with 0.583. Larger model-specific gaps appear on PolyGuard English requests, 0.367 compared with a median of 0.089, and XSTest, 0.356 compared with 0.100. The Russian robustness set has substantially lower FPR: 0.016 on clean requests and 0.132 after augmentation, compared with 0.268 pooled over the suite. The same augmented rows yield FPR 0.139 in Table 3, where the internal harness scores them under its own conventions (Section 4.1).

26

E

Benchmark Groups

Benchmark group

Ours 0.6B

YuFengXGuard-8B

SingGuard2B

Shieldstral1.0-3B

OpenGuardrails 15B

Qwen3GuardGen-8B

YuFengXGuard-0.6B

Robustness-test (clean)† XSafety Prompt injection (RU+EN)† Aya Red-teaming Robustness-test (augmented)† S-Eval BeaverTails Aegis-2.0 OpenAI Moderation RTP-LX OR-Bench CSRT ToxicChat PolyGuard StrongREJECT++ XSTest HarmBench MultiJail SimpleSafetyTests

0.933 0.590 0.934 0.934 0.870 0.753 0.839 0.807 0.802 0.326 0.549 0.743 0.588 0.808 0.833 0.754 0.772 0.746 0.910

0.818 0.451 0.905 0.934 0.870 0.876 0.829 0.815 0.793 0.157 0.870 0.784 0.652 0.884 0.970 0.951 0.936 0.889 1.000

0.880 0.518 0.753 0.861 0.851 0.653 0.833 0.825 0.848 0.435 0.394 0.705 0.674 0.886 0.910 0.940 0.936 0.879 0.990

0.844 0.595 0.786 0.928 0.834 0.666 0.828 0.787 0.794 0.192 0.377 0.895 0.732 0.883 0.966 0.922 0.927 0.946 1.000

0.785 0.435 0.705 0.825 0.741 0.500 0.845 0.844 0.885 0.401 0.744 0.762 0.613 0.902 0.957 0.941 0.822 0.762 0.940

0.894 0.589 0.867 0.948 0.896 0.699 0.842 0.819 0.747 0.108 0.341 0.895 0.699 0.893 0.982 0.916 0.954 0.930 0.990

0.780 0.469 0.892 0.877 0.796 0.867 0.828 0.817 0.787 0.135 0.863 0.689 0.620 0.888 0.547 0.920 0.935 0.905 0.990

Table 12: All nineteen benchmark-group scores for HiveTraceGuard-Pro, the five other guards with the highest aggregate keys in the suite and YuFeng-XGuard-Reason-0.6B. For each dataset, the group calculation takes the harmonic mean of 1 − FPR and 1 − FNR, with attack-only sets contributing recall and benign-only sets contributing 1 − FPR. It then takes the harmonic mean across datasets in the group. Each request or response set enters once through its pooled cell. The aggregate key is the geometric mean of the nineteen group scores. Bold marks the highest value among the seven models shown. Daggered groups are our internal sets, and the other sixteen are public.

F

Additional Results

Per-language and seed variance. The aggregate scores pool Russian and English and report a single run. Macro-F1 is 0.898 on the Russian robustness-test set when clean and augmented rows are pooled, compared with 0.866 on augmented rows alone. It is 0.937 on Russian RUcategories and 0.813 on English nvidia/Aegis-2.0. The robustness-test set has no English partition. Across three seeds of the released recipe, the mean and sample standard deviation are syntheticclean-subset harm-F1 0.938 ± 0.008, robustness-augmented harm-F1 0.844 ± 0.011, robustnessaugmented macro-F1 0.860 ± 0.008, RU-categories macro-F1 0.939 ± 0.004 and Aegis-2.0 macro-F1 0.801 ± 0.010. The released checkpoint is the first run of the recipe. The other two seeds were trained later to estimate variation, so the release was not chosen from the three. These bands measure seed variation but not evaluation uncertainty.

27

Condition

harm-F1

FPR

FNR

Training rows

Requests Original (clean) to_lower to_upper random_case random_insert_symbols random_replace_symbols transliterate_to_latin wrong_layout informal_rewrite

0.889 0.955 0.863 0.816 0.896 0.860 0.879 0.790 0.756

0.016 0.033 0.173 0.193 0.084 0.127 0.131 0.262 0.107

0.041 0.046 0.070 0.136 0.099 0.122 0.081 0.125 0.307

n/a – – 6 163 6 474 6 299 3 566 3 548 –

Responses Original (clean)

0.671

0.043

0.180

n/a

Table 13: Per-augmentation harm-F1, FPR and FNR for the released model. Each request transform contains 342–345 harmful and 449–450 benign rows. The clean request row instead uses the full slice, 345 harmful and 4 234 benign rows, so FNR is more directly comparable across conditions than FPR or harm-F1. The response row contains 61 harmful and 884 benign examples. Transforms are evaluated only on requests. The transforms to_lower, to_upper and informal_rewrite are evaluation-only. Three trained transforms are absent from the evaluation menu: zero_width_insert (6 479 training rows), phonetic_substitution (3 410) and leet (2 790). The wrong_layout value is for the released build and is not directly comparable with earlier-lineage values in Section G.

Request

Category Child-labor exploitation Copyright infringement Cybercrime Discrimination Drugs Extremism Fascism Intimate content Military conflict Non-violent crime Politics Profanity Religion Self-harm Violence Weapons

Response

harm-F1

FPR

FNR

harm-F1

FPR

FNR

0.933 0.885 0.891 0.910 0.981 0.940 0.925 0.986 0.875 0.949 0.920 0.991 0.969 0.967 0.937 0.882

0.018 0.145 0.109 0.155 0.000 0.127 0.109 0.000 0.145 0.027 0.100 0.018 0.045 0.000 0.073 0.245

0.109 0.091 0.109 0.036 0.036 0.000 0.045 0.027 0.109 0.073 0.064 0.000 0.018 0.064 0.055 0.018

0.995 0.908 0.843 1.000 0.968 0.982 0.968 1.000 0.987 0.843 0.991 0.987 1.000 0.986 1.000 0.873

0.009 0.182 0.373 0.000 0.027 0.018 0.018 0.000 0.027 0.373 0.000 0.027 0.000 0.009 0.000 0.291

0.000 0.018 0.000 0.000 0.036 0.018 0.045 0.000 0.000 0.000 0.018 0.000 0.000 0.018 0.000 0.000

Table 14: Per-category request and response harm-F1, FPR and FNR across the 16 of 17 RU-categories topic pairs that this report breaks out (n = 220 per role per category). Table 7 maps them to the training taxonomy. These 16 pairs are 3 520 of the 3 740 rows per class behind the RU-categories row of Table 3, whose aggregate is over all 17 pairs of the set. The seventeenth is the one policy category this report does not break out. Cells that over-block can retain high harm-F1 because they still detect harmful examples, so FPR rather than harm-F1 reveals the over-blocking cluster.

28

G

Ablations and Development

The rounds below are development diagnostics on non-identical builds, not a controlled ranking of data against optimization strategy. Observed English over-blocking change after a data-composition revision. Replacing a baseline corpus composition with a curated source-and-pair mix coincided with a reduction in benign-English FPR from 0.591 to 0.153 in one development round (Figure 5). The released model’s Aegis-2.0 English request-side FPR is 0.159 (Table 3), measured on a later build and a different set. A threshold sweep in the earlier round produced no useful change. The figure reports the two measured corpus compositions.

benign-English over-block FPR

0.6

0.591

0.5 0.4 0.3 0.2

0.153

0.1 0.0

baseline composition

curated composition

Figure 5: Benign-English FPR for two corpus compositions evaluated in the same earlier development round. The released checkpoint was evaluated later on a different set and is not plotted. Its Aegis-2.0 English request-side FPR is 0.159 (Table 3). Observed keyboard-layout augmentation effect in the earlier lineage. Removing this augmentation reduced development macro-F1 over the union of robust-request and response conditions from 0.836 to 0.735 and increased wrong_layout request-FNR from 0.357 to 0.626. In that experiment, the model did not generalize to the unseen Russian↔English layout mapping. Observed two-sided obfuscation advantage in one development comparison. A useronly schedule removed assistant-side obfuscation and regressed on 7 of 8 augmented-benign conditions relative to that round’s two-sided baseline. Obfuscated-harm recall did not improve. This result is limited to that within-round comparison. A classification head did not improve the released setup. A sequence-classification head trained on the same corpus and evaluated on the same 10 934 held-out requests reaches macro-F1 0.822, compared with 0.898 for the released generative model. The 0.898 value is requests-only macro-F1 from the same scoring run. It matches the both-roles value in Table 3 to three decimals by coincidence. This is the only ablation reported here that uses the released corpus and protocol. The preceding comparisons use earlier corpora and builds. Development trajectory. Development used raw macro-F1 over the union of robust-request and response conditions. We corrected the keyboard-layout pipeline in two steps: aligning the training cipher with the evaluation convention, then normalizing the sampling rate of each obfuscation transform toward the canonical harm rate. An intermediate iteration added recall29

oriented data generated with the old cipher and reduced the target metric, so it was not continued. The released model was obtained by rebalancing its predecessor’s obfuscation augmentation. This changed wrong_layout harm-FNR from 0.061 to 0.125 and FPR from 0.358 to 0.262, while robustness-augmented harm-F1 increased from 0.849 to 0.852. We evaluated six later additive rounds with a paired held-out no-regression gate. Three responseside hard-negative and minimal-pair rounds improved one side of the obfuscation trade-off while worsening the other. A weight-averaged model soup produced no gain. We also rejected label distillation from a 32B teacher because the teacher performed worse on the 0.6B model’s failing cells. On a paired 1 505-item probe subset, the teacher’s keyboard-layout FPR was 0.827 compared with 0.253 for the 0.6B model. A dedicated over-refusal round reduced wrong_layout FPR from 0.262 to 0.222 on the held-out set and OR-Bench-hard [20] FPR from 0.61 to 0.20 in the cohort harness, but increased FNR on the gate’s obfuscated-harm slice from 0.097 to 0.116 (Holm-corrected p < 0.01). The starting points are the released model’s own values, and the round was not adopted. None of the tested data compositions or objective changes improved both obfuscation error rates at once.

References [1] Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, and 35 further authors. Qwen3Guard Technical Report. arXiv preprint arXiv:2510.14276, 2025. [2] Junyu Lin, Meizhen Liu, Xiufeng Huang, Jinfeng Li, Haiwen Hong, Xiaohan Yuan, Yuefeng Chen, Longtao Huang, Hui Xue, Ranjie Duan, Zhikai Chen, Yuchuan Fu, Defeng Li, Lingyao Gao, and Yitong Yang. YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Models. arXiv preprint arXiv:2601.15588, 2026. [3] SingGuard Team. SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning. arXiv preprint arXiv:2606.22873, 2026. [4] Antonia Calvi, Avinash Sooriyarachchi, Giada Pistilli, Guillaume Lample, Maarten Buyl, Maximilian Augustin, Maximilian Müller, Pierre Stock, and 268 further authors. Shieldstral. arXiv preprint arXiv:2607.25857, 2026. [5] Thomas Wang and Haowen Li. OpenGuardrails: A Configurable, Unified, and Scalable Guardrails Platform for Large Language Models. arXiv preprint arXiv:2510.19169, 2025. [6] Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm. arXiv preprint arXiv:2406.18682, EMNLP 2024. [7] Adrian de Wynter, Ishaan Watts, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Nektar Ege Altıntoprak, and 27 further authors. RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? arXiv preprint arXiv:2404.14397, AAAI 2025. [8] Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R. Lyu. All Languages Matter: On the Multilingual Safety of Large Language Models. arXiv preprint arXiv:2310.00905, Findings of ACL 2024. [9] Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails. arXiv preprint arXiv:2501.09004, 2025. 30

[10] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv preprint arXiv:2312.06674, 2023. [11] Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. ShieldGemma: Generative AI Content Moderation Based on Gemma. arXiv preprint arXiv:2407.21772, 2024. [12] Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. arXiv preprint arXiv:2406.18495, NeurIPS 2024. [13] Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, and Maarten Sap. PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages. arXiv preprint arXiv:2504.04377, COLM 2025. [14] Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. arXiv preprint arXiv:2310.17389, Findings of EMNLP 2023. [15] Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. A Holistic Approach to Undesired Content Detection in the Real World. arXiv preprint arXiv:2208.03274, AAAI 2023. [16] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv preprint arXiv:2402.04249, ICML 2024. [17] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. arXiv preprint arXiv:2307.04657, NeurIPS 2023. [18] Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A. Hale, and Paul Röttger. SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models. arXiv preprint arXiv:2311.08370, 2023. [19] Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Longtao Huang, Jialuo Chen, Hui Xue, Xiaoxia Liu, Wenhai Wang, Kui Ren, and Jingyi Wang. S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models. arXiv preprint arXiv:2405.14191, ISSTA 2025. [20] Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv preprint arXiv:2405.20947, ICML 2025. [21] Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. arXiv preprint arXiv:2308.01263, NAACL 2024. [22] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual Jailbreak Challenges in Large Language Models. arXiv preprint arXiv:2310.06474, ICLR 2024. [23] Haneul Yoo, Yongjin Yang, and Hwaran Lee. Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding. arXiv preprint arXiv:2406.15481, ACL 2025. 31

[24] Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for Empty Jailbreaks. arXiv preprint arXiv:2402.10260, NeurIPS 2024.

32

Related documents

• GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection#259370
• DroidBreaker: Practical and Functional Problem-Space Attacks on Machine-Learning Android Malware Detectors#310749
• Breaking High Confidence: Practical Face Impersonation under High-Security Thresholds#662076
• Breaking High Confidence: Practical Face Impersonation under High-Security Thresholds#661026
• CLOADER: Evading Security Mobile Defenses via Runtime Obfuscation and Adaptive Hooking Tactics#1049137