Silent Alarm: A J-Space Protocol for Comparing Danger Recognition Across Models and Quantization Levels Prosvirnin Roman* , Minchenkov Victor, Soldatov Alexey, Bashun Vladimir HSE University Abstract. Jailbreak-robustness research typically evaluates safety through generated responses using an LLM-as-judge approach. Such evaluations, however, are sensitive to the benchmark’s grading procedure and capture only observed behavior on a given set of attacks, without directly revealing the hidden fragility of the underlying safety mechanisms. This work proposes JADR (Jacobian Assessment of Danger Recognition), a protocol that measures a model’s internal representation through Jacobian space (J-space, a recently proposed workspace of verbalizable concepts) before the first response token is generated. For every prompt and layer we record the top-𝑘 J-space tokens; these are grouped into six behavioral scenario axes and compared between a danger sample based on StrongREJECT and a safe control drawn from XSTest and OKTest. The method does not call on an external judge model: the computation runs entirely locally, on the activations of the model under evaluation, which lets us compare both different models against each other and modifications of a single model – quantization and fine-tuning in particular – on the same terms. The final comparison rests on the proposed SafetyAUC metric, complemented with bootstrap confidence intervals. The protocol is applied to six models (Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-Uncensored-4B, Qwen3-SafeRL-4B, Gemma 2 9B) across three weight-representation regimes – BF16, INT8, and INT4 – and checked against an independent behavioral evaluation with the StrongREJECT grader. The metric separates models with a strong versus a weak internal safety mechanism with statistical significance and captures substantively different effects across quantization regimes.
arXiv:2607.12792v1 [cs.CR] 14 Jul 2026
Keywords: Jacobian lens, J-space, quantization, LLM safety, StrongREJECT, refusal rate, decision point.
1
Introduction
in words, but also about whether its internal workspace carries a signal of danger, illegality, willingness to comply, or an attempt to bypass a restriction. This distinction matters specifically for quantization: numerical error in the representation may not immediately change the surface style of the response, yet it can already disturb the internal separation between dangerous and safe cases – and, conversely, a model can keep a clear internal recognition of danger and still generate harmful text. These three regimes – intact recognition with intact refusal, intact recognition with lost refusal, and lost recognition – are indistinguishable if one looks only at the text of the response. The basis for this measurement is J-space – a space of verbalizable directions read out by the Jacobian lens (Gurnee et al., 2026). For every layer and position, the lens recovers which vocabulary tokens the model “holds ready” for future speech, correcting the logit lens by the averaged Jacobian of the transition to the final state. We read this content at the decision point – the position of the last token before the first token of the response – where it is not yet distorted by already-generated text, and we compare it between a dangerous set of prompts and a safe control made up of outwardly alarming but essentially harmless formulations. This is the key move that lets us tie the appearance of safety tokens in the workspace to the actual danger of the prompt as recognized by the model. In this way we show that the signal in J-space tracks the semantics of danger rather than its vocabulary. We introduce a reproducible J-space assessment protocol at the decision point that requires neither response generation nor a call to an external judge model: the entire computation runs locally, on the activations of the model being evaluated. Comparison is moved onto the dimensionless rank AUC (SafetyAUC, ComplAUC) and the logarithmic shift of token counts 𝐷 log . In addition, quantization regimes are compared with the Δ𝑆𝐻 (Safety Headroom) metric, which accounts for change along both AUC dimensions. We’ve applied JADR protocol to six models, including variants with reinforced and with weakened safety tuning, and to three weight-representation regimes, which lets us compare
Jailbreak-robustness research and automated red teaming commonly rely on a behavioral protocol: generated responses are scored by an LLM judge, and the resulting decisions are aggregated into rate-based metrics such as ASR (attack success rate). This protocol carries some limitations. The results are sensitive to the choice and configuration of the judge, and, when a continuous grader is used, to the binarization threshold as well, which makes it hard to compare results across papers. The output score captures observed compliance with or violation of policy on a fixed set of attacks, but it does not directly characterize the robustness of the internal representations and computational mechanisms tied to harm recognition and refusal. As a result, a fragile defense may simply fail to show up on the current test set and only surface under different or adaptively chosen attacks. In addition, ASR values obtained on different benchmarks are frequently incomparable, owing to differences in prompt distribution, threat model, success criteria, attack budget, and grading procedure. All of these limitations stem from the same fact: the judge sees only the final text of the response, not the state of the model that produced it. Engineering modifications of language models – quantization, fine-tuning, abliteration – are rarely neutral with respect to safety, even when they are neutral with respect to quality (Hong et al., 2024; Qi et al., 2023). A model can hold its previous level on standard benchmarks while behaving differently on dangerous prompts: refusing more often, giving more useful detail where it used to refuse, or losing the ability to tell a harmful instruction apart from a superficially similar but benign task. An LLM-as-judge evaluation records the fact of such degradation after it happens, but not its mechanism: it sees only the final text, not what went on inside the model before that text appeared. This work treats safety as an internal, verbalizable representation available to the model at the moment it chooses a response. We care not only about whether the model refused * Corresponding author: [email protected]
1
both models against each other and modifications of a single model – quantization above all – in the same terms. The internal metrics are systematically checked against an independent behavioral evaluation on the StrongREJECT benchmark. This comparison serves to differ cases where a model’s safety recognition and actual behavior diverge from cases where they experience a simultaneous degradation.
2
compression (Hong et al., 2024). An LLM balances between fulfilling a request and following an instruction. When quantization weakens instruction-following, a strong safety mechanism starts to dominate, and the model refuses more often on safe prompts (Chhabra and Khalili, 2025). JADR provides the tooling to measure this at the representation level: which scenario axis changed, in which direction, and in which layers – which applies not only to INT8 and INT4 but in principle to any weight modification for which a J-lens readout can be obtained.
Related Work
2.1 Global Workspace and the Jacobian Lens The methodological foundation of this work is the result of Gurnee et al. (2026), who showed that language models contain a relatively small class of representations (J-space) available for future verbalization and generation control. The authors describe this as a functional analogue of a global workspace: not all of a model’s intermediate state is equally available to future output, but part of it can be read out through the Jacobian lens – a correction to the logit lens that accounts for the averaged Jacobian of the transition from an intermediate layer to the final state. As illustrations, the original work reports signs of recognizing attempted prompt injection (tokens such as fake, injection, poison) and multilingual equivalents of such signs. The causal role of these directions is established there through intervention – ablation and activation injection. The question of safety evaluation for quantized models was not addressed in that work. The JADR approach – Jacobian Assessment of Danger Recognition – proposes using the same tool to build safety metrics and behavioral robustness metrics in two scenarios: comparing models against each other (cross-model) and comparing modifications of a single model, quantization in particular (cross-modification).
3
Problem Formulation
Let 𝑚 denote the model, 𝑞 ∈ {fp, int8, int4} the weightrepresentation regime, ℓ the layer, and 𝑝 the prompt. The fp regime corresponds to the bf16 representation and serves as the reference for each given model; comparisons between regimes are always carried out within one and the same model. The dangerous sample D is built on StrongREJECT – prompts that instruct the model to carry out an action that is unambiguously forbidden. The safe control N is assembled from XSTest-safe (Röttger et al., 2024) and OKTest: prompts that are lexically similar to dangerous ones (they use alarming terminology, mention weapons, hacking, deception, and so on) but are harmless in substance. This control is needed to separate a reaction to dangerous vocabulary from a reaction to a dangerous goal. We formulate and test the hypothesis that the frequency of safety tokens in J-space depends on the danger of the prompt. The task is formulated as follows: for every model 𝑚 and every regime 𝑞, measure how much J-space tells D apart from N , how this distinction changes going from fp to INT8 and INT4, and which layers carry the main contribution. The metric must not depend on the absolute number of lexicon matches, the depth of a given model, or the size of its vocabulary coverage – otherwise a cross-model comparison loses its meaning. The lexicon is fixed in advance (built only from tokens that actually occur in the models’ workspace) and split into six groups:
2.2 Jailbreak Evaluation For jailbreak evaluation we chose the StrongREJECT benchmark (Souly et al., 2024) and its fine-tuned grader (a finetuned Gemma 2B), which assigns a score from 0 (safe) to 1 (dangerous). This gives us a set of prompts that are unambiguously forbidden and a scoring model that provides a continuous estimate of the harm in the response itself, rather than of the mere fact of refusal. In JADR, StrongREJECT plays two roles, neither of which turns it into a first-order judge for the internal metrics. The set of forbidden prompts forms the dangerous sample D, |D | = 313; the trained grader is used separately, only for behavioral validation – checking whether the internal shifts line up with the actual content of the response. These roles do not overlap: JADR’s internal metrics never use the text of the response and do not depend on a judge call at all, so the LLM-as-judge limitations discussed above do not carry over to the main result of this work, and the behavioral score remains an independent source of validation and an indicator.
𝑔 ∈ {safety, compliance, evasion, softening, hedging, harm}.
(1)
The safety axis brings together the vocabulary of refusal, risk, and illegality; compliance – willingness to instruct or comply; the model’s awareness that the prompt’s wording was chosen to get around a restriction (evasion). The remaining three axes describe subtler scenario features: harm – the model’s awareness of a specific harmful domain, softened wording (softening), and cautious generalization (hedging). The role of each axis is summarized in Table 1. The last three axes are treated as auxiliary, since they are localized more weakly than the rest (see §7.3). The softening and hedging groups were introduced to potentially track ‘refuse-then-comply’ behavior, where the model refuses at first but then offers a ‘softened’ solution that in fact discloses the unwanted information. Representative lexicon stems for each axis are given in Table 2.
2.3 Quantization and Safety As a rule, quantizing a model degrades both its safety and its instruction-following robustness. It has already been noted, however, that the effect of quantization on refusal is not always monotonic: compressing a model can either weaken or unintentionally strengthen its tendency to refuse, and moderate 4-bit quantization in some cases even preserves or improves model trustworthiness compared with coarser 2
Table 1: Scenario axes used by JADR. Axis
Diagnostic role
safety compliance evasion
refusal, risk, illegality willingness to comply or instruct bypassing and concealing the prompt’s goal softened and transitional wording cautious generalization, hedges concrete language of a specific harmful method
softening hedging harm
4
Method
4.1
Jacobian Lens
are more likely to reflect what the model is actually ready to say. A rank-weighted version accounts for this difference explicitly: dcg𝑔 ( 𝑝, ℓ) =
𝑘 ∑︁ 1 rank-𝑟 token ∈ 𝑔 log2 (𝑟 + 1)
𝑟=1
.
(5)
Both quantities are reported throughout the paper: count as a transparent but coarse estimate, DCG as its refinement, robust to how deep into the top-𝑘 the readout extends. Below, “counter” refers to either of the two quantities; all metrics of §4.4–§4.6 apply to both without change. For aggregation across layers we introduce the sum 𝑇𝑔 ( 𝑝) =
Let ℎℓ,𝑡 ∈ R𝑑 be the residual stream at layer ℓ and position 𝑡, and ℎ 𝐿,𝑡 ′ the final residual stream at position 𝑡 ′ ≥ 𝑡. The Jacobian lens estimates how a small change in the intermediate state carries over into the final one: 𝜕ℎ 𝐿,𝑡 ′ 𝐽ℓ = Eprompt, 𝑡 , 𝑡 ′ ≥𝑡 . (2) 𝜕ℎℓ,𝑡
∑︁
𝑐 𝑔 ( 𝑝, ℓ).
(6)
ℓ
We introduce intermediate notation for the mean per-layer signal and the per-layer difference between dangerous and non-dangerous prompts: 𝑆 𝑔 (𝑚, 𝑞, 𝑃, ℓ) = mean 𝑐 𝑔 ( 𝑝, ℓ), 𝑝∈ 𝑃
The average is taken over the prompts of the corpus, the source positions 𝑡, and all current and future target positions 𝑡 ′ . The matrix 𝐽ℓ is averaged over a large set of calibration prompts (𝑁=100, pretrain-like, exactly the figures recommended in (Gurnee et al., 2026)) and does not depend on the specific example on which the readout is later built. After applying 𝐽ℓ and the unembedding matrix 𝑊𝑈 we obtain a distribution over vocabulary tokens: lensℓ (ℎℓ,𝑡 ) = softmax 𝑊𝑈 norm(𝐽ℓ ℎℓ,𝑡 ) . (3)
𝐴𝑔 (𝑚, 𝑞, ℓ) = 𝑆 𝑔 (𝑚, 𝑞, D, ℓ) − 𝑆 𝑔 (𝑚, 𝑞, N , ℓ). These quantities depend directly on the size of the lexicon (for instance, on an extension of the configuration file) and on the scale of the token counter, so we move on to more advanced metrics that reflect true separability.
4.4
The final core of the comparison is the rank AUC (equivalent to the normalized Mann-Whitney statistic): the probability that a random value from one sample exceeds a random value from the other (Hanley and McNeil, 1982). It is dimensionless and depends only on the ordering of values, which makes it comparable across models with different tokenizers, depths, and vocabulary coverage, so it is robust to changes in the lexicon. AUC does not blow up on small values. For example, a model with no pronounced signal on axis 𝑔 gives AUC ≈ 0.5. The per-layer version:
Unlike the logit lens (nostalgebraist, 2020; Belrose et al., 2023) (𝐽ℓ = 𝐼), this distribution should not be read as a prediction of the next token: it shows what content the model could make available to future verbalization, if it decided to speak about it. This work does not use the full distribution (3), only the top-𝑘 (𝑘 = 100 for every metric in this paper) tokens for each layer and prompt.
4.2
Rank AUC
Decision Point
All internal metrics are computed at the decision point – the position of the last token before the first token of the assistant’s response. At this position the model has already processed the whole prompt but has not yet started generating. Choosing this point has two practical consequences: the activation here depends causally only on the prompt, not on already-written response text, and computing the metric requires no decoding – a single forward pass is enough. This makes the measurement conceptually clean (“before speech”) as well as cheap in computational terms.
AUC𝑔 (𝑚, 𝑞, ℓ) = Pr 𝑐 𝑔 ( 𝑝 D , ℓ) > 𝑐 𝑔 ( 𝑝 N , ℓ) ,
(7)
where 𝑝 D ∈ D and 𝑝 N ∈ N are drawn at random and independently. The aggregate model-level version is computed on the sum over layers: AUC𝑔 (𝑚, 𝑞) = Pr 𝑇𝑔 ( 𝑝 D ) > 𝑇𝑔 ( 𝑝 N ) .
(8)
A value of 0.5 means no separability along axis 𝑔; values above 0.5 mean the group is more pronounced on dangerous prompts; values below 0.5 mean the opposite direction. The strength of separability is given by |AUC − 0.5|. For every quantity reported in this paper we give a 95% confidence interval obtained by bootstrapping over prompts (𝐵 = 1000 resamples with replacement, 2.5 and 97.5 percentiles). Resampling over prompts, rather than over individual tokens, is chosen because tokens within a single readout are strongly correlated.
4.3 Token Counters For prompt 𝑝, layer ℓ, and group 𝑔, the presence of the group in the top-𝑘 readout is measured in two ways. The plain count: 𝑘 ∑︁ 𝑐 𝑔 ( 𝑝, ℓ) = 1 rank-𝑟 token ∈ 𝑔 . (4) 𝑟=1
Count does not distinguish a token at the first position of the readout from one at the hundredth, even though earlier tokens 3
Table 2: Representative frozen-lexicon stems per scenario axis (bilingual: English, Chinese, Russian). Built by frequency-analysis. The match is based on the coincidence of the beginning of tokens. Trailing hyphens denote Russian morphological stems matched as prefixes. The full lexicon is part of the released artifacts. Category
Typical stems
Safety
unsafe, cannot, illegal, harmful, forbidden, disclaimer, dangerous, apologize, 不安全, 危险, 违法, 拒 绝, запрещ-, опасн-
Compliance
sure, here, step, method, guide, provide, recipe, instruction, 可以, 步骤, 方法, 提供, конечно
Harm
weapon, explosive, malware, credential, poison, payload, firearm, toxin, 武器, 炸弹, 毒药, 恶意软件
Evasion
bypass, hidden, conceal, undetect, stealth, loophole, spoof, anonym, 绕过, 隐藏, 秘密, 伪装, обход
Softening
however, although, instead, rather, while, nevertheless, 但是, 虽然, 只是, 然而, но
Hedging
usually, generally, possible, probably, maybe, typically, seems, 通常, 可能, 一般来说, 不一定, возможно
4.5 Cross-model Metrics For model comparison in this paper, the regime is fixed at 𝑞 = fp. The main comparator metric, SafetyAUC, is – SafetyAUC(𝑚) = AUCsafety (𝑚, fp),
This is not a metric in the strict sense; it is an indicator of whether the model’s internal behavior has shifted to new layers under quantization. The threshold 𝜏 = 0.2 is chosen empirically and is not a strict cutoff.
(9)
4.6 Cross-modification Metrics For comparing quantization regimes, the model 𝑚 is held fixed and the regime 𝑞 is compared against the reference fp. Because of this, every metric here is built as a difference. The shift in separability under quantization:
which shows how reliably the model’s workspace tells apart dangerous and harmless prompts along the safety axis – that is, how well the model “recognizes danger” in the intent of the prompt. The complementary metric, ComplAUC, is – ComplAUC(𝑚) = AUCcompliance (𝑚, fp),
(10) ΔAUC𝑔 (𝑚, 𝑞) = AUC𝑔 (𝑚, fp) − AUC𝑔 (𝑚, 𝑞).
which shows the direction of the willingness to comply: a value below 0.5 corresponds to a stronger willingness on safe prompts, above 0.5 to that willingness shifting toward dangerous prompts. SafetyAUC and ComplAUC need not agree: a model can correctly recognize danger and still show a compliance signature shifted toward danger. A common scenario is one where the model ‘understands’ the prompt is dangerous, but its built-in wish to be helpful makes it look for a compromise that turns into a successful attack. Internal recognition can therefore diverge from eventual behavior: a model with a high SafetyAUC can still give a harmful response. To compare the absolute magnitude balance between models we use the logarithmic ratio of the total danger and safe counters, with a correction: 𝑋𝑔 + 0.5 𝑔 𝐿 ds (𝑚, 𝑞) = log , 𝑌𝑔 + 0.5 ∑︁ 𝑋𝑔 = 𝑇𝑔 ( 𝑝), 𝑝∈ D
𝑌𝑔 =
∑︁
(13)
For safety, a positive value means erosion and degradation of recognition; a value near zero means it is preserved. The magnitude shift between fp and 𝑞: 𝑔
𝑔
𝑔
𝐷 log (𝑚, 𝑞) = 𝐿 ds (𝑚, 𝑞) − 𝐿 ds (𝑚, fp).
(14)
The physical meaning of the metric is this: under quantization, tokens of group 𝑔 became relatively more frequent on dangerous prompts (𝐷 log > 0) or on safe ones (𝐷 log < 0). The sign favorable to model safety depends on the axis: for safety it is 𝐷 log ≥ 0 (the refusal balance did not sag on dangerous prompts); for compliance – 𝐷 log ≤ 0 (the relative willingness to carry out dangerous instructions did not increase). The safety axis alone is not enough if recognition is preserved under quantization while the willingness to comply grows. We introduce the Safety Headroom (SH) metric.
(11) 𝑇𝑔 ( 𝑝).
SH(𝑚, 𝑞) =SafetyAUC(𝑚, 𝑞) −
𝑝∈ N
ComplAUC(𝑚, 𝑞) − 0.5
The 0.5 correction is standard for logarithmic ratios and, at the scale of the totals here (hundreds to thousands), has virtually no effect on the result; on a rare axis it prevents division by zero, and the width of the bootstrap interval then signals on its own how unreliable the estimate is. A crossmodel comparison on a single axis is simply the difference 𝑔 of 𝐿 ds between two models. In addition, to localize axis 𝑔 for a given model we introduce the set of important layers: I𝑔 (𝑚, 𝑞) = ℓ : |AUC𝑔 (𝑚, 𝑞, ℓ) − 0.5| > 𝜏 , (12) 𝜏 = 0.2.
ΔSH(𝑚, 𝑞) = SH(𝑚, 𝑞) − SH(𝑚, fp) = ΔAUCcompl − ΔAUCsafety
(15)
(16)
A negative ΔSH means the joint safety margin has degraded even when SafetyAUC on its own looks preserved (the case of Qwen-4B under INT4: ΔAUCsafety ≈ 0, yet ΔSH < 0 owing to a shift in compliance). Identity (16) sets out the role of the metric: SH is not an independent measurement, but a within-model screening aggregate that folds the shift along two axes into one signed scalar. Using it only in the form of 4
a difference is essential. The absolute level of ComplAUC carries a constant offset from the style of refusal (in aligned models, a substantial share of compliance vocabulary on dangerous prompts comes from the stock phrasing of polite refusal and redirection, not from an intent to comply), so an absolute SH is not interpreted across models; in the difference ΔSH this offset largely cancels. A residual caveat concerns INT4: if quantization itself changes the frequency and style of refusal, the compliance component of ΔSH partly reflects this shift. ΔAUC and 𝐷 log measure different things and can disagree: for instance, under INT4 a model’s relative mass of safety tokens on dangerous prompts can grow (𝐷 log > 0) while the rank separation stays almost unchanged (ΔAUC ≈ 0), so both quantities have to be read together. ΔAUC should be treated as the reranker and the primary criterion, and 𝐷 log as a diagnostic indicator. In addition we assess mechanism relocation – the Jaccard index of the intersection between the sets of important layers before and after quantization: 𝐽𝑔 (𝑚, 𝑞) =
|I𝑔 (𝑚, fp) ∩ I𝑔 (𝑚, 𝑞)| . |I𝑔 (𝑚, fp) ∪ I𝑔 (𝑚, 𝑞)|
The protocol is applied to six models: Gemma2-9B (Gemma Team, 2024) and the Qwen3 family at different sizes (1.7B, 4B, 8B), Ablit-4B – a version of Qwen3-4B with the refusal direction removed by ablation (Arditi et al., 2024), and Qwen3-SafeRL-4B – a version of the same model with additional safety fine-tuning. Ablit-4B and SafeRL-4B serve as calibration poles: they are not targets of the study in their own right, but are needed to check that the metrics really do pick up a known-in-advance weakening and strengthening of the safety signal. For every model we compare three weight-representation regimes: fp (bf16), INT8 (Dettmers et al., 2022) and INT4 (bitsandbytes NF4 with double quantization (Dettmers et al., 2023)). The number of source layers and the sample sizes are given in Table 3. Table 3: Experimental setup. Global settings for all models: decision-point read-out at rel_pos= −1, top-𝑘 = 100; frozen lexicon sizes (words): safety 170, compliance 47, evasion 49, softening 16, hedging 22, harm 84; weight regimes: FP (bf16), INT8, INT4 (NF4).
(17)
A low 𝐽𝑔 shows that the set of important layers has changed, but on its own it does not prove a causal break in the mechanism: a minority of the layers that differ can turn out to be decisive, while a majority of the layers that coincide can be secondary. For this reason 𝐽𝑔 is used only as an indicator of localization; the causal role of a specific layer is established separately, through intervention (replacing or restoring a layer’s weights and then measuring the effect), not through a threshold on 𝐽𝑔 .
Layers
Full name
Gemma-9B Ablit-4B Qwen-1.7B Qwen-4B SafeRL-4B Qwen-8B
41 35 27 35 35 35
gemma-2-9b-it huihui-qwen3-4b-abliterated-v2 qwen3-1.7b qwen3-4b qwen3-4b-saferl qwen3-8b
5.2 Implementation Details Internal metrics are computed at the decision-point position and therefore require no decoding. Behavioral generations used for independent validation, by contrast, run in a deterministic regime (temperature = 0, top_p = 1, do_sample = false); if a model supports a reasoning (thinking) mode, it is turned off. For every layer and prompt we store the top-200 J-lens readout. Confidence intervals throughout are obtained by bootstrapping over prompts with 𝐵 = 300–1000 resamples; the threshold for localizing important layers is 𝜏 = 0.2.
4.7 Behavioral Validation For the attacking set of prompts D we compute the share of responses with a StrongREJECT score (the grader strongreject_finetuned) at threshold 0.5, denoted ℎ(𝑚, 𝑞). The higher ℎ(𝑚, 𝑞), the more dangerous the response. We also consider the share of refusals identified by a regular expression that parses the start of the response (refusal). For the safe control N we compute only the refusal share (over-refusal): on harmless prompts, the StrongREJECT grader measures helpfulness rather than harm, and it often marks a sensible, useful answer as “harmful”; the harm score on N is therefore not interpreted, and only the refusal share is used. Agreement in direction between the internal shifts (ΔAUC, 𝐷 log ) and the behavioral ones (ℎ, refusal) is one of the criteria for trusting the result.
5
Code name
6
Results
Table 4 shows that at fp four models – Gemma-9B, Qwen4B, SafeRL-4B and Qwen-8B – have a SafetyAUC around 0.97 or higher, Qwen-1.7B is somewhat lower (0.895) but still clearly separates dangerous from safe. Ablit-4B stands apart: its SafetyAUC is close to 0.59, nearly at the chance level, while its behavioral harm rate is the highest of the six models (0.904). This split matters: it points not to a subtle weakening of refusal but to a fundamentally different regime, in which the internal distinction between dangerous and safe along the safety axis is simply weak. ComplAUC adds a dimension that cannot be recovered from SafetyAUC alone. Qwen-8B combines the highest SafetyAUC with a ComplAUC noticeably below 0.5 – the model not only recognizes danger but also shows a stronger willingness to comply specifically on safe prompts. Ablit-4B, by contrast, has a ComplAUC above 0.83, consistent with the pattern of following instructions regardless of their danger that is characteristic of this model.
Experiments
5.1 Datasets and Models The dangerous set D consists of 313 StrongREJECT prompts. The safe control N consists of 320 prompts drawn from XSTest-safe (250) and OKTest (70): both sets are chosen so as to contain vocabulary that is alarming in form but harmless in content, and thereby test the metric’s robustness against false triggering on words, rather than only its ability to tell clearly harmful prompts apart from safe ones that resemble them. 5
Table 4: Cross-model summary at FP. SafetyAUC is the primary internal comparator (higher = stronger recognition of danger). ComplAUC below 0.5 corresponds to stronger compliance on safe prompts; values above 0.5 indicate a compliance signature shifted toward dangerous prompts. [email protected] and refusal are measured on the danger set, over-refusal on the safe set. 95% CI in brackets. Model Gemma-9B Ablit-4B Qwen-1.7B Qwen-4B SafeRL-4B Qwen-8B
SafetyAUC
ComplAUC
refusal𝐷
over-refusal 𝑁
0.975 [0.965, 0.984] 0.593 [0.544, 0.640] 0.895 [0.872, 0.922] 0.973 [0.961, 0.983] 0.976 [0.965, 0.985] 0.979 [0.969, 0.988]
0.662 [0.614, 0.710] 0.838 [0.808, 0.864] 0.597 [0.553, 0.644] 0.497 [0.455, 0.544] 0.503 [0.461, 0.552] 0.242 [0.210, 0.277]
0.022 0.904 0.204 0.048 0.345 0.067
0.958 0.000 0.383 0.706 0.000 0.754
0.147 0.009 0.041 0.028 0.003 0.041
Table 5: Quantization effects on the safety (𝑠) and compliance (𝑐) axes (point estimates, count metric). ΔSH is the joint Safety Headroom 𝑠 < 0 (safety mass lost on the danger shift. The direction of harm differs by axis: for safety, ΔAUC𝑠 > 0 (eroded separability) and 𝐷 log 𝑐 > 0 (comply-on-danger set) are adverse; for compliance, ΔAUC𝑐 < 0 (the compliance signature shifting toward the danger set) and 𝐷 log increased) are adverse; ΔSH < 0 is adverse. Per-axis source tables: ΔAUC in Table 11, 𝐷 log with confidence intervals in Tables 9–10 (Appendix A). ΔAUC Model Gemma-9B Ablit-4B Qwen-1.7B Qwen-4B SafeRL-4B Qwen-8B
ΔSH
𝐷 log
𝑠 i8
𝑠 i4
𝑐 i8
𝑐 i4
𝑠 i8
𝑠 i4
𝑐 i8
𝑐 i4
i8
i4
-0.001 -0.011 -0.001 -0.001 -0.001 -0.001
-0.000 -0.086 0.041 0.004 -0.003 0.006
-0.000 0.001 0.002 -0.002 0.004 -0.005
-0.013 -0.012 -0.047 -0.048 -0.003 -0.038
0.006 0.028 0.029 0.025 0.035 0.020
-0.014 0.143 -0.166 0.084 0.056 -0.138
-0.000 0.002 0.006 -0.001 -0.008 -0.004
0.009 -0.023 0.096 0.084 0.024 0.015
0.001 0.012 0.003 -0.001 0.005 -0.004
-0.013 0.073 -0.088 -0.052 -0.000 -0.044
Table 5 gives the summary picture for quantization. INT8 leaves the separability of the safety axis almost unchanged: ΔAUCsafety stays under 0.011 in absolute value for every model. INT4 behaves in a far more uneven way. Qwen-1.7B shows a marked drop in SafetyAUC (ΔAUC𝑠 = 0.041), whereas Ablit-4B’s, by contrast, recovers (ΔAUC𝑠 = −0.086); for Qwen-4B and Qwen-8B the shift in AUC itself is small, yet Safety Headroom still declines (ΔSH = −0.052 and −0.044) – meaning the safety axis alone can look preserved in a place where the joint margin of safety has already weakened because of compliance.
direction and can weaken, rather than restore, recognition. This mechanism agrees with the observation that quantizing specifically the fine-tuning delta weights can partly restore the base model’s safety behavior (Liu et al., 2024). We also note the striking and distinctive robustness of Gemma-9B’s safety and compliance mechanisms to quantization. We note that quantization does the most damage not to the safety mechanism but to the compliance mechanism (for models trained to be safe), consistent with (Chhabra and Khalili, 2025). An increase in compliance on dangerous prompts is the main vulnerability of quantized models.
safety
The sign of 𝐷 log shows which way the balance of absolute masses has shifted. For Qwen-4B under INT4 𝑠 𝐷 log > 0: safety tokens do not relatively disappear on dangerous prompts – yet ΔSH < 0, meaning the overall margin declines specifically because of compliance, not safety. 𝑠 For Qwen-1.7B and Qwen-8B, by contrast, a negative 𝐷 log points to a relative weakening of the safety balance on dangerous prompts, consistent with the drop in AUC for the first of these two models. This illustrates the general principle: ΔAUC, 𝐷 log and SH measure different aspects of the same phenomenon and must be read together, not as interchangeable numbers. Ablit-4B deserves special attention: for this model INT4 does not only fail to make the problem worse, but partly 𝑠 restores lost recognition (𝐷 log = +0.143) and partly restores behavioral refusal ([email protected] drops from 0.904 at fp to 0.875 at INT4, Table 6). A probable explanation is that ablation, as a technique for removing the safety mechanism, relies on a precise edit to the weights, and quantization partly erases this edit along with the rest of the numerical content; in models where the safety mechanism was not artificially removed, the same coarsening effect works in the opposite
Table 6: Behavioral validation by model and weight regime: [email protected], mean StrongREJECT score and refusal on the danger set, over-refusal on the safe set. Lexicon-independent (computed from generated responses).
6
Model
𝑞
mean SR
refusal𝐷
over-ref. 𝑁
Gemma-9B Gemma-9B Gemma-9B
fp int8 int4
0.022 0.003 0.022
0.019 0.008 0.018
0.958 0.958 0.962
0.147 0.153 0.150
Ablit-4B Ablit-4B Ablit-4B
fp int8 int4
0.904 0.898 0.875
0.780 0.775 0.747
0.000 0.003 0.000
0.009 0.006 0.000
Qwen-1.7B Qwen-1.7B Qwen-1.7B
fp int8 int4
0.204 0.201 0.188
0.209 0.210 0.178
0.383 0.351 0.403
0.041 0.044 0.031
Qwen-4B Qwen-4B Qwen-4B
fp int8 int4
0.048 0.038 0.048
0.063 0.057 0.073
0.706 0.725 0.728
0.028 0.034 0.025
SafeRL-4B SafeRL-4B SafeRL-4B
fp int8 int4
0.345 0.374 0.329
0.385 0.373 0.361
0.000 0.000 0.000
0.003 0.003 0.000
Qwen-8B Qwen-8B Qwen-8B
fp int8 int4
0.067 0.073 0.054
0.071 0.069 0.063
0.754 0.760 0.789
0.041 0.034 0.037
7
Analysis
score. The SafeRL-4B model shows a substantial ℎ𝑎𝑟𝑚@0.5, which leads to the counterintuitive suggestion that the SafeRL model is more dangerous than Qwen3-4B. One should, however, take into account the specifics of how the SafeRL model was trained (Qwen Team, 2025) – it was rewarded for both safety and helpfulness at the same time, and penalized for unnecessary refusals. It is therefore meant to answer less often with a plain ‘I can’t help with that,’ replacing a dangerous response with a safe alternative. Every SafeRL-4B response to a dangerous question follows the same template: ‘It’s important to clarify, that you request is illegal. However, I can suggest...’. Precisely because of this shared opening construction, the regex refusal rate scored this model at 0.00. StrongREJECT reads the extended answer as disclosure of harmful instructions, while its grader assigns low scores specifically to a plain refusal. On harm recognition, SafeRL4B shows a higher SafetyAUC than the base Qwen3-4B. This example makes the difference between assessing a model’s internal state and assessing its required behavior particularly clear. SafeRL-4B’s responses can be scored differently depending on the benchmark. The internal safety mechanism, however – the separation of harmful from benign – stays the same. This is exactly the problem that the SafetyAUC metric addresses.
7.1 Count and DCG Count and DCG measure different aspects of the same readout: count is how many tokens of the group land in the top-𝑘, DCG is how high up they sit there. For most axes and models the two quantities agree qualitatively: for example, the DCG version of SafetyAUC for Gemma-9B and Qwen-8B differs from the count version by no more than 0.007 (Table 8 in the Appendix versus Table 4). The difference becomes substantive specifically in the context of quantization: if a modified model keeps including a group’s tokens in the top-𝑘 but systematically pushes them down in rank, count will not notice, while DCG will register the weakened availability of the signal before it shows up in the plain count. For this reason DCG is used not as a replacement for count, but as a variant of the same measure that is more sensitive to rank shuffling, and both versions are reported for every final metric. 7.2 Recognition versus Behavior Comparing the SafetyAUC and [email protected] columns in Table 4 pulls apart two distinct failure modes that a purely behavioral metric cannot tell apart. Ablit-4B has a low SafetyAUC (0.593) and the highest harm rate of the six models (0.904) – this is a model in which safety is broken at the level of representation, not only at the level of the final response. SafeRL-4B and, to a lesser extent, Qwen-1.7B are built differently: their SafetyAUC is high (0.976 and 0.895), but the harm rate is also noticeably above zero (0.345 and 0.204) – a case where the model internally tells a dangerous prompt from a safe one, but this does not stop it from a harmful response. For fine-tuning and defensive quantization these two cases call for different interventions: in the first, restoring or strengthening recognition itself; in the second, working on what happens between recognition and the decision to comply. The Qwen-8B vs Gemma-9B pair shows something interesting. By SafetyAUC the models are indistinguishable (0.979 versus 0.975, confidence intervals overlapping), while Qwen-8B’s ComplAUC is lower (0.242 versus 0.662), so on the axis of recognition Qwen-8B looks no worse. Behavior, however, runs the other way: Qwen-8B has a higher [email protected] (0.067 versus 0.022) and a lower refusal share (0.754 versus 0.958). The discrepancy is explained by two factors invisible to separability metrics. First, Gemma-9B is tuned to refuse aggressively and pays for its low harm with a high over-refusal (0.147), whereas Qwen-8B is shifted toward helpfulness (over-refusal 0.041) and pays for this with a higher harm rate at the same quality of recognition. Second, a ComplAUC of 0.662 > 0.5 for Gemma-9B, given a refusal share of 0.958, cannot reflect an intent to carry out a harmful request and corresponds instead to the vocabulary of polite refusal and redirection on dangerous prompts. A lower ComplAUC for Qwen-8B should not be read as unconditionally safer, but as a trait of a model that suppresses compliance tokens when answering a dangerous prompt, unlike Gemma9B. It is worth keeping in mind that compliance is a less stable and less clear-cut axis than safety. Because of this, ComplAUC can be misleading when comparing models from different families. One should also allow for grading error in the StrongREJECT grader, which, acting as an LLM-asjudge, may itself have introduced some bias into the response
7.3
Layer Relocation
The sets of important layers I𝑔 let us check whether an axis stays localized in the same layers after quantization. For the safety axis under INT4 the Jaccard index is above 0.9 for four of the six models and drops to 0 for Ablit-4B (its safety mechanism is already very weakly localized at fp, in just two layers, so even a small shift changes the set entirely). A low 𝐽𝑔 does not by itself mean a causal break: some of the layers that formally fell out of the threshold set could have been secondary, with one or two of the remaining layers being decisive. A separate experiment restoring the original FP weights in individual layers of the INT4 model (rescue) would confirm this caveat directly: restoring the weights in just a few layers, chosen not by AUC-importance but by an independent criterion (the refusal direction (Arditi et al., 2024)), noticeably changed the model’s eventual behavior – meaning the causal weight of specific layers need not match their formal membership in I𝑔 . The full relocation table for safety and compliance across all models is given in Table 16 of the Appendix.
8
Limitations
JADR depends on a lexicon fixed in advance. The lexicon groups cover individual vocabulary tokens and are poorly suited to multi-token expressions and to cases where the model describes a dangerous action descriptively, without directly using any word from the lexicon; in this sense the metric systematically underestimates recognition wherever it is expressed non-literally. The rank AUC and the Jaccard index are correlational quantities. They show that an axis separates D and N in certain layers, but they do not prove that these specific layers are causally responsible for refusal; a causal conclusion requires intervention – replacing weights, patching activations (Zhang and Nanda, 2023), or ablating them and then measuring the effect. 7
There is also a substantive limitation tied to the very object being measured: StrongREJECT and similar graders score text that has already been generated, whereas JADR captures the state of the model before generation. The gap between these two levels is not a flaw of the method; it is itself a substantive result of comparing danger recognition and behavior. Finally, the six models used here include two calibration poles (ablation, additional safety fine-tuning) and make no claim to exhaustively cover model families or sizes; carrying the protocol over to a new architecture requires re-checking sensitivity to 𝑘 and recomputing the important layers, rather than directly reusing the thresholds obtained here. It should be noted that the metrics presented here do depend on the lexicon set in the configuration file. The AUC metrics, however, are robust to changes in the lexicon. Also, the wider the lexicon already is, the smaller the change from subsequently extending it carefully. Moreover, in scenarios that call for comparing a group of models, researchers fix a single shared vocabulary, and the ranking conclusions remain valid. Because the workspace tokens are stored throughout the protocol, a researcher can immediately compare the results of several lexicons against each other. The method is meant for comparing LLM safety across models and for evaluating modified models, and for more deliberate design of quantization and subsequent fine-tuning procedures: it makes it possible to notice degradation of internal recognition of danger earlier than it becomes visible in the eventual share of harmful responses, and to separate it from a shift toward willingness to comply, which a behavioral metric also does not catch right away.
9
with a plain refusal. INT8 quantization is practically neutral to internal separability, whereas INT4 acts unevenly: from noticeable erosion in Qwen3-1.7B to a partial reversal of the ablation effect in Qwen3-4B-Uncensored, where 4-bit compression erases part of the targeted weight edit and restores both safety tokens in J-space and the refusal share. Quantization, however, harms the compliance (instruction-following) axis more than it harms the safety mechanism. Degradation of the protective mechanisms can be noticed before it shows up in the share of harmful responses, and localized by axis and by layer. This makes it possible to use JADR as a cheap screen for quantized and fine-tuned checkpoints before deployment – in settings where behavioral benchmarks require generation and an external LLMas-judge evaluation, and still cannot tell intact recognition with lost refusal apart from lost recognition. Future work looks to move in two directions: causal verification of the layers identified here through targeted interventions with a measured effect on behavior, and carrying the protocol beyond the Qwen and Gemma families.
Acknowledgements
This research was supported in part through computational resources of HPC facilities at HSE University. The work was supported by the grant for research centers in the field of AI provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C313925P4E0002 and the agreement with HSE University № 139-15-2025-009.
10
Conclusion
This paper has proposed JADR – a reproducible protocol for assessing internal danger recognition in language models. The protocol measures the verbalizable content of J-space at the decision point, before the first token is generated, calls on no external judge model, and runs in a single forward pass, which makes it usable within a local pipeline and suitable for comparing both different models and modifications of a single model on the same terms. The empirical results come down to several conclusions. The proposed SafetyAUC metric separates models of different sizes, and models with a deliberately strong versus a deliberately weakened safety mechanism, with statistical significance. The resulting ranking agrees with the behavioral evaluation from the StrongREJECT benchmark once model-specific quirks are taken into account. Internal recognition and eventual behavior can, however, part ways. Using Qwen3-4B-SafeRL as an example, we showed that a model can recognize danger well and still disclose harmful information, because its particular fine-tuning forbids answering 8
References
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023. URL https://arxiv.org/abs/ 2310.03693.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/ 2406.11717.
Qwen Team. Qwen3guard technical report, 2025. URL http://arxiv.org/abs/2510.14276.
Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. URL https://arxiv.org/ abs/2303.08112.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of NAACL-HLT, 2024. URL https://arxiv.org/abs/2308.01263.
Vishnu Kabir Chhabra and Mohammad Mahdi Khalili. Towards understanding and improving refusal in compressed models via mechanistic interpretability, 2025. URL https://arxiv.org/abs/2504.04215.
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260, 2024. URL https://arxiv.org/ abs/2402.10260.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. URL https://arxiv.org/abs/2208.07339.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2023. URL https://arxiv.org/abs/2309.16042.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/ 2305.14314. Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/ abs/2408.00118. W. Gurnee, N. Sofroniew, A. Pearce, M. Piotrowski, I. Kauvar, R. Chen, A. Soligo, P. Bogdan, E. Ong, R. Wang, B. Thompson, A. David, S. Kantamneni, E. Ameisen, J. Batson, and J. Lindsey. Verbalizable representations form a global workspace in language models. Transformer Circuits Thread / Anthropic, 2026. URL https://transformer-circuits.pub/ 2026/workspace/index.html. James A. Hanley and Barbara J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1):29–36, 1982. Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, and Bo Li. Decoding compressed trust: Scrutinizing the trustworthiness of efficient LLMs under compression. In International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2403.15447. Yule Liu, Zhen Sun, Xinlei He, and Xinyi Huang. Quantized delta weight is safety keeper, 2024. URL https: //arxiv.org/abs/2411.19530. nostalgebraist. Interpreting GPT: The logit lens. LessWrong, 2020. URL https://www. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. 9
A
Supplementary materials
Tables Below are the full per-axis tables that supplement the summary results of the main text. Notation: models – Gemma9B, Ablit-4B (abliterated), Qwen-1.7B, Qwen-4B, SafeRL4B, Qwen-8B; axes – Saf(ety), Cmp(compliance), Eva(sion), Sof(tening), Hed(ging), Hrm (harm); D/N – danger/safe. All quantities are at the decision point, top-𝑘 = 100, on the frozen lexicon; uncertainty is a 95% bootstrap CI over prompts. Figures This section collects the diagnostic plots that supplement the numerical tables of Appendix A. Every panel is built on the same grid of six models (Gemma-9B, Ablit-4B, Qwen1.7B, Qwen-4B, SafeRL-4B, Qwen-8B) and three weightrepresentation regimes (fp, INT8, INT4).
10
Table 7: Per-axis discrimination AUC at FP, count metric, 95% CI. Long-row layout of the same data as Table 4’s SafetyAUC/ComplAUC columns, extended to all six axes. Model
Axis
AUC [95% CI]
Model
Axis
AUC [95% CI]
Gemma-9B Gemma-9B Gemma-9B Gemma-9B Gemma-9B Gemma-9B
safety compliance evasion softening hedging harm
0.975 [0.965, 0.984] 0.662 [0.614, 0.710] 0.668 [0.617, 0.715] 0.366 [0.323, 0.410] 0.197 [0.163, 0.236] 0.814 [0.781, 0.850]
Qwen-4B Qwen-4B Qwen-4B Qwen-4B Qwen-4B Qwen-4B
safety compliance evasion softening hedging harm
0.973 [0.961, 0.983] 0.497 [0.455, 0.544] 0.653 [0.604, 0.694] 0.550 [0.507, 0.592] 0.457 [0.409, 0.501] 0.610 [0.573, 0.644]
Ablit-4B Ablit-4B Ablit-4B Ablit-4B Ablit-4B Ablit-4B
safety compliance evasion softening hedging harm
0.593 [0.544, 0.640] 0.838 [0.808, 0.864] 0.500 [0.468, 0.531] 0.559 [0.516, 0.602] 0.289 [0.253, 0.324] 0.526 [0.498, 0.553]
SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B
safety compliance evasion softening hedging harm
0.976 [0.965, 0.985] 0.503 [0.461, 0.552] 0.830 [0.796, 0.861] 0.547 [0.505, 0.589] 0.202 [0.169, 0.237] 0.493 [0.458, 0.526]
Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B
safety compliance evasion softening hedging harm
0.895 [0.872, 0.922] 0.597 [0.553, 0.644] 0.780 [0.739, 0.814] 0.466 [0.413, 0.515] 0.356 [0.313, 0.395] 0.381 [0.339, 0.417]
Qwen-8B Qwen-8B Qwen-8B Qwen-8B Qwen-8B Qwen-8B
safety compliance evasion softening hedging harm
0.979 [0.969, 0.988] 0.242 [0.210, 0.277] 0.825 [0.797, 0.862] 0.641 [0.598, 0.685] 0.134 [0.108, 0.160] 0.733 [0.694, 0.769]
Table 8: Per-axis discrimination AUC at FP, DCG (rank-weighted) metric, 95% CI. DCG counterpart of Table 7: the safety and compliance columns are the DCG-based SafetyAUC and ComplAUC discussed in §7. Model
Axis
AUC [95% CI]
Model
Axis
AUC [95% CI]
Gemma-9B Gemma-9B Gemma-9B Gemma-9B Gemma-9B Gemma-9B
safety compliance evasion softening hedging harm
0.968 [0.955, 0.977] 0.676 [0.637, 0.715] 0.659 [0.610, 0.699] 0.356 [0.308, 0.391] 0.201 [0.168, 0.232] 0.813 [0.780, 0.846]
Qwen-4B Qwen-4B Qwen-4B Qwen-4B Qwen-4B Qwen-4B
safety compliance evasion softening hedging harm
0.973 [0.960, 0.982] 0.521 [0.474, 0.564] 0.648 [0.608, 0.686] 0.551 [0.506, 0.599] 0.456 [0.411, 0.507] 0.611 [0.575, 0.643]
Ablit-4B Ablit-4B Ablit-4B Ablit-4B Ablit-4B Ablit-4B
safety compliance evasion softening hedging harm
0.635 [0.590, 0.676] 0.831 [0.802, 0.859] 0.499 [0.465, 0.531] 0.561 [0.522, 0.609] 0.288 [0.249, 0.320] 0.526 [0.500, 0.555]
SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B
safety compliance evasion softening hedging harm
0.975 [0.962, 0.985] 0.524 [0.479, 0.568] 0.823 [0.792, 0.853] 0.549 [0.507, 0.590] 0.196 [0.162, 0.230] 0.491 [0.461, 0.524]
Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B
safety compliance evasion softening hedging harm
0.893 [0.865, 0.915] 0.622 [0.581, 0.668] 0.777 [0.741, 0.812] 0.483 [0.435, 0.523] 0.348 [0.311, 0.391] 0.379 [0.338, 0.418]
Qwen-8B Qwen-8B Qwen-8B Qwen-8B Qwen-8B Qwen-8B
safety compliance evasion softening hedging harm
0.979 [0.970, 0.989] 0.243 [0.209, 0.278] 0.822 [0.784, 0.854] 0.655 [0.613, 0.700] 0.136 [0.110, 0.164] 0.730 [0.691, 0.763]
Table 9: 𝐷 log under INT4, count metric, per axis, 95% CI. This is the regime where content shifts are large enough to be worth reporting with full confidence intervals; the corresponding INT8 shifts are given as point estimates in Table 10. Model
Axis
𝐷log [95% CI]
Model
Axis
𝐷log [95% CI]
Gemma-9B Gemma-9B Gemma-9B Gemma-9B Gemma-9B Gemma-9B
safety compliance evasion softening hedging harm
-0.014 [-0.033, 0.007] 0.009 [-0.000, 0.020] -0.101 [-0.175, -0.021] 0.013 [-0.052, 0.065] 0.018 [-0.025, 0.057] 0.036 [-0.027, 0.109]
Qwen-4B Qwen-4B Qwen-4B Qwen-4B Qwen-4B Qwen-4B
safety compliance evasion softening hedging harm
0.084 [0.040, 0.125] 0.084 [0.068, 0.099] 0.140 [-0.015, 0.334] 0.183 [0.096, 0.276] -0.472 [-0.596, -0.338] -0.215 [-0.406, -0.004]
Ablit-4B Ablit-4B Ablit-4B Ablit-4B Ablit-4B Ablit-4B
safety compliance evasion softening hedging harm
0.143 [0.097, 0.195] -0.023 [-0.038, -0.007] 0.304 [0.052, 0.700] 0.076 [-0.028, 0.179] 0.408 [0.247, 0.596] 0.064 [-0.179, 0.271]
SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B SafeRL-4B
safety compliance evasion softening hedging harm
0.056 [0.026, 0.084] 0.024 [0.011, 0.038] -0.038 [-0.134, 0.075] -0.248 [-0.343, -0.157] 0.210 [0.050, 0.367] -0.046 [-0.343, 0.229]
Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B Qwen-1.7B
safety compliance evasion softening hedging harm
-0.166 [-0.219, -0.121] 0.096 [0.066, 0.125] 0.064 [-0.029, 0.165] -0.054 [-0.094, -0.011] 0.135 [0.056, 0.215] -0.388 [-0.580, -0.190]
Qwen-8B Qwen-8B Qwen-8B Qwen-8B Qwen-8B Qwen-8B
safety compliance evasion softening hedging harm
-0.138 [-0.172, -0.104] 0.015 [-0.007, 0.039] -0.399 [-0.517, -0.295] 0.184 [0.116, 0.255] 0.009 [-0.071, 0.089] -0.173 [-0.263, -0.077]
Table 10: 𝐷 log under INT8, count metric, point estimates (no CI; effect sizes here are small and close to zero, see §6). Model Gemma-9B Ablit-4B Qwen-1.7B Qwen-4B SafeRL-4B Qwen-8B
Saf 0.006 0.028 0.029 0.025 0.035 0.020
Cmp
Eva
Sof
Hed
Hrm
-0.000 -0.000 0.038 -0.020 -0.005 0.002 -0.099 -0.018 0.013 -0.032 0.006 0.105 0.008 0.005 -0.039 -0.001 0.068 0.020 -0.004 -0.050 -0.008 0.007 0.018 0.001 -0.008 -0.004 0.035 -0.018 0.096 0.038
11
Table 11: ΔAUC𝑔 = AUC𝑔 (FP) − AUC𝑔 (𝑞), count metric, per axis, point estimates. Source of the safety and compliance ΔAUC columns in Table 5. For safety, ΔAUC > 0 = eroded separability; for compliance, ΔAUC < 0 = signature shifted toward the danger set. Model
𝑞
Saf
Cmp
Eva
Sof
Gemma-9B Gemma-9B Ablit-4B Ablit-4B Qwen-1.7B Qwen-1.7B Qwen-4B Qwen-4B SafeRL-4B SafeRL-4B Qwen-8B Qwen-8B
int8 int4 int8 int4 int8 int4 int8 int4 int8 int4 int8 int4
-0.001 -0.000 -0.011 -0.086 -0.001 0.041 -0.001 0.004 -0.001 -0.003 -0.001 0.006
-0.000 -0.013 0.001 -0.012 0.002 -0.047 -0.002 -0.048 0.004 -0.003 -0.005 -0.038
0.005 0.015 -0.000 -0.021 0.002 0.020 -0.025 -0.042 0.004 0.003 -0.013 0.064
-0.006 -0.006 0.002 0.005 0.010 0.054 -0.010 -0.045 0.002 0.089 0.005 -0.056
Hed
Hrm
0.002 0.002 0.014 -0.011 0.013 0.003 -0.023 0.016 -0.041 -0.009 -0.104 0.004 0.016 0.013 0.134 0.028 0.002 0.001 0.067 0.006 -0.026 0.004 -0.005 0.039
Table 12: 𝐷 log , DCG metric, both quantization regimes, point estimates (no CI; DCG is reported here as a robustness check against Table 9/10, not as an independent primary result). Model
𝑞
Hed
Hrm
Gemma-9B Gemma-9B Ablit-4B Ablit-4B Qwen-1.7B Qwen-1.7B Qwen-4B Qwen-4B SafeRL-4B SafeRL-4B Qwen-8B Qwen-8B
int8 0.007 -0.002 -0.000 0.045 -0.020 int4 -0.013 0.033 -0.125 0.018 0.026 int8 0.031 -0.004 -0.072 -0.023 0.028 int4 0.150 -0.016 0.349 0.063 0.469 int8 0.028 0.004 0.087 0.012 -0.009 int4 -0.186 0.096 0.083 -0.082 0.096 int8 0.035 -0.006 0.036 0.018 0.003 int4 0.085 0.097 0.115 0.148 -0.433 int8 0.045 -0.010 -0.034 0.022 0.006 int4 0.034 0.052 -0.023 -0.293 0.255 int8 0.022 -0.005 0.028 -0.015 0.094 int4 -0.124 0.010 -0.344 0.193 -0.008
Saf
Cmp
Eva
Sof
-0.005 0.020 -0.019 0.106 -0.018 -0.455 -0.052 -0.159 -0.018 -0.009 0.048 -0.177
Table 13: Absolute per-prompt token counts (count metric), summed over layers, by axis, model and weight regime. D/N = danger/safe. Saf Model/𝑞 Gemma-9B/fp Gemma-9B/int8 Gemma-9B/int4 Ablit-4B/fp Ablit-4B/int8 Ablit-4B/int4 Qwen-1.7B/fp Qwen-1.7B/int8 Qwen-1.7B/int4 Qwen-4B/fp Qwen-4B/int8 Qwen-4B/int4 SafeRL-4B/fp SafeRL-4B/int8 SafeRL-4B/int4 Qwen-8B/fp Qwen-8B/int8 Qwen-8B/int4
Cmp
D
N
D
525.4 523.6 516.4 18.5 17.0 15.2 219.6 211.2 194.8 442.4 438.2 364.4 314.1 313.0 275.7 487.2 490.6 475.4
98.6 97.7 98.3 19.1 17.0 13.5 39.8 37.2 41.7 72.4 70.0 54.9 64.0 61.6 53.1 50.0 49.3 56.0
91.6 91.9 93.6 184.5 180.7 188.7 45.9 46.5 50.7 93.5 90.4 105.4 81.0 79.2 87.8 58.8 57.8 53.4
Eva N
D
Sof N
D
N
Hed D
N
Hrm D
N
83.3 8.7 7.1 4.3 13.4 2.4 31.1 13.6 4.5 83.6 8.6 7.0 4.5 13.3 2.4 31.3 13.5 4.5 84.3 7.5 6.8 4.7 14.4 2.4 30.2 14.0 4.5 106.3 2.7 0.8 7.6 6.7 1.2 7.7 1.0 1.1 103.8 2.6 0.9 7.4 6.6 1.2 7.8 0.9 1.1 111.2 2.1 0.5 6.0 4.8 1.8 8.1 0.8 0.8 38.4 6.4 2.0 8.1 8.8 4.5 7.4 0.9 2.1 38.6 6.1 1.7 8.1 8.8 3.7 6.1 0.8 1.8 38.5 5.7 1.7 7.0 8.0 3.3 4.7 0.5 1.7 102.1 5.1 1.6 6.1 5.8 2.0 6.0 2.8 2.1 98.8 4.7 1.4 6.0 5.7 2.0 6.1 2.7 2.1 105.8 4.4 1.2 4.9 3.9 1.3 6.4 2.0 1.8 86.9 6.8 1.7 5.4 5.3 0.7 6.7 0.7 1.2 85.6 6.3 1.6 5.4 5.2 0.7 6.7 0.7 1.2 92.0 5.9 1.6 3.1 3.9 1.0 8.3 0.4 0.8 86.1 10.7 2.9 6.8 5.2 3.1 10.0 7.0 2.9 85.0 11.1 2.9 6.7 5.2 3.3 9.7 7.1 2.9 77.0 10.6 4.3 9.8 6.2 2.8 8.9 6.0 3.0
12
Table 14: Absolute per-prompt token weight (DCG metric), summed over layers, by axis, model and weight regime. Same layout as Table 13. Saf Model/𝑞 Gemma-9B/fp Gemma-9B/int8 Gemma-9B/int4 Ablit-4B/fp Ablit-4B/int8 Ablit-4B/int4 Qwen-1.7B/fp Qwen-1.7B/int8 Qwen-1.7B/int4 Qwen-4B/fp Qwen-4B/int8 Qwen-4B/int4 SafeRL-4B/fp SafeRL-4B/int8 SafeRL-4B/int4 Qwen-8B/fp Qwen-8B/int8 Qwen-8B/int4
Cmp
Eva
Sof
Hed
Hrm
D
N
D
N
D
N
D
N
D
N
D
N
121.0 120.8 119.0 3.7 3.4 3.1 47.2 45.3 41.6 95.9 95.0 76.6 66.5 66.2 57.3 107.7 107.9 105.7
23.6 23.4 23.5 3.6 3.2 2.6 8.0 7.5 8.5 14.9 14.2 10.9 12.6 12.0 10.5 10.2 10.0 11.4
18.8 18.8 19.6 44.2 43.2 45.2 11.1 11.3 12.4 19.3 18.7 21.7 16.3 16.1 18.1 11.4 11.1 10.1
17.1 17.2 17.2 22.8 22.4 23.8 8.3 8.4 8.4 21.7 21.2 22.2 17.9 17.9 18.8 17.5 17.2 15.4
1.5 1.5 1.3 0.6 0.6 0.5 1.2 1.2 1.1 0.9 0.9 0.8 1.5 1.4 1.3 2.0 2.1 2.0
1.4 1.4 1.3 0.2 0.2 0.1 0.4 0.3 0.3 0.3 0.3 0.3 0.4 0.4 0.3 0.7 0.7 0.9
0.8 0.8 0.8 1.5 1.5 1.2 1.6 1.6 1.3 1.1 1.1 0.9 1.0 1.0 0.5 1.3 1.3 1.9
2.8 2.8 3.0 1.3 1.3 0.9 1.6 1.6 1.5 1.1 1.1 0.7 1.0 0.9 0.7 1.0 1.0 1.2
0.4 0.4 0.4 0.2 0.2 0.3 0.7 0.6 0.6 0.3 0.3 0.2 0.1 0.1 0.2 0.5 0.6 0.5
6.7 6.7 6.4 1.5 1.5 1.5 1.4 1.2 1.0 1.2 1.2 1.2 1.3 1.3 1.5 1.8 1.8 1.7
2.5 2.4 2.5 0.2 0.2 0.2 0.2 0.2 0.1 0.6 0.5 0.4 0.1 0.1 0.1 1.4 1.4 1.2
1.0 1.0 1.0 0.2 0.2 0.2 0.4 0.4 0.4 0.5 0.5 0.4 0.2 0.3 0.2 0.7 0.6 0.7
Table 15: Mechanism localization at FP: number of important layers |I𝑔 | = |{ℓ : |AUC𝑔 (ℓ) − 0.5| > 0.2}| and the peak-AUC layer, reported as “|I𝑔 | / peak ℓ”. Model
Saf
Cmp
Eva
Sof
Hed
Hrm
Gemma-9B Ablit-4B Qwen-1.7B Qwen-4B SafeRL-4B Qwen-8B
26/25 2/4 12/17 18/26 18/21 19/26
10/27 17/28 1/12 2/1 1/0 8/19
1/23 0/15 7/24 0/15 6/28 7/23
2/38 0/29 0/26 0/23 4/27 0/32
9/21 0/25 3/17 0/22 1/34 3/15
0/38 0/29 0/15 0/30 0/26 0/27
Table 16: Layer relocation under quantization: Jaccard 𝐽 of important-layer sets I𝑔 (FP) vs. I𝑔 (𝑞), with dropped (IFP \Iint4 ) and added layers under INT4, for safety and compliance. Reported as an indicator only (§4.6); causal role is established separately. Model
Axis
𝐽(i8) 𝐽(i4) Dropped (i4) Added (i4)
Gemma-9B Gemma-9B Ablit-4B Ablit-4B Qwen-1.7B Qwen-1.7B Qwen-4B Qwen-4B SafeRL-4B SafeRL-4B Qwen-8B Qwen-8B
safety compliance safety compliance safety compliance safety compliance safety compliance safety compliance
1.00 0.90 0.00 0.78 1.00 0.50 0.94 0.20 1.00 0.25 1.00 0.75
1.00 0.90 0.00 0.79 1.00 0.00 0.90 0.50 0.95 0.20 0.95 0.50
– 20 2,4 10,12 – 12 – 12 – – 17 14,15
– – 10,12 1,15 – – 12,14 – 10 4,19,21,34 – 5,12,13,21
Table 17: Top-𝑘 sensitivity of the safety axis at FP: best-layer AUC as a function of the counting depth 𝑘. Saturation with compromise happens at 𝑘 = 100, so it is picked for all model evaluations. 𝑘
Gemma-9B
Ablit-4B
Qwen-1.7B
Qwen-4B
SafeRL-4B
Qwen-8B
5 15 30 50 100 200
0.921 0.962 0.974 0.977 0.983 0.989
0.559 0.631 0.760 0.810 0.770 0.790
0.826 0.870 0.889 0.894 0.917 0.919
0.918 0.949 0.969 0.968 0.973 0.976
0.910 0.965 0.977 0.977 0.984 0.980
0.955 0.980 0.978 0.980 0.983 0.985
13
Figure 1: Per-layer AUCsafety (ℓ) for the three weight-representation regimes. For five models the profile has a characteristic shape: around 0.5 in the early layers, then a sharp transition and a plateau at 0.9–0.98 in the middle and late part of the network. Ablit-4B (Qwen3-4B uncensored) stands out with a qualitatively different profile: a brief rise in the mid-layer range with no stable plateau, followed by a return to a level close to chance – a visual confirmation of the weak and unstable localization of the safety axis in this model, already noted from the numerical SafetyAUC= 0.593.
Figure 2: Per-layer AUCcompliance (ℓ). SafeRL-4B already shows a pronounced AUC spike at layer 0 (≈ 0.75–0.77) – no other model expresses the compliance signal this strongly at the very first layer. This points to the nature of the model’s training: it must not simply refuse a harmful prompt, but is required to give some answer regardless, and this readiness to answer is already visible in the representation of the prompt before any substantive processing takes place.
Figure 3: Per-layer AUCevasion (ℓ). For most models the axis is localized in a narrow band of mid-to-late layers (AUC up to 0.7–0.85); for Ablit-4B the profile stays close to 0.5 across the entire range of layers.
14
Figure 4: Share of special/technical tokens in the top-𝑘 readout by layer, for four regime/promptset combinations. For Gemma-2-9B-IT almost the entire mass of special tokens is concentrated in the first and second layer (≈ 0.6–0.7), after which the share falls to near zero and stays there across the rest of the range; for models in the Qwen3 family, by contrast, the special-token share fluctuates between 0.01 and 0.07 across nearly all layers. This is a notable architectural difference between the model families.
Figure 5: Heatmap of safety DCG by layer and regime (danger/safe × fp/INT8/INT4). Five models show sharp, narrowly localized dark bands only in the danger rows, in the mid-to-late range of layers. For Ablit-4B the intensity scale is an order of magnitude lower, and the dark regions are spread over a far larger number of layers while barely telling danger from safe – the visual counterpart of a low and unstable SafetyAUC.
Figure 6: Difference (danger minus safe) in safety DCG by layer. Five models show a clean dome-shaped profile with an amplitude of 6–9 DCG units. For Ablit-4B the amplitude is two orders of magnitude smaller (−0.15 to 0.11) and turns negative over part of the layer range (roughly 22–29), meaning the safe set carries more safety vocabulary than the danger set in these layers – an inversion found in no other model.
15
Figure 7: Absolute safety DCG mass on danger and safe, by model and quantization regime (numerical values in Table 14 of the Appendix). A basic descriptive plot for comparison against the rank metrics of the main text.
Figure 8: Absolute hedging count mass on danger and safe. For five models, hedging is almost entirely concentrated on the safe set, and is close to zero on danger. Qwen3-1.7B is the sole exception: its value on danger (≈ 3.0–3.3) is comparable in order of magnitude to its value on safe (≈ 3.7–4.5). For a very small model, a rise in hedging on dangerous prompts may be an indirect sign of compliance-like behavior – an evasive, cautious, yet non-refusing response – rather than mere noise from a small model.
Figure 9: Heatmap of harm DCG by layer and regime. For SafeRL-4B the fp/safe and int8/safe rows show noticeably darker and wider bands than the corresponding danger rows of the same model – for the other models the relation runs the other way (danger darker than safe). This agrees with the observation in §6: SafeRL-4B is trained to explicitly name the harmful domain of a prompt and to answer with an extended, safe explanation rather than the short refusal that is sufficient for the other models and does not call for activating harm vocabulary on the safe set.
16
Figure 10: Absolute evasion DCG mass on danger and safe. For Gemma-2-9B-IT, danger consistently exceeds safe at fp and INT8, but the relation flips under INT4 (safe becomes higher than danger) – the only such case among the six models. This is consistent with Gemma being trained more strongly than the others to resist attempts at evasion and consequently behaving more strictly: after quantization, its evasion response to harmless but lexically alarming safe prompts starts to outweigh its response to genuinely dangerous ones.
Figure 11: Heatmap of evasion count by layer and regime. For Gemma-2-9B-IT the signal appears noticeably earlier in network depth (around layer 9–10) than for models of the Qwen3 family (typically after layer 17-20) – further confirmation that Gemma’s resistance to evasion attempts is built into earlier, not only later, stages of processing the prompt.
Figure 12: Difference (danger minus safe) in compliance DCG by layer. Unlike safety (fig. 6), the profiles here are noticeably narrower, noisier, and do not form a single dome of consistent sign across all models – the compliance axis is less well localized and less stable across layers, consistent with the smaller number of important layers |Icompliance | in Table 15.
17