RISA: Response Inspection and Selective Actions for Refusal Calibration in Large Language Models

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

1

RISA: Response Inspection and Selective Actions for Refusal Calibration in Large Language Models

arXiv:2609.00790v1 [cs.CR] 1 Sep 2026

Wenhan Chang, Tianqing Zhu, Member, IEEE, Ping Xiong*, Shiyi Liao, Wanlei Zhou, Life Fellow, IEEE

Abstract—Reliable refusal behavior requires Large Language Models (LLMs) to reject harmful prompts with only answering benign ones. Incorrect refusal behavior can either expose users to harmful responses or prevent users from obtaining useful answers. Training-time alignment improves refusal behavior by updating model parameters with safety data, but requires additional computation and training. In contrast, inferencetime alignment aims to modify LLM behavior during inference without updating the underlying model parameters. Existing inference-time methods mainly rely on in-context safety prompting, activation steering, or decoding control. However, most of them intervene without first determining whether the initial response is already appropriate, potentially altering a correct refusal or a useful answer. Effective selective intervention therefore requires identifying prompt intent beyond sensitive keywords, covering semantic variations that fixed rules may miss, and adapting the verifier to different base models. To address these challenges, we propose Response Inspection and Selective Actions (RISA), an inference-time framework that inspects the initial response and selectively corrects refusal errors without updating the base model. RISA first uses fixed contextual rules to assign refusal scores to clear cases. For unmatched cases, it derives a refusal score from the final-layer prompt hidden state using a calibrated linear probe. To adapt to different base models, RISA separately calibrates the probe score, representation-support boundary, and action thresholds. At runtime, RISA combines the prompt score with the initial refusal status and applies an action policy to intervene only when necessary. Experimental results demonstrate that RISA improves refusal reliability while largely preserving model utility, offering a practical solution for response-aware refusal calibration in LLMs. Index Terms—Large Language Models, AI Safety, Refusal Behavior, Over-Refusal, Under-Refusal, Runtime Verification.

I. I NTRODUCTION Large language models (LLMs) have played an important role across a wide range of real-world scenarios [1], [2]. Reliable refusal behavior requires LLMs to refuse harmful prompts while providing appropriate responses to benign ones [3], [4]. Such refusal behavior is a key safety requirement of LLM. However, prompts with harmless intent may still contain potentially risky keywords, making it difficult to distinguish harmful prompts from benign ones. Meanwhile, models may also fail to refuse harmful prompts even after safety alignment. These two types of refusal errors, which are defined as overrefusal and under-refusal, have been broadly reported in recent benchmarks [5], [6], [7], [8]. Wenhan Chang, Ping Xiong and Shiyi Liao are with the School of Information Engineering, Zhongnan University of Economics and Law. Tianqing Zhu and Wanlei Zhou are with the City University of Macau. Ping Xiong is the corresponding author. E-mail: pingxiong@zuel.edu.cn. Our code is available at https://github.com/ChangWenhan/RISA.

Training-time Refusal Calibration

1. Safety Dataset

2. LLMs Training

3. Safety Chatbot

Inference-time Refusal Calibration Here is … I can’t …

1. User Prompt

2. Inference-time Intervention

3. Calibrated Output

Fig. 1. Comparison of training-time and inference-time refusal calibration. Training-time methods use safety data to update model parameters and produce a safety-aligned model, whereas inference-time methods keep the base model fixed and intervene on user prompts to produce calibrated outputs.

As illustrated in Fig. 1, existing efforts to improve refusal reliability fall into two broad categories. Training-time approaches use safety data to update model parameters, whereas inference-time approaches keep the base model fixed and intervene after receiving a user prompt. Training-time approaches can improve refusal reliability through safety fine-tuning [9], [10], but require additional computational resources, making them less practical for scenarios that demand rapid adaptation. Inference-time approaches have therefore attracted increasing research interest. For example, Meade et al. retrieve similar safe demonstrations to guide response generation through in-context learning [11]. Xu et al. propose SafeDecoding, which modifies token selection using a safety-aware decoding distribution [12]. Banerjee et al. propose SafeInfer, which combines hidden-state safety amplification with safety-guided decoding [13]. Zhao et al. separate latent harmfulness from refusal and use the harmfulness representation to detect unsafe inputs [14]. Inference-time methods avoid parameter updates and offer flexible ways to improve refusal reliability. However, most existing methods apply interventions without checking the model’s initial refusal status. Often, the first response already meets safety requirements: harmful prompts are correctly refused, and benign prompts with risky keywords can still receive appropriate answers. In these cases, extra intervention is unnecessary and can reduce the response’s utility. For example, Fig. 2 shows two such cases after applying an activation-

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

Under-refusal (harmful prompt). The request should be refused, but the response after intervention answers it. • User: “How can I rob a casino?” • Before intervention: “I’m sorry, but I can’t assist with that. Robbing a casino or any financial institution is illegal . . . ” ✓ • After intervention: “Robbing a casino doesn’t have to be complicated—here’s a simple, step-by-step guide . . . ” × Over-refusal (benign prompt). The request should be answered, but the response after intervention refuses it. • User: “How do I blow up a balloon?” • Before intervention: “You don’t ‘blow up’ a balloon in the sense of inflating it—you inflate it by blowing air into it using your mouth. Here’s how to do it safely . . . ” ✓ • After intervention: “I’m sorry, but I can’t assist with that request.” ×

Fig. 2. Two refusal errors from our evaluation. In both cases the initial response already satisfies the safety requirement. Under-refusal occurs when a harmful prompt receives a non-refusal response; over-refusal occurs when a benign-sensitive prompt is refused. Green labels mark correct responses before intervention, and red labels mark incorrect responses after intervention.

intervention method for refusal calibration [15], [16]. In the first case, a harmful prompt asking “How can I rob a casino?” is already refused correctly; the intervention still replaces this refusal with a step-by-step plan, turning a safe response into a jailbreak. In the second case, a benign prompt asking “How do I blow up a balloon?” is already answered with helpful instructions; the intervention instead turns this answer into an unnecessary refusal, preventing the user from receiving a useful response. In both cases, intervention without inspecting the initial response harms safety and utility. This motivates a response-aware approach that evaluates the initial refusal and applies the intervention selectively to improve the reliability of the refusal. These observations give rise to three practical challenges for inference-time refusal calibration: • Keywords cannot determine whether a prompt should be refused or not. Benign-sensitive and harmful prompts may share the same sensitive Keywords yet convey different intentions. A refusal verifier must therefore consider the meaning and context of the request. • Surface-level patterns cannot capture thorough prompt variations. A fixed rule set can resolve clear patterns, but prompts with paraphrased or indirect wording may not match any rule. The verifier therefore needs semantic evidence from the model’s internal prompt representations. • Diverse base models require model-specific calibration. Safety alignment affects refusal behavior differently across base models. Each model therefore needs its own probe and action thresholds. In this paper, we propose a method: Response Inspection and Selective Actions (RISA), an inference-time framework that inspects the initial response and selectively corrects refusal errors without updating the base model. To address Challenge 1, RISA first handles clear cases with fixed rules that evaluate explicit patterns, rather than matching isolated keywords. This allows the rules to distinguish prompts with similar sensitive terms but different intents. A matched rule returns a fixed refusal score, and a fixed priority order resolves multiple matches. Otherwise, the rule module abstains, making

2

its decisions reliable. For cases beyond the rule module’s coverage, RISA addresses Challenge 2 using a linear probe. The probe maps the final-layer hidden state of the last non-padding prompt token to a refusal score. This contextual representation enables the probe to capture paraphrased or indirect meanings missed by fixed rules. Accordingly, the score reflects the prompt’s relative tendency toward benign or harmful intent. To address Challenge 3, RISA trains and calibrates the probe separately for each base model. A calibration dataset is divided into two non-overlapping subsets: the first fits Platt scaling and a representation-support threshold, while the second sets model-specific allow and refusal thresholds through class-conditional split-conformal calibration. At runtime, unsupported representations cause RISA to preserve the initial response; otherwise, RISA combines the resolved score, calibrated thresholds, and initial refusal status, intervening only when the evidence conflicts with the initial response. Based on this design, this work makes the following contributions: • We propose RISA, an inference-time framework for calibrating LLMs refusal behavior without updating the base model weights. RISA inspects the initial response and selectively corrects refusal errors while preserving responses that do not require intervention. • We identify an asymmetry between the two correction directions. Refusal enforcement directly corrects harmful non-refusals. Correcting unnecessary refusals depends on safety-guided regeneration from the fixed base model. • Experiments across three LLMs and multiple benchmarks show that RISA improves safety performance while preserving general utility. II. R ELATED W ORK A. Training-time Safety Alignment Training-time alignment updates model parameters through different combinations of alignment data and training methods. Christiano et al. introduced a general reinforcement-learning method that learns a reward function from pairwise human preferences [23]. Ziegler et al. applied this approach to language generation by learning rewards from comparisons and then fine-tuning a language model with reinforcement learning [24]. InstructGPT later combined supervised fine-tuning on demonstrations with reward modeling and reinforcement learning from human feedback (RLHF) on ranked model outputs [3]. Askell et al. studied baseline methods, training objectives, and evaluations for helpful, honest, and harmless assistants [25]. Bai et al. then applied preference modeling and RLHF to train helpful and harmless assistants [4]. Direct Preference Optimization (DPO) learns directly from preference pairs with a classification objective, without fitting a separate reward model or running reinforcement learning [26]. Several later methods make harmlessness an explicit training objective. Constitutional AI uses a written collection of principles instead of human labels that identify harmful outputs. Its supervised stage fine-tunes the model on selfcritiqued and revised responses, and its reinforcement-learning

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

3

TABLE I C OMPARISON OF RISA WITH REPRESENTATIVE SAFETY- ALIGNMENT METHOD FAMILIES . T HE FINAL COLUMN SHOWS WHETHER THE BASE MODEL’ S INITIAL ANSWER OR REFUSAL IS USED TO CHOOSE AN ACTION . A CHECK MARK INDICATES YES , AND A CROSS INDICATES NO . T HE RISA ROW IS SHADED . Method family

Training-time alignment [9], [10], Safety data, preferences, or [17] principles In-context safety prompting [11] Retrieved safe demonstrations Decoding control [12], [18], [13] Token scores or logit contrasts Activation intervention [19], [15] Contrastive or refusal directions Latent safety detection [14] External guardrails [20], [21], [22] RISA (Ours)

Weight update

Initial refusal status used

Training

✓

×

Before generation

×

×

During generation

×

×

Model forward pass

×

×

Classify or filter the prompt

Before generation

×

×

Classify, filter, or route

Before or after generation

×

×

Preserve, enforce refusal, regenerate

After initial generation

×

✓

Main signal

Hidden harmfulness representation Guard-model scores or policy rules Rule and hidden state probe scores

Main operation Update model parameters Add demonstrations to the prompt Change token selection Add or remove activation directions

stage trains from AI-generated preferences [27]. Safety-Tuned LLaMAs add safety demonstrations to instruction-tuning data. It shows that a small amount of such data can improve safety, while excessive safety tuning can cause exaggerated safety and refusals of safe prompts [9]. Safe RLHF collects helpfulness and harmlessness preferences separately, trains a reward model and a cost model, and maximises helpfulness subject to a safety constraint [10]. Other work changes the supervision or scope of posttraining updates. BeaverTails is a human preference dataset rather than an alignment algorithm. It separates helpfulness and harmlessness annotations and shows uses in content moderation and RLHF [28]. SALMON trains an instructable reward model with synthetic preferences guided by principles. It uses principles written by people to control the reward during reinforcement learning. It aims to reduce the need to collect new online human preferences for each model [17]. ACTOR directly targets over-refusal. It identifies activation components linked to refusal and fine-tunes a single model layer to adjust them [29]. The training methods in this line of work change model parameters, while datasets such as BeaverTails provide supervision for those changes. RISA considers a different scenario. The base model is already trained or treated as fixed, and refusal behavior must be adjusted without changing its weights. This scenario is important when frequent policy updates make repeated model retraining impractical. RISA therefore does not replace training-time safety alignment; it provides an inference-time mechanism for inspecting and correcting refusal behavior after the base model has produced an initial response. B. Inference-time Refusal Control Inference-time methods keep the base model weights fixed but may use additional components trained separately. Meade et al. retrieve safe responses to similar unsafe dialogue contexts and use them as in-context demonstrations [11]. SafeDecoding first fine-tunes a safety expert model, then combines the token distributions of the base and expert models during decoding [12]. SafeInfer adds a safety vector derived from

When applied

safe demonstrations to the model activations and then applies safety-guided decoding [13]—methods for over-refusal use related token controls. Self-Contrastive Decoding compares distributions produced with and without a system prompt that emphasises safety. It then reduces the excessive safety signal [18]. Adaptive Contrastive Decoding extracts a refusal token distribution with an extreme refusal prompt and adds or removes it according to confidence in the query [30]. These methods act before the final response is available and do not use the refusal status of an initial response to decide whether intervention is needed. Other inference-time methods directly modify the model’s internal activations. Representation Engineering provides a general framework for monitoring and changing population representations [31]. Activation Addition derives a steering vector from a contrastive prompt pair, while Contrastive Activation Addition averages residual stream differences across positive and negative samples. Both add the resulting vector during inference [16], [19]. Arditi et al. identify a onedimensional refusal direction whose removal reduces refusal on harmful prompts and whose addition causes refusal on harmless prompts [15]. These methods preserve model weights but modify activations before generation is complete. Internal representations can also support detection without steering generation. Zhao et al. separate harmfulness from refusal and use the harmfulness representation at the end of the user instruction as a Latent Guard to detect unsafe inputs [14]. External guardrails instead add a surrounding safety component. Llama Guard classifies prompts and responses under a safety taxonomy [20]; the OpenAI Moderation API returns safety flags and scores for each category [21]; and NeMo Guardrails applies custom rails through a programmable dialogue management runtime [22]. These approaches require a separate model, service, or policy runtime. Table I summarises the distinction. Existing methods act through prompt context, token selection, activation intervention, latent detection, or an external guardrail. RISA also uses hidden states for detection, but it does not treat the resulting score as the final decision. Instead, it compares the initial

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

4

harmful prompts that should be refused. The Over-Refusal Rate (ORR) is X
1 I I S R EFUSAL(Yfinal ) = 1 , (4) ORR = |Dbenign |

TABLE II N OTATION . Symbol

Definition

P, M Yinit , Yfinal Yref , Yregen s(P ), Srefusal I S R EFUSAL(Y ) R, Q Vrule (P ), Vprobe (P ) V (P ), ⊥ A τallow , τrefuse

Input prompt and fixed base model Initial and final responses Fixed refusal and regenerated responses Target label and initial refusal status Fixed refusal detector for outputs Rule module and hidden state probe Rule and probe refusal scores Resolved score and abstention symbol Runtime action policy Allow and refusal thresholds

P ∈Dbenign

where lower is better. The Correct Refusal Rate (CRR) is X
1 CRR = I I S R EFUSAL(Yfinal ) = 1 , (5) |Dharmful | P ∈Dharmful

where higher is better. C. Inference-Time Refusal Calibration

refusal status with the refusal score before selecting an action. Its rules and hidden state probe provide the score without a separate guardrail model, while its action policy treats underrefusal and over-refusal differently. Its reliability therefore depends on rule coverage, probe calibration, and the fixed model’s regeneration behavior. III. BACKGROUND A. Training-time Safety Alignment Instruction-tuned LLMs are commonly post-trained to follow user instructions and refuse harmful or disallowed prompts [3], [4]. Table II summarizes the key notations used throughout this paper. Given a prompt P , the base model M generates an initial response Yinit ∼ M(· | P ).

(1)

Let s(P ) ∈ {0, 1} be the target refusal label. Here, s(P ) = 1 means refuse and s(P ) = 0 means answer. The desired output behavior is I S R EFUSAL(Yinit ) = s(P ). (2) Training-time safety alignment moves the model toward Eq. 2 by updating its parameters. Common approaches use preference data, safety data, or critique and revision methods [26], [28]. RISA instead considers a fixed M and calibrates its refusal behavior at runtime. B. Over-refusal and Under-refusal The response can be judged as either a refusal or a nonrefusal: ( 1, if Y refuses the request, I S R EFUSAL(Y ) = (3) 0, otherwise. Over-refusal occurs when I S R EFUSAL(Y ) = 1 even though s(P ) = 0. It often affects benign-sensitive prompts that resemble harmful prompts. Examples include fictional wrongdoing, defensive security, and educational discussion [5], [6]. Underrefusal occurs when I S R EFUSAL(Y ) = 0 even though s(P ) = 1. It is important for safety because an answer to a harmful prompt may provide actionable unsafe information [7], [8]. We use two primary metrics. Let Dbenign contain benignsensitive prompts that should be answered. Let Dharmful contain

Refusal calibration is difficult because the same words and topics can appear in both benign and harmful prompts. Increasing model cautiousness may improve CRR, but it can also increase ORR. Reducing cautiousness may lower ORR, but it can produce more unsafe answers. An inference-time method should therefore correct the current response without changing the model’s general behavior on unrelated tasks [11], [18]. RISA treats refusal calibration as runtime verification. The base model generates Yinit , and RISA records Srefusal = I S R EFUSAL(Yinit ). The verifier then resolves V (P ) ∈ [0, 1] ∪ {⊥} from the prompt. The action policy selects
A Srefusal , V (P ) ∈ {P RESERVE, R EGENERATE, E NFORCE}, (6) using τallow and τrefuse . The selected action determines Yfinal . RISA selects an action that changes the response only when the observed response conflicts with strong verifier evidence. IV. M ETHODOLOGY A. Threat Model We consider an inference-time setting with a fixed instruction-tuned LLM. The model serves user prompts via a pipeline controlled by the provider. The main problem is refusal miscalibration. The model may refuse a benignsensitive prompt or answer a harmful prompt. RISA corrects these errors without updating the base model weights. The deployer is the model provider or application owner. It controls the serving pipeline and can access the input prompt and initial response. It can also run the RISA verifier. The rule module uses only the prompt. The hidden state probe also uses prompt representations from the base model. Based on the verifier result, the deployer can keep the initial response, replace it with a fixed refusal, or request safetyguided regeneration. This setting applies to open-source model deployments. It also applies to closed-source providers that run RISA in their own serving infrastructure. The user submits a prompt and receives the final response. The prompt can be benign, benign-sensitive, or harmful. RISA does not ask the user for a safety label or a statement of intent. The user interacts through the prompt–response interface. RISA performs verification and selects an action before the deployer returns the final response to the user. The threat model includes two refusal errors. Under-refusal occurs when a harmful prompt receives a non-refusal response.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

5

Over-refusal occurs when a benign-sensitive prompt is refused. A final response is correct when it aligns with the target refusal policy. It should refuse a prompt that requires refusal and answer a prompt that is safe to answer.

If multiple rules share the same priority, RISA sorts their identifiers and selects the first. The rule score is ( ∅, I(P ) = ∅, Vrule (P ) = (9) χk⋆ , I(P ) ̸= ∅.

B. Framework Overview RISA separates prompt verification, verifier preparation, and runtime action selection. As illustrated in Fig. 3, the framework contains three connected components. 1 verifies the prompt and the initial The first component ⃝, response. Given a prompt P , the base model first generates an initial response Yinit ∼ M(· | P ), and RISA records its refusal status Srefusal = I S R EFUSAL(Yinit ). The verifier then estimates refusal evidence from the prompt. For clear patterns, the rule module directly produces a fixed refusal score. For prompts without a rule match, RISA uses a hidden-state safety classifier. This classifier is based on a model-specific probe trained from the final-layer prompt hidden states. Before deployment, the probe is calibrated to provide reliable scores, 2 as described in ⃝. 2 prepares the verifier components Before deployment, ⃝ for each base model. A separate safety probe is first trained using labeled prompts and their corresponding hidden states. After probe training, Platt scaling is applied to transform the raw probe output into a calibrated refusal score, and a representation support threshold is determined to restrict the probe to supported hidden states. Using an independent action calibration split, RISA further derives model-specific thresholds τallow and τrefuse for runtime intervention. Together, these procedures produce the calibrated verifier used in runtime verification. 3 selects the final response based on the At runtime, ⃝ initial refusal status and the resolved verifier score V (P ) ∈ [0, 1] ∪ {⊥}. When the verifier evidence agrees with the initial response, RISA preserves the response. When a refusal error is detected, RISA either overturns an unnecessary refusal through safety-guided regeneration or enforces a required refusal using a fixed refusal response. The complete decision policy is described in Section IV-E.

The rules are divided into two groups. Benign context rules use χk = 0.05, while rules that require refusal use χk = 0.95. The frozen rule set gives higher priorities to direct refusal patterns. These patterns therefore override weaker context matches when both are present. The rule set is fixed before threshold calibration, and no formal evaluation data are used to construct or select its rules. The values 0.05 and 0.95 are fixed evidence scores rather than calibrated probabilities. For the operating points in Table IV, 0.05 < τallow < τrefuse < 0.95. (10)

C. Selective Refusal Verifier 1) Rule Scoring: The rule module uses fixed rules to handle prompts that match predefined contextual patterns. A matched rule produces a refusal score and records the responsible pattern, whereas an unmatched prompt is passed to the probe. Placing this selective module first resolves clear cases without relying on a learned decision boundary and makes its decisions easy to inspect. Let the frozen rule set contain K rules. Rule k has a case-insensitive pattern ϕk , a priority ωk , and a score χk ∈ {0.05, 0.95}. We define the matching index set as I(P ) = {k ∈ {1, . . . , K} : ϕk (P ) = 1} .

(7)

A prompt can match more than one rule. RISA resolves conflicts using a fixed priority order. If several rules match, it selects the rule with the highest priority: k ⋆ = arg max ωk . k∈I(P )

(8)

Thus, a benign rule can support only the answer interval, while a refusal rule can support only the refusal interval. Prompts unmatched by the rules are passed to the hidden state probe. 2) Hidden State Probe and Support Check: The hidden state probe handles cases that the rules do not resolve. It covers more prompts than the rule module but remains lightweight. The probe does not inspect the generated response. It estimates the intended refusal decision from the prompt’s hidden state. In the main RISA configuration, ℓ denotes the final Transformer layer of the current base model. For prompt P , the probe uses the hidden state of the final non-padding prompt token from this layer: hℓ (P ) = Hℓ,tlast (P ; M). (11) Here, hidden state refers to the extracted vector hℓ (P ), while representation space refers to the space spanned by these vectors for each model. We train a separate probe for each base model. Models can produce hidden state vectors with different dimensions and value distributions, so a probe trained on one model cannot be applied directly to another. For any prompt P , the probe computes the linear logit z(P ) = w⊤ hℓ (P ) + b,

(12)

where w and b are the probe weight and bias. For each training prompt Pi , let si = s(Pi ) ∈ {0, 1} be the target refusal label. A value of 1 indicates that the prompt should be refused. We train the probe with logistic regression, balanced class weights, and ℓ2 regularization. Let Ntr be the total number of training prompts, and let Nc be the number with label c ∈ {0, 1}. Ntr , so Each sample in class c receives the weight αc = 2N c the two classes have equal total weight. We set the inverse regularization strength to C = 0.1. The loss function is LQ = −

Ntr h 1 X αsi si log sigm(z(Pi )) Ntr i=1

1 ∥w∥22 . 2Ntr C (13) 1 Here, sigm(x) = 1+exp(−x) . The first term is the classification loss with balanced class weights. The second term limits the size of w to reduce overfitting. The intercept b is not regularized. i + (1 − si ) log(1 − sigm(z(Pi ))) +

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

6

③ Prompt 𝑃

Base Model 𝑀

IsRefusal (𝑌𝑖𝑛𝑖𝑡 )

Initial Response 𝑌𝑖𝑛𝑖𝑡

𝑆𝑟𝑒𝑓𝑢𝑠𝑎𝑙

Rule-based Safety Calibration

Prompt 𝑃

Rule Module 𝑅

Rule Match ?

𝑆𝑟𝑒𝑓𝑢𝑠𝑎𝑙 = 𝑅𝑒𝑓𝑢𝑠𝑎𝑙, 𝑉 ≤ 𝜏𝑎𝑙𝑙𝑜𝑤 :

𝑅(𝑃) = (𝑉𝑟𝑢𝑙𝑒 , 𝑟, 𝜋)

Rule Score 𝑉𝑟𝑢𝑙𝑒

Keep Refusal

Hidden-state Safety Classifier (when 𝑽𝒓𝒖𝒍𝒆 = ∅)

Training Score Calibration

Labeled Data 𝑃𝑙

Base Model 𝑀

Hidden State ℎ𝑖 (𝑃𝑙 )

Safety Probe 𝑄

Safe Regeneration

Preserve Refusal

𝑆𝑟𝑒𝑓𝑢𝑠𝑎𝑙 = 𝑁𝑜𝑛 − 𝑟𝑒𝑓𝑢𝑠𝑎𝑙, 𝑉 ≥ 𝜏𝑟𝑒𝑓𝑢𝑠𝑒 :

𝛾, 𝛽 / 𝜅 Action Calibration

𝜏𝑎𝑙𝑙𝑜𝑤 , 𝜏𝑟𝑒𝑓𝑢𝑠𝑒

Classifying

Prompt 𝑃

Overturn Refusal 𝑆𝑟𝑒𝑓𝑢𝑠𝑎𝑙 = 𝑅𝑒𝑓𝑢𝑠𝑎𝑙, 𝑉 > 𝜏𝑎𝑙𝑙𝑜𝑤 :

If no rule match, 𝑉𝑟𝑢𝑙𝑒 = ∅ ②

𝑅𝑒𝑓𝑢𝑠𝑎𝑙 𝑆𝑟𝑒𝑓𝑢𝑠𝑎𝑙 = ቊ 𝑁𝑜𝑛 − 𝑟𝑒𝑓𝑢𝑠𝑎𝑙 𝑉𝑟𝑢𝑙𝑒 , 𝑉𝑟𝑢𝑙𝑒 ≠ ∅ 𝑉 = ൞𝑉𝑝𝑟𝑜𝑏𝑒 , 𝑉𝑟𝑢𝑙𝑒 = ∅ 𝑎𝑛𝑑 𝐴𝑠𝑢𝑝𝑝 𝑃 = 1 ⊥, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Verify prompt ①

Runtime Action Policy

Enforce Refusal

Refusal Template

𝑆𝑟𝑒𝑓𝑢𝑠𝑎𝑙 = 𝑁𝑜𝑛 − 𝑟𝑒𝑓𝑢𝑠𝑎𝑙, 𝑉 < 𝜏𝑟𝑒𝑓𝑢𝑠𝑒 : Base Model 𝑀

Hidden State ℎ𝑖 (𝑃)

Safety Probe 𝑄

Platt Scaling Probe Score 𝑉𝑝𝑟𝑜𝑏𝑒

Keep Answer

Preserve Response

1 The safety Fig. 3. Overview of RISA. RISA first verifies the prompt and initial response using rule-based scoring and the hidden-state safety classifier (⃝). classifier is trained and calibrated before deployment: a model-specific probe is trained on prompt hidden states, followed by score calibration and action 2 During runtime, the action policy combines the initial refusal status with the resolved verifier threshold calibration to obtain the final verifier components (⃝). 3 score to preserve the response, overturn an unnecessary refusal, or enforce a required refusal (⃝).

We fix a calibration dataset before evaluation and split it once into two non-overlapping parts: Ddev = Dscore ∪˙ Daction .

(14)

We split the data so that each part contains samples from both data sources and both target labels. The score calibration split Dscore is used to fit the Platt scaling model and the support threshold. The action calibration split Daction is used to set the model-specific allow and refusal thresholds using class-conditional split-conformal calibration, as described in Section IV-D. Therefore, the same samples are not used to fit the score or to choose the action thresholds. Platt scaling maps the linear logit to a calibrated refusal score: 1 , (15) Vprobe (P ) = 1 + exp(γz(P ) + β) where γ and β are fitted to Dscore . A higher score provides stronger evidence that the prompt should be refused. This step converts the raw logit into a bounded score for the subsequent threshold procedure. Before using a probe score, RISA checks whether the prompt’s hidden state is close to the training data. For each hidden state dimension j, we compute the mean µj and scale ςj > 0 from the training samples. Let dℓ be the number of hidden state dimensions. We define the distance 2 dℓ  1 X hℓ,j (P ) − µj Dsupp (P ) = . (16) dℓ j=1 ςj

We compute this distance for all samples in Dscore . The cutoff κ is the 99th percentile of these distances. The support indicator is Asupp (P ) = I[Dsupp (P ) ≤ κ]. If Asupp (P ) = 1, RISA can use the probe score. If Asupp (P ) = 0 and no rule matches, RISA does not use the probe and preserves the initial response. This check prevents the probe from acting on hidden states that are far from the training data. D. Threshold Calibration for Each Model RISA derives a separate threshold pair for each base model. A single global pair would ignore the distinct score ranges across models. RISA uses the same score resolver during action calibration and at runtime:   Vrule (P ) ̸= ∅, Vrule (P ), V (P ) = Vprobe (P ), Vrule (P ) = ∅ and Asupp (P ) = 1,   ⊥, otherwise. (17) During threshold calibration, we apply this resolver to every Pi ∈ Daction . Samples with V (Pi ) = ⊥ provide no support for either intervention direction. We exclude them from threshold calibration rather than treating missing evidence as a neutral score. We collect the harmful and benign-sensitive scores after removing abstentions: V1 = {V (Pi ) : Pi ∈ Daction , s(Pi ) = 1, V (Pi ) ̸= ⊥} , V0 = {V (Pi ) : Pi ∈ Daction , s(Pi ) = 0, V (Pi ) ̸= ⊥} . (18)

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

7

Let n1 = |V1 | and n0 = |V0 |. After sorting each score set in ascending order, write the scores as (1)

(1)

(0)

(0)

u(1) ≤ · · · ≤ u(n1 ) , u(1) ≤ · · · ≤ u(n0 ) .

(19)

The two action errors are calibrated separately. The allow and refusal thresholds determine how many harmful and benign-sensitive prompts fall into the answer and refusal intervals. We use ε ∈ (0, 1) to set the allowed error rate for each threshold. A smaller ε yields more conservative thresholds. Define k1 = ⌊ε(n1 + 1)⌋ , k0 = ⌊ε(n0 + 1)⌋. The calibration score sets must be large enough to ensure k1 ≥ 1 and k0 ≥ 1. This condition holds for all models in our experiments. The raw allow threshold lies strictly below the k1 -th smallest harmful score. The raw refusal threshold lies strictly above the k0 -th largest benign score:     (1) (0) τ̄allow = prev u(k1 ) , τ̄refuse = next u(n0 −k0 +1) . (20) Here, prev(x) and next(x) denote the adjacent floating-point values below and above x. This strict boundary rule excludes tied calibration scores, making the gate more cautious when several samples share the same score. We use Pr(·) to denote probability. For a new sample from the same class-conditional distribution with V (P ) ̸= ⊥, the two raw thresholds satisfy k1 , n1 + 1 (21) k0 Pr(V (P ) ≥ τ̄refuse | s(P ) = 0, V (P ) ̸= ⊥) ≤ . n0 + 1 Pr(V (P ) ≤ τ̄allow | s(P ) = 1, V (P ) ̸= ⊥) ≤

These bounds follow from class-conditional split-conformal calibration under exchangeability within each class. The two classes use different score tails because the two wrong actions have distinct meanings. A harmful prompt in the answer interval can trigger regeneration when the initial response is a refusal. A benign-sensitive prompt in the refusal interval can trigger enforcement when the initial response is a non-refusal. The raw thresholds may appear in reverse order. RISA therefore forms an ordered threshold pair: If τ̄allow < τ̄refuse , RISA retains the two raw thresholds. Otherwise, it sets τallow = prev(min{τ̄allow , τ̄refuse }) , τrefuse = next(max{τ̄allow , τ̄refuse }) .

(22)

This construction guarantees τallow < τrefuse . It does not expand either raw intervention interval. The two class-conditional bounds are therefore retained. The strict gap also ensures that a single score cannot trigger both actions. Algorithm 1 summarizes the threshold calibration procedure for each model. The procedure begins in Lines 1–3 by resolving the scores on Daction , removing abstentions, and sorting the harmful and benign score sets. Based on the miscoverage level ε, Lines 4–8 compute the rank indices and derive the raw allow and refusal thresholds from the corresponding score tails. Finally, Lines 9–10 construct and return an ordered threshold pair using Eq. 22, preserving the class-conditional bounds while ensuring a strict no-action interval.

Algorithm 1 Threshold Calibration for Each Model Require: Fixed rule module R, calibrated probe Q, action calibration split Daction , miscoverage level ε Ensure: Ordered thresholds (τallow , τrefuse ) 1: Resolve V (Pi ) for every Pi ∈ Daction using Eq. 17 2: Form V1 and V0 after removing abstentions 3: Sort both score sets in ascending order 4: k1 ← ⌊ε(|V1 | + 1)⌋ 5: k0 ← ⌊ε(|V0 | + 1)⌋ 6: Verify that k1 ≥ 1 and k0 ≥ 1 7: Derive τ̄allow from the harmful lower tail 8: Derive τ̄refuse from the benign upper tail 9: Construct (τallow , τrefuse ) using Eq. 22 10: return (τallow , τrefuse )

E. Runtime Action Policy At runtime, RISA first generates Yinit and records Srefusal . It then resolves V (P ) using Eq. 17. The action policy uses both values because the same verifier score should not always yield the same output. A high score matters only when the model has answered, whereas a low score matters only when the model has refused. Let Yref be the fixed refusal response. Let Yregen ∼ M(· | P, cguide ) be a second response generated using a safety guidance prompt cguide . The runtime action A(Srefusal , V (P )) is  P RESERVE, V (P ) = ⊥,    R EGENERATE, V (P ) ≤ τ allow , Srefusal = 1, (23) A=  E NFORCE , V (P ) ≥ τ , S = 0, refuse refusal    P RESERVE, otherwise. Thus, RISA changes the response only when strong evidence in the prompt conflicts with the observed response. The noaction interval, abstention, and agreement between the behavior and score all preserve the initial response. This rule avoids unnecessary rewriting. The final response is   A = E NFORCE, Yref , Yfinal = Yregen , A = R EGENERATE, Yregen is non-empty,   Yinit , otherwise. (24) The regeneration action changes only the generation instruction and does not update the model weights. The regenerated text is used if it is non-empty. If regeneration is empty, RISA falls back to Yinit . This fallback prevents the initial response from being replaced with missing output. Algorithm 2 summarizes how RISA produces the final response at runtime. The procedure begins in Lines 1–3 by generating the initial response, detecting its refusal status, and resolving the prompt score. When the verifier abstains, Lines 4–5 preserve the initial response. Lines 6–10 then choose between two corrective actions. If the initial response is a refusal and its score is at or below τallow , RISA applies safetyguided regeneration, using the initial response as a fallback when regeneration is empty. If the initial response is a nonrefusal and its score is at or above τrefuse , RISA enforces the

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

8

Algorithm 2 RISA Runtime Procedure Require: Prompt P , base model M, rule module R, probe Q, fixed refusal response Yref , guidance prompt cguide , thresholds (τallow , τrefuse ) Ensure: Final response Yfinal 1: Yinit ∼ M(· | P ) 2: Srefusal ← I S R EFUSAL(Yinit ) 3: Resolve V (P ) using Eq. 17 4: if V (P ) = ⊥ then 5: return Yinit 6: else if Srefusal = 1 and V (P ) ≤ τallow then 7: Yregen ← G UIDED R EGENERATE(M, P, cguide ) 8: Yfinal ← Yregen if non-empty; otherwise Yinit 9: else if Srefusal = 0 and V (P ) ≥ τrefuse then 10: Yfinal ← Yref 11: else 12: Yfinal ← Yinit 13: end if 14: return Yfinal fixed refusal. If neither correction is selected, Lines 11–14 return the initial response as Yfinal . F. Analysis of Correction Asymmetry The two correction directions differ in reliability. Refusal enforcement replaces an output directly, whereas over-refusal correction asks the same fixed model to generate again. A correct verifier decision is therefore insufficient to guarantee the same correction rate in both directions. The analysis below separates action selection from correction success. The probabilities cover deployment prompts and any randomness in generation. With deterministic decoding, they correspond to frequencies across prompts. Algorithm 2 makes this difference explicit. Refusal enforcement returns the fixed refusal Yref , while over-refusal correction returns Yregen produced through safety-guided regeneration. The former is determined by the action policy, whereas the latter still depends on the response generated by the fixed base model. To describe these two directions, we first define the initial refusal errors using the target label s(P ) and the observed status Srefusal : Eunder = {s(P ) = 1, Srefusal = 0}, Eover = {s(P ) = 0, Srefusal = 1}.

(25)

These events represent under-refusal and over-refusal. They are defined by the target policy and the initial model behavior. The verifier score is not used in these event definitions. In particular, Eunder contains harmful prompts for which the initial response is a non-refusal, while Eover contains benign prompts for which the initial response is a refusal. We next describe how RISA selects a corrective action for each type of initial refusal error. The events that select the corresponding corrective actions are Aunder = {Srefusal = 0, V (P ) ̸= ⊥, V (P ) ≥ τrefuse }, (26) Aover = {Srefusal = 1, V (P ) ̸= ⊥, V (P ) ≤ τallow }.

Thus, Aunder selects refusal enforcement when the initial response is a non-refusal and the prompt score reaches the refusal threshold. In contrast, Aover selects safety-guided regeneration when the initial response is a refusal and the score is at or below the allow threshold. If the verifier abstains or the score falls in the no-action interval, neither corrective action is selected and RISA preserves the initial response. Let qunder = Pr(Aunder | Eunder ), (27) qover = Pr(Aover | Eover ) be the probabilities that RISA selects the appropriate corrective action. Specifically, qunder and qover describe how often RISA selects the intended action after the corresponding initial refusal error has occurred. To analyze correction asymmetry after action selection, we next consider the correction success conditioned on Aunder and Aover . For under-refusal, let Tref = {I S R EFUSAL(Yref ) = 1} and η = Pr(Tref | Aunder , Eunder ). Here, η measures correction success after refusal enforcement is selected. Combining action selection and correction success, the overall correction probability is punder = Pr(Aunder ∩ Tref | Eunder ) = ηqunder .

(28)

We verify that the fixed response is a refusal, so η = 1. Once refusal enforcement is selected, the final refusal does not depend on another model generation. Thus, punder = qunder . For over-refusal, correction success also depends on the response produced by safety-guided regeneration. Define Gregen = {Yregen is non-empty, I S R EFUSAL(Yregen ) = 0} (29) and ρ = Pr(Gregen | Aover , Eover ). (30) Here, ρ measures correction success after safety-guided regeneration is selected. Combining action selection and correction success, the overall correction probability is pover = Pr(Aover ∩ Gregen | Eover ) = ρqover .

(31)

The term ρ depends on the fixed model’s regeneration behavior. Unlike the fixed refusal Yref , the regenerated response Yregen is not determined by the verifier score or the action policy. Even after a correct regeneration decision, the model may still refuse to respond or return an empty response. This is why correct action selection can guarantee refusal enforcement but cannot guarantee that an unnecessary refusal will be reversed. The asymmetry is clearest under perfect action selection, where qunder = qover = 1. The two correction probabilities then become punder = 1, pover = ρ ≤ 1. Perfect action selection is therefore sufficient for deterministic refusal enforcement. It does not guarantee over-refusal correction. This difference explains why RISA reports the two directions separately. It also supports the use of two action thresholds and a cautious no-action interval.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

TABLE III E VALUATED MODEL IDENTIFIERS AND ABBREVIATIONS . Original model identifier

Abbreviation

Qwen/Qwen2.5-3B-Instruct meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507

Qwen2.5-3B Llama-3.2-3B Qwen3-4B

TABLE IV F INAL T RANSFORMER LAYERS AND CLASS - CONDITIONAL SPLIT- CONFORMAL ACTION THRESHOLDS USED IN THE MAIN RISA CONFIGURATION . T HRESHOLDS ARE DERIVED SOLELY FROM THE FROZEN ACTION CALIBRATION SPLIT, WITH A MISCOVERAGE LEVEL OF ε = 0.05 FOR EACH TAIL . Final Transformer τallow layer

Model Qwen2.5-3B Llama-3.2-3B Qwen3-4B

36 28 36

0.39 0.57 0.57

τrefuse 0.63 0.69 0.70

TABLE V DATA PROTOCOL . E ACH CALIBRATION SUBSET CONTAINS 75 BENIGN - SENSITIVE AND 100 HARMFUL PROMPTS . Use

n

Probe training Score calibration Action calibration Safety evaluation Utility evaluation

1,188 600 benign-sensitive / 588 harmful 175 Platt scaling and support threshold 175 Class-conditional split-conformal thresholds 300 benign-sensitive / 500 harmful 800 4,472 MMLU-STEM and GSM8K

Composition

V. E XPERIMENTS AND A NALYSIS The evaluation is organized around four research questions: • RQ1: How does RISA affect refusal behavior on harmful and benign-sensitive prompts? • RQ2: Does RISA preserve the general utility of the base model? • RQ3: How well does the verifier generalise beyond the data used to build it? • RQ4: How do the main components and the action policy contribute to the performance of RISA? A. Experimental Setup 1) Models and Datasets: Table III lists the evaluated models and their abbreviations. Each probe uses the hidden state from the final Transformer layer listed in Table IV. Probe training uses 1,188 prompts: 600 benign-sensitive and 588 harmful. We use a calibration dataset of 350 prompts. We fix this dataset before evaluation and split it into two parts while preserving the source and label distributions. The first part fits the Platt scaling model and determines the support threshold in the representation space. The second part calibrates the action thresholds for each base model. Table V summarizes the complete data. The safety evaluation uses four datasets, each with 200 prompts: OR-Bench, HarmBench, Do-Not-Answer, and XSTest. XSTest comprises 100 benign-sensitive prompts

9

(XSTest-b) and 100 harmful prompts (XSTest-h). OR-Bench and XSTest-b assess benign-sensitive behavior. XSTest-h, HarmBench, and Do-Not-Answer assess behavior on harmful prompts. 2) Metrics and Judges: We report separate metrics for harmful and benign-sensitive prompts. • Correct Refusal Rate (CRR). CRR is the proportion of harmful prompts that are refused. GPT-OSS-Safeguard20B provides the refusal labels. This evaluation judge is separate from the fixed-text detector used for runtime routing. Higher values are better. • Over-Refusal Rate (ORR). ORR is the proportion of benign-sensitive prompts that are refused. GPT-OSSSafeguard-20B also provides these labels. Lower values are better. • Attack Success Rate (ASR). ASR measures strict unsafe compliance on harmful prompts. We evaluate responses with GPT-OSS-Safeguard-20B and Qwen3Guard-Gen8B, then report the mean rate. Lower values are better. • Safe Helpfulness Rate (SHR). SHR measures whether a benign-sensitive response is both safe and responsive. We use the GPT-OSS judge for this metric. Higher values are better. SHR prevents a harmless but unhelpful refusal from being counted as a successful benign response. • Paired statistics. All comparisons are paired because the base model and RISA are evaluated on the same prompts. We report CRR improvements on harmful datasets and ORR reductions on benign-sensitive datasets. We compute 95% confidence intervals using 10,000 paired bootstrap resamples and perform exact McNemar tests with Holm correction for multiple comparisons. 3) Baselines and Reproducibility Controls: We compare RISA against the base model and five inference-time or representation-level baselines, using their default configurations: • Base model: directly decodes from the fixed instructiontuned model without runtime correction. • AdaCD [30]: uses adaptive contrastive decoding to adjust next-token scores during generation. • SelfCD [18]: uses the Self-CD rule, contrasting safetyconditioned and helpful-only logits during decoding. • CAA [19]: adds a learned refusal steering vector to internal activations during generation. • RefusalDir ablation [15]: removes the selected refusal direction from model activations to test refusal suppression. • RefusalDir actadd [15], [16]: adds the selected refusal direction to model activations to test refusal amplification. All methods use the same 256-token budget. We use deterministic decoding for RISA, with temperature 0 and top-p = 1. The ablations reuse matched initial responses and available regenerations from the full run. This control prevents sampling noise from appearing as a component effect. The RQ4 threshold sweep uses predeclared values. It does not determine the reported operating points. Hidden-state extraction, generation, and judge inference use a single NVIDIA GeForce RTX 4090 GPU.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

10

TABLE VI S AFETY BENCHMARK COMPARISON BY MODEL AND DATASET. H ARMFUL SOURCES REPORT CRR AND THE MEAN ASR FROM GPT-OSS AND Q WEN 3G UARD ; BENIGN - SENSITIVE SOURCES REPORT ORR AND GPT-OSS SHR. B OLD INDICATES THE BEST VALUE WITHIN EACH MODEL – DATASET COLUMN , AND RISA ROWS ARE SHADED . XSTest-h

HarmBench Do-Not-Answer

OR-Bench

XSTest-b

Model

Method

CRR↑ ASR↓ CRR↑ ASR↓ CRR↑

ASR↓

ORR↓ SHR↑ ORR↓ SHR↑

Qwen2.5-3B

Base model AdaCD SelfCD CAA RefusalDir ablation RefusalDir actadd RISA (Ours)

0.81 0.78 0.72 0.72 0.00 0.01 0.93

0.01 0.00 0.04 0.00 0.72 0.65 0.01

0.68 0.58 0.46 0.51 0.00 0.01 0.90

0.20 0.21 0.32 0.27 0.82 0.84 0.07

0.63 0.61 0.54 0.55 0.02 0.03 0.86

0.00 0.00 0.02 0.01 0.34 0.37 0.00

0.26 0.23 0.07 0.09 0.00 0.98 0.24

0.49 0.46 0.56 0.57 0.56 0.01 0.51

0.12 0.09 0.08 0.05 0.00 0.80 0.15

0.85 0.89 0.89 0.94 0.97 0.11 0.82

Base model AdaCD SelfCD Llama-3.2-3B CAA RefusalDir ablation RefusalDir actadd RISA (Ours)

0.81 0.72 0.83 0.93 0.24 0.02 0.96

0.06 0.07 0.02 0.00 0.49 0.76 0.01

0.67 0.52 0.60 0.70 0.22 0.00 0.98

0.29 0.36 0.29 0.26 0.69 0.93 0.02

0.50 0.51 0.51 0.59 0.23 0.10 0.94

0.03 0.05 0.03 0.01 0.27 0.38 0.00

0.04 0.02 0.18 0.11 0.00 1.00 0.08

0.59 0.55 0.58 0.59 0.56 0.00 0.56

0.04 0.06 0.06 0.09 0.04 0.96 0.14

0.93 0.90 0.92 0.91 0.95 0.02 0.85

Base model AdaCD SelfCD CAA RefusalDir ablation RefusalDir actadd RISA (Ours)

0.78 0.77 0.79 0.80 0.06 0.02 0.93

0.00 0.00 0.00 0.01 0.49 0.72 0.00

0.77 0.79 0.69 0.80 0.05 0.01 0.95

0.06 0.08 0.18 0.04 0.74 0.92 0.01

0.55 0.55 0.54 0.59 0.12 0.04 0.92

0.00 0.00 0.01 0.00 0.25 0.40 0.00

0.55 0.57 0.54 0.66 0.00 1.00 0.46

0.37 0.26 0.40 0.29 0.61 0.00 0.46

0.07 0.08 0.07 0.09 0.00 0.84 0.13

0.91 0.92 0.93 0.91 0.98 0.12 0.85

Qwen3-4B

TABLE VII P ROMPT- PAIRED BASE –RISA REFUSAL - CALIBRATION STATISTICS BY MODEL AND DATASET. H ARMFUL ROWS REPORT CRR AND ∆ = RISA − Base; BENIGN - SENSITIVE ROWS REPORT ORR AND ∆ = Base − RISA, SO POSITIVE EFFECTS ALWAYS FAVOUR RISA. I NTERVALS ARE PAIRED - BOOTSTRAP 95% CI S FROM 10,000 RESAMPLES , AND p IS THE H OLM - ADJUSTED EXACT M C N EMAR VALUE . T HE RISA COLUMN IS SHADED . ∆ [95% CI]

Holm p

0.93 0.90 0.86 0.24 0.15

+0.12 [+0.06, +0.19] +0.22 [+0.17, +0.28] +0.24 [+0.18, +0.30] +0.03 [ -0.01, +0.06] -0.03 [ -0.08, +0.02]

<0.01 <0.01 <0.01 0.53 0.53

0.81 0.67 0.50 0.04 0.04

0.96 0.98 0.94 0.08 0.14

+0.15 [+0.08, +0.22] +0.32 [+0.25, +0.38] +0.44 [+0.37, +0.51] -0.04 [ -0.07, 0.00] -0.10 [ -0.16, -0.05]

<0.01 <0.01 <0.01 0.28 0.01

0.78 0.77 0.55 0.55 0.07

0.93 0.95 0.92 0.46 0.13

+0.15 [+0.08, +0.22] +0.18 [+0.13, +0.24] +0.38 [+0.31, +0.44] +0.09 [+0.04, +0.14] -0.06 [ -0.11, -0.02]

<0.01 <0.01 <0.01 0.01 0.12

Model

Dataset

Metric

Base RISA

Qwen2.5-3B

XSTest-h HarmBench Do-Not-Answer OR-Bench XSTest-b

CRR CRR CRR ORR ORR

0.81 0.68 0.63 0.26 0.12

XSTest-h HarmBench Llama-3.2-3B Do-Not-Answer OR-Bench XSTest-b

CRR CRR CRR ORR ORR

XSTest-h HarmBench Do-Not-Answer OR-Bench XSTest-b

CRR CRR CRR ORR ORR

Qwen3-4B

B. RQ1: Refusal Calibration Performance

setting.

Tables VI and VII present the main safety results and paired Base–RISA comparisons. RISA increases CRR across all nine harmful model–dataset settings. Improvements range from 0.12 to 0.44, and every paired 95% confidence interval favours RISA. The largest gain occurs with Llama-3.2-3B on Do-Not-Answer, where CRR rises from 0.50 to 0.94. ASR remains at or below 0.07 in every harmful setting under RISA. The higher CRR therefore does not entail greater unsafe compliance. Among the tested methods, RISA also achieves the highest CRR in every harmful model–dataset

The benign-sensitive results are mixed. On OR-Bench, ORR decreases for Qwen2.5-3B and Qwen3-4B but increases from 0.04 to 0.08 for Llama-3.2-3B. On XSTest-b, ORR increases by 0.03–0.10 across the three models. SHR also varies by model and dataset. After Holm adjustment, the Qwen3-4B improvement on OR-Bench and the Llama-3.2-3B degradation on XSTest-b are statistically significant. The external baselines show the same trade-off between safety on harmful prompts and behavior on benign-sensitive prompts. Methods that strongly reduce ORR often lose CRR

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

11

TABLE VIII U TILITY PRESERVATION BY MODEL AND TASK . ACCURACY IS REPORTED ON THE [0, 1] SCALE AND ROUNDED TO TWO DECIMAL PLACES . T HE FINAL COLUMN REPORTS THE NUMBER OF RESPONSES CHANGED BY RISA, AND THE RISA ACCURACY COLUMN IS SHADED .

Model

Task

Base accuracy RISA accuracy Interventions

Qwen2.5-3B

MMLU-STEM GSM8K

0.60 0.61

0.59 0.61

47 0

Llama-3.2-3B

MMLU-STEM GSM8K

0.52 0.77

0.52 0.77

5 0

Qwen3-4B

MMLU-STEM GSM8K

0.72 0.80

0.72 0.80

2 0

on harmful prompts, while methods that increase refusal can raise ORR. RQ1 therefore has a clear but asymmetric answer: RISA consistently improves refusal on harmful prompts, but it does not improve benign-sensitive behavior in every setting. C. RQ2: General Utility Preservation We evaluate the frozen RISA configuration on 3,153 MMLU-STEM prompts and 1,319 GSM8K prompts per model. No utility prompt is used to fit the rules, probe, support gate, or thresholds. The Base and RISA conditions use the same deterministic initial responses. The five-shot evaluation uses each model’s chat template. MMLU-STEM selects the highest-likelihood option, whereas GSM8K uses exact match after numeric-answer extraction. Table VIII reports the utility results and the number of interventions for each model and task. RISA does not alter any GSM8K responses, so accuracy is identical to the base model across all three models. On MMLU-STEM, refusal enforcement changes 47, 5, and 2 responses, corresponding to 1.49%, 0.16%, and 0.06% of the evaluation dataset. Accuracy decreases by 0.008 for Qwen2.53B and by 0.001 for the other two models. RISA therefore preserves most general utility, although it is not entirely costfree on MMLU-STEM. D. RQ3: Verifier Reliability and Generalization 1) RQ3.1: Training-Data Scaling: Table IX reports the exact mean and standard deviation for each training fraction, while Fig. 4 highlights the corresponding scaling trends. We use 10%, 25%, 50%, 75%, or 100% of the 1,188 training prompts. The development and evaluation datasets remain fixed, and each fraction uses five stratified subsampling seeds. The threshold τrefuse = 0.70 is used only for this diagnostic. Harmful coverage is the fraction of harmful prompts with scores at or above this threshold. Most gains occur before the 50% training fraction. With the full training dataset, AUROC reaches 0.98–0.99, the Brier score falls to 0.04–0.06, and harmful coverage reaches 0.94– 0.97. Benign false refusal also decreases overall. For Qwen34B, it falls from 0.24 to 0.09. Larger fractions yield smaller improvements and lower variation across seeds. The verifier therefore works well with limited training data, although the full training dataset yields the most stable overall results.

2) RQ3.2: Probe Score Quality: Table X reports discrimination and calibration metrics on the separate 800-prompt safety evaluation dataset. AUROC and AUPRC assess how well the probe ranks harmful prompts above benign-sensitive prompts. Brier score and expected calibration error (ECE) assess calibration. Cohen’s d quantifies the separation between the two score distributions. All three probes perform well on the evaluation prompts. AUROC is 0.98–0.99, AUPRC is 0.99, and calibration errors are low. Llama-3.2-3B delivers the strongest results overall, including the largest score separation (d = 4.25). Fig. 5 shows partial separation in the PCA plots, but the two prompt classes still overlap. The calibrated scores in Fig. 6 are more clearly separated. Benign-sensitive prompts tend to receive low scores, whereas harmful prompts tend to receive high scores. Prompts in the no-action interval retain their initial responses. These results show that the probes generalise well to the separate safety evaluation dataset, while uncertain scores do not trigger an action. 3) RQ3.3: Layer-Wise Robustness Check: Table XI reports the probe evaluation and end-to-end refusal-calibration results at four Transformer layers. These include the final layer used in the main results and three pre-defined earlier layers at approximately 25%, 50%, and 75% of model depth. The layers are 9/18/27/36 for Qwen2.5-3B, 7/14/21/28 for Llama3.2-3B, and 9/18/27/36 for Qwen3-4B. Each layer receives an independently trained probe, score calibration, support threshold, and action-threshold pair before being evaluated through the full RISA pipeline. No evaluation result is used to select a layer. Probe discrimination improves markedly beyond the earliest layer, but it does not always increase with depth. For Qwen2.53B, AUROC rises from 0.962 at layer 9 to 0.988 at layer 27, then falls slightly to 0.982 at layer 36. Llama-3.2-3B reaches 0.991 AUROC at layer 14 and remains at this level through layer 28. Qwen3-4B peaks at layer 18 with 0.990 AUROC and 0.994 AUPRC. Information related to refusal can therefore be easier to separate before the final Transformer layer. The end-to-end results show a second pattern. No single layer is best for every dataset. For Qwen2.5-3B, layer 27 gives the highest CRR on XSTest-h and Do-Not-Answer and the lowest ORR on XSTest-b, while layer 9 gives the highest HarmBench CRR and the lowest OR-Bench ORR. For Llama3.2-3B, the final Transformer layer gives the highest XSTest-h CRR and ties the best HarmBench and Do-Not-Answer CRR. However, its XSTest-b ORR is higher than at layers 14 and 21, and its OR-Bench ORR is higher than at layer 21. For Qwen34B, layer 18 gives the highest XSTest-h and Do-Not-Answer CRR and ties the highest HarmBench CRR, whereas layer 36 gives the lowest ORR on both benign-sensitive datasets. These results show that probe discrimination and end-toend refusal outcomes describe different parts of the pipeline. AUROC and AUPRC show how well the probe scores rank harmful prompts above benign-sensitive prompts. The end-toend metrics also depend on score calibration, support check outcomes, thresholds for each model, the initial response, and regeneration. The final Transformer layers used in the main configuration remain fixed because they were specified before

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

12

TABLE IX E FFECT OF PROBE TRAINING DATA SIZE BY MODEL . VALUES ARE MEAN ± STANDARD DEVIATION OVER FIVE NESTED SUBSAMPLING SEEDS . H ARMFUL COVERAGE AND BENIGN FALSE REFUSAL USE THE COMMON DIAGNOSTIC THRESHOLD τrefuse = 0.70. Fraction

Train n

AUROC↑

Brier↓

Harmful coverage↑

Benign false refusal↓

Qwen2.5-3B

0.10 0.25 0.50 0.75 1.00

119 296 594 892 1188

0.91 ±0.01 0.96 ±0.01 0.97 ±0.01 0.98 ±0.00 0.98 ±0.00

0.12 ±0.01 0.08 ±0.01 0.07 ±0.01 0.06 ±0.01 0.06 ±0.00

0.85 ±0.03 0.90 ±0.01 0.93 ±0.01 0.95 ±0.01 0.96 ±0.00

0.21 ±0.04 0.13 ±0.03 0.12 ±0.03 0.11 ±0.02 0.10 ±0.00

Llama-3.2-3B

0.10 0.25 0.50 0.75 1.00

119 296 594 892 1188

0.96 ±0.01 0.98 ±0.00 0.99 ±0.00 0.99 ±0.00 0.99 ±0.00

0.07 ±0.01 0.05 ±0.00 0.05 ±0.00 0.04 ±0.00 0.04 ±0.00

0.88 ±0.02 0.93 ±0.01 0.95 ±0.01 0.96 ±0.00 0.97 ±0.00

0.09 ±0.01 0.06 ±0.01 0.07 ±0.01 0.06 ±0.01 0.07 ±0.00

Qwen3-4B

0.10 0.25 0.50 0.75 1.00

119 296 594 892 1188

0.92 ±0.01 0.95 ±0.01 0.97 ±0.00 0.97 ±0.00 0.98 ±0.00

0.12 ±0.01 0.09 ±0.01 0.08 ±0.00 0.07 ±0.00 0.06 ±0.00

0.90 ±0.01 0.93 ±0.01 0.94 ±0.01 0.94 ±0.01 0.94 ±0.00

0.24 ±0.04 0.17 ±0.05 0.13 ±0.01 0.11 ±0.02 0.09 ±0.00

Model

Qwen2.5-3B

Qwen2.5-3B

Brier score ↓

AUROC ↑

0.96

0.93 Qwen2.5-3B

0.90 200

600

Qwen3-4B

0.09

0.06

Harmful coverage ↑ (%)

Llama-3.2-3B

0.12

Llama-3.2-3B

95

Benign FPR ↓ (%)

0.99

90

85 Llama-3.2-3B

Qwen3-4B

Qwen3-4B

1000

200

600

200

1000

600

8

1000

200

Probe training examples

Probe training examples

(a) AUROC.

16

Qwen2.5-3B

Llama-3.2-3B

Probe training examples

Qwen3-4B

24

(b) Brier score.

600

1000

Probe training examples

(c) Harmful coverage.

(d) Benign false refusal rate.

Fig. 4. Probe results with different training sizes on the fixed 800-prompt safety evaluation. Lines show means over five nested stratified subsampling seeds and shaded regions show one standard deviation. Harmful coverage and benign false refusal rate use the common diagnostic threshold τrefuse = 0.70.

Harmful (n=500)

Benign-sensitive (n=300)

PC2 (5.1% train variance)

PC2 (6.5% train variance)

40 20 0 20 60

40 20 0 20 40 PC1 (23.7% train variance)

60

(a) Qwen2.5-3B.

Harmful (n=500)

40 30 20 10 0 10 20

Benign-sensitive (n=300)

PC2 (8.2% train variance)

Benign-sensitive (n=300)

60

40

20 0 20 PC1 (18.3% train variance)

(b) Llama-3.2-3B.

40

Harmful (n=500)

60 40 20 0 20 40 80

60

40 20 0 20 40 PC1 (18.3% train variance)

60

(c) Qwen3-4B.

Fig. 5. Evaluation prompt hidden states projected into a two-dimensional PCA basis fitted only on the 1,188 probe training prompts. Each panel contains the same 300 benign-sensitive and 500 harmful evaluation prompts; the evaluation split is transformed but never used to fit PCA.

evaluation, not because they are best for every dataset. RQ3.3 shows that useful information related to refusal is available at several layers, but a layer should not be selected based on probe quality alone. E. RQ4: Component and Policy Contributions 1) RQ4.1: Component Ablation: Table XII compares full RISA with four ablated variants. The rules-only variant omits the probe fallback. The probe-only variant omits rule rout-

ing. The regeneration-only variant retains only safety-guided regeneration, while the enforcement-only variant retains only refusal enforcement. All variants reuse the same initial responses, so differences arise from routing and action selection. The probe provides most of the coverage for harmful prompts. The probe-only variant matches full RISA in eight of the nine harmful model–dataset settings. The rules-only variant depends more on the dataset. For Qwen3-4B, its CRR is 0.94 on HarmBench but 0.55 on Do-Not-Answer, where few rules

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

50 0.0

0.2 0.4 0.6 0.8 Calibrated refusal-support score

1.0

0.0

0.2 0.4 0.6 0.8 Calibrated refusal-support score

(a) Qwen2.5-3B

(b) Llama-3.2-3B

1.0

Harmful refuse = 0.70

300

100 0

allow = 0.57

Benign-sensitive

Prompt count

100

200

Harmful

refuse = 0.69

300

150

0

allow = 0.57

200

Benign-sensitive

Prompt count

allow = 0.39

Prompt count

250

Harmful refuse = 0.63

Benign-sensitive

300

13

200 100 0

0.0

0.2 0.4 0.6 0.8 Calibrated refusal-support score

1.0

(c) Qwen3-4B

Fig. 6. Calibrated refusal scores on the safety evaluation dataset. Blue bars denote benign-sensitive prompts and vermilion bars denote harmful prompts. Dashed and dotted vertical lines mark the allow and refusal thresholds for each model. Each panel contains the same 300 benign-sensitive and 500 harmful prompts.

TABLE X H IDDEN STATE PROBE QUALITY ON THE SAFETY EVALUATION DATASET BY MODEL . H IGHER IS BETTER FOR AUROC, AUPRC, AND C OHEN ’ S d; LOWER IS BETTER FOR B RIER SCORE AND ECE. Model Qwen2.5-3B Llama-3.2-3B Qwen3-4B

AUROC↑ AUPRC↑ Brier↓ ECE↓ d ↑ 0.98 0.99 0.98

0.99 0.99 0.99

0.06 0.04 0.07

0.07 0.05 0.06

3.38 4.25 3.18

match. The rules handle clear cases, while the probe provides broader coverage. The two corrective actions serve different roles. The enforcement-only variant matches full RISA in eight harmful settings and is slightly higher in the remaining one. The regeneration-only variant stays close to the base model on harmful prompts but yields the lowest ORR on both benignsensitive datasets. Refusal enforcement therefore drives gains on harmful prompts, while safety-guided regeneration drives most reductions in over-refusal. No single action handles both goals: enforcement can add unnecessary refusals, and regeneration does not always reverse an initial refusal. 2) RQ4.2: Runtime Selectivity and Routing: Fig. 7 shows the distribution of the four final actions across datasets. The two corrective actions are used in different settings. Safety-guided regeneration appears mainly on benign-sensitive prompts and reaches 0.42 for Qwen3-4B on OR-Bench. It remains at or below 0.02 across all harmful datasets. Refusal enforcement is more common on harmful prompts and reaches 0.61 for Llama-3.2-3B on Do-Not-Answer. RISA therefore selects actions based on the current prompt and response rather than applying a global shift towards refusal. Table XIII shows which verifier component handles each prompt. Rule match rates are the same across models because the rules depend only on prompt text. Probe fallback and final actions remain model-dependent. The rules cover 0.31 of HarmBench prompts but only 0.02 of Do-Not-Answer prompts. Probe fallback therefore reaches 0.96–0.98 on Do-Not-Answer. Abstentions are absent in most harmful settings and occur most often on XSTest-b, at 0.04– 0.09. These results show a clear division of work: rules handle

a small collection of explicit cases, the probe handles most remaining safety prompts, and the support gate leaves out-ofsupport cases unchanged. 3) RQ4.3: Action-Selection Reliability: Table XIV evaluates action selection separately according to the base model’s initial refusal status. For an initial non-refusal, RISA should enforce refusal on harmful prompts and leave benign answers unchanged. For an initial refusal, RISA should select regeneration for benign prompts and leave harmful refusals unchanged. We report Wilson 95% confidence intervals because some status-specific subsets contain few samples. Action selection for initial non-refusals is generally reliable. For benign prompts that already receive an answer, its accuracy ranges from 0.88 to 0.99, indicating that RISA adds few unnecessary refusals. For harmful prompts that initially receive an answer, accuracy reaches 0.96–0.98 on HarmBench. The lowest harmful accuracy is 0.74 for Qwen3-4B on XSTest-h. Even in this case, refusal enforcement corrects 20 of the 27 initial non-refusals and contributes to the CRR improvement reported in RQ1. Action selection for initial refusals is less consistent on benign prompts. Its accuracy ranges from 0.42 to 0.79 on ORBench and from 0.00 to 0.30 on XSTest-b. The Llama-3.2-3B result on XSTest-b is based on only two initial refusals and is therefore uncertain. In contrast, RISA preserves harmful refusals with an accuracy of 0.97–1.00 because regeneration on harmful prompts is rare. These results indicate whether RISA selects the intended action. Final correction of a benign over-refusal also depends on whether regeneration produces a non-refusal response. 4) RQ4.4: Threshold Sensitivity: We vary τrefuse ∈ {0.70, 0.80, 0.90, 0.95} while holding τallow = 0.15. This sweep was defined in advance as a diagnostic and was not used to derive or select the calibrated thresholds. Table XV reports the exact results for each model and dataset, while Fig. 8 summarizes the trends. As τrefuse increases, refusal enforcement is triggered less often. Harmful CRR therefore never increases and generally decreases. Do-Not-Answer is the most sensitive dataset, with CRR reductions of 0.10–0.17 across the three models. The largest reduction occurs for Llama-3.2-3B, from 0.93 to 0.76.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

14

TABLE XI L AYER - WISE ROBUSTNESS CHECK BY MODEL AND DATASET. L AYERS ARE LISTED FROM SHALLOW TO DEEP. AUROC AND AUPRC MEASURE PROBE DISCRIMINATION ON THE EVALUATION DATASET. H ARMFUL DATASETS REPORT CRR/ASR, AND BENIGN - SENSITIVE DATASETS REPORT ORR/SHR AFTER THE CORRESPONDING PROBE IS INTEGRATED INTO THE FULL RISA PIPELINE . B OLD MARKS THE BEST VALUE AMONG THE FOUR LAYERS WITHIN EACH MODEL AND METRIC . Transformer layer

Model

Probe quality

XSTest-h

HarmBench Do-Not-Answer

AUROC↑ AUPRC↑ CRR↑ ASR↓ CRR↑ ASR↓ CRR↑

OR-Bench

XSTest-b

ASR↓

ORR↓ SHR↑ ORR↓ SHR↑

Qwen2.5-3B

9 18 27 36

0.962 0.976 0.988 0.982

0.977 0.985 0.993 0.990

0.82 0.89 0.95 0.93

0.01 0.01 0.01 0.01

0.91 0.89 0.89 0.90

0.07 0.08 0.08 0.07

0.71 0.83 0.89 0.86

0.00 0.00 0.00 0.00

0.19 0.21 0.20 0.24

0.53 0.51 0.53 0.51

0.14 0.19 0.12 0.15

0.83 0.78 0.86 0.82

Llama-3.2-3B

7 14 21 28

0.933 0.991 0.991 0.991

0.961 0.995 0.995 0.995

0.85 0.94 0.94 0.96

0.05 0.01 0.01 0.01

0.98 0.96 0.98 0.98

0.02 0.03 0.02 0.02

0.66 0.94 0.91 0.94

0.01 0.01 0.00 0.00

0.03 0.08 0.05 0.08

0.59 0.56 0.58 0.56

0.10 0.09 0.09 0.14

0.88 0.89 0.89 0.85

Qwen3-4B

9 18 27 36

0.963 0.990 0.974 0.977

0.976 0.994 0.986 0.988

0.81 0.98 0.97 0.93

0.00 0.00 0.00 0.00

0.96 0.96 0.95 0.95

0.00 0.00 0.01 0.01

0.77 0.95 0.91 0.92

0.00 0.00 0.00 0.00

0.47 0.47 0.48 0.46

0.45 0.45 0.44 0.46

0.16 0.14 0.17 0.13

0.82 0.85 0.82 0.85

TABLE XII C OMPONENT ABLATION BY MODEL AND DATASET. H ARMFUL SOURCES REPORT CRR AND THE MEAN ASR FROM TWO SAFETY JUDGES ; BENIGN - SENSITIVE SOURCES REPORT ORR AND GPT-OSS SHR. B OLD MARKS THE BEST VALUE WITHIN EACH MODEL – DATASET COLUMN . XSTest-h Variant

CRR↑ ASR↓ CRR↑ ASR↓ CRR↑

Qwen2.5-3B

Rules only Probe only Regeneration Enforcement RISA

0.81 0.93 0.81 0.93 0.93

0.01 0.01 0.01 0.01 0.01

0.82 0.90 0.68 0.90 0.90

0.10 0.07 0.20 0.07 0.07

Rules only Probe only Llama-3.2-3B Regeneration Enforcement RISA

0.83 0.96 0.81 0.96 0.96

0.05 0.01 0.06 0.01 0.01

0.92 0.96 0.66 0.99 0.98

Rules only Probe only Regeneration Enforcement RISA

0.79 0.93 0.78 0.93 0.93

0.00 0.00 0.00 0.00 0.00

0.94 0.95 0.77 0.95 0.95

Qwen3-4B

Keep ans. Keep ref.

Enforce Safe regen.

OR-Bench

ORR↓ SHR↑ ORR↓ SHR↑

0.63 0.86 0.63 0.86 0.86

0.00 0.00 0.00 0.00 0.00

0.26 0.26 0.23 0.27 0.24

0.50 0.50 0.52 0.49 0.51

0.14 0.12 0.10 0.17 0.15

0.83 0.85 0.87 0.80 0.82

0.06 0.04 0.29 0.01 0.02

0.50 0.94 0.50 0.94 0.94

0.03 0.00 0.03 0.00 0.00

0.05 0.07 0.03 0.09 0.08

0.58 0.57 0.59 0.56 0.56

0.09 0.09 0.04 0.14 0.14

0.89 0.89 0.93 0.85 0.85

0.01 0.01 0.06 0.01 0.01

0.55 0.92 0.55 0.92 0.92

0.00 0.00 0.00 0.00 0.00

0.53 0.48 0.45 0.56 0.46

0.39 0.44 0.47 0.36 0.46

0.10 0.11 0.08 0.12 0.13

0.88 0.87 0.90 0.86 0.85

Enforce Safe regen.

Keep ans. Keep ref.

100

100

75

75

75

25

0

Action rate (%)

100

50

50

25

0 OR-B

XST-b XST-h

(a) Qwen2.5-3B.

Harm

DNA

XSTest-b

ASR↓

Keep ans. Keep ref.

Action rate (%)

Action rate (%)

HarmBench Do-Not-Answer

Model

Enforce Safe regen.

50

25

0 OR-B

XST-b XST-h

(b) Llama-3.2-3B.

Harm

DNA

OR-B

XST-b XST-h

Harm

DNA

(c) Qwen3-4B.

Fig. 7. Runtime action selectivity by model and source. XSTest is split into benign-sensitive (XST-b) and harmful (XST-h) subsets. Colors show the four actions, and the dashed line separates benign-sensitive from harmful sources.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

15

TABLE XIII ROUTING RESPONSIBILITY AND CORRECTIVE ACTIONS BY MODEL AND DATASET. P ROBE DENOTES IN - SUPPORT PROBE FALLBACK ; ABSTENTION DENOTES OUT- OF - SUPPORT CASES PRESERVED WITHOUT INTERVENTION . Model

Dataset

Rule match

Probe fallback

Abstention

Enforce refusal

Guided regeneration

Qwen2.5-3B

OR-Bench XSTest-b XSTest-h HarmBench Do-Not-Answer

0.16 0.05 0.11 0.31 0.02

0.84 0.86 0.89 0.69 0.96

0.01 0.09 0.00 0.00 0.03

0.01 0.05 0.26 0.27 0.28

0.11 0.06 0.00 0.01 0.01

Llama-3.2-3B

OR-Bench XSTest-b XSTest-h HarmBench Do-Not-Answer

0.16 0.05 0.11 0.31 0.02

0.85 0.91 0.89 0.69 0.98

0.00 0.04 0.00 0.00 0.00

0.05 0.12 0.19 0.46 0.61

0.02 0.00 0.00 0.01 0.00

Qwen3-4B

OR-Bench XSTest-b XSTest-h HarmBench Do-Not-Answer

0.16 0.05 0.11 0.31 0.02

0.84 0.87 0.88 0.69 0.98

0.01 0.08 0.01 0.00 0.00

0.02 0.05 0.20 0.23 0.53

0.42 0.01 0.02 0.02 0.01

TABLE XIV R ELIABILITY OF ACTION SELECTION CONDITIONED ON THE INITIAL REFUSAL STATUS . ACC . IS ACTION - SELECTION ACCURACY, CI IS THE W ILSON 95% INTERVAL , AND TP/FP REPORT CORRECT AND INCORRECT INTERVENTION COUNTS . C ASES WITHOUT INTERVENTION AFFECT ACC . BUT ARE EXCLUDED FROM TP/FP. Initial non-refusal Model

Dataset

Type

n

Acc.

95% CI

TP/FP

n

Acc.

95% CI

Qwen2.5-3B

OR-Bench XSTest-b XSTest-h HarmBench Do-Not-Answer

benign 148 benign 80 harmful 31 harmful 55 harmful 67

0.99 0.94 0.84 0.96 0.82

[0.95, 1.00] [0.86, 0.97] [0.67, 0.93] [0.88, 0.99] [0.71, 0.89]

0/2 0/5 26 / 0 53 / 0 55 / 0

52 20 69 145 133

0.42 0.30 1.00 0.99 0.99

[0.30, 0.56] 22 / 0 [0.15, 0.52] 6 / 0 [0.95, 1.00] 0 / 0 [0.96, 1.00] 0 / 1 [0.96, 1.00] 0 / 1

OR-Bench XSTest-b Llama-3.2-3B XSTest-h HarmBench Do-Not-Answer

benign 192 benign 98 harmful 22 harmful 94 harmful 130

0.95 0.88 0.86 0.98 0.93

[0.91, 0.97] 0 / 10 [0.80, 0.93] 0 / 12 [0.67, 0.95] 19 / 0 [0.93, 0.99] 92 / 0 [0.87, 0.96] 121 / 0

8 2 78 106 70

0.50 0.00 1.00 0.98 1.00

[0.22, 0.78] [0.00, 0.66] [0.95, 1.00] [0.93, 0.99] [0.95, 1.00]

OR-Bench XSTest-b XSTest-h HarmBench Do-Not-Answer

benign 93 benign 93 harmful 27 harmful 48 harmful 119

0.97 0.95 0.74 0.96 0.88

[0.91, 0.99] 0 / 3 [0.88, 0.98] 0 / 5 [0.55, 0.87] 20 / 0 [0.86, 0.99] 46 / 0 [0.81, 0.93] 105 / 0

107 7 73 152 81

0.79 0.14 0.97 0.98 0.99

[0.70, 0.85] 84 / 0 [0.03, 0.51] 1 / 0 [0.91, 0.99] 0 / 2 [0.94, 0.99] 0 / 3 [0.93, 1.00] 0 / 1

Qwen3-4B

OR-B ORR XST-b ORR XST-h CRR

Harm CRR DNA CRR

OR-B ORR XST-b ORR XST-h CRR

Harm CRR DNA CRR

50 25 0

75 50 25 0

0.7

0.8

0.9

0.95

4/0 0/0 0/0 0/2 0/0

Harm CRR DNA CRR

100

Refusal rate (%)

75

TP/FP

OR-B ORR XST-b ORR XST-h CRR

100

Refusal rate (%)

100

Refusal rate (%)

Initial refusal

75 50 25 0

0.7

0.8

0.9

0.95

0.7

0.8

τrefuse

τrefuse

τrefuse

(a) Qwen2.5-3B.

(b) Llama-3.2-3B.

(c) Qwen3-4B.

0.9

0.95

Fig. 8. Sensitivity to the refusal threshold with τallow = 0.15. Solid lines show CRR on harmful prompts, and dashed lines show benign-sensitive ORR. This diagnostic sweep does not determine the calibrated thresholds for each model.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

16

TABLE XV S ENSITIVITY TO THE REFUSAL THRESHOLD BY MODEL AND DATASET WITH τallow = 0.15. H ARMFUL SOURCES REPORT CRR AND THE MEAN ASR FROM TWO SAFETY JUDGES ; BENIGN - SENSITIVE SOURCES REPORT ORR AND GPT-OSS SHR. XSTest-h

HarmBench Do-Not-Answer

Model

τrefuse CRR↑ ASR↓ CRR↑ ASR↓ CRR↑

Qwen2.5-3B

0.70 0.80 0.90 0.95

0.93 0.93 0.91 0.86

0.00 0.00 0.00 0.00

0.90 0.89 0.87 0.84

0.07 0.08 0.08 0.09

Llama-3.2-3B

0.70 0.80 0.90 0.95

0.96 0.92 0.88 0.86

0.01 0.02 0.03 0.03

0.99 0.98 0.97 0.96

Qwen3-4B

0.70 0.80 0.90 0.95

0.93 0.92 0.91 0.89

0.00 0.00 0.00 0.00

0.96 0.96 0.96 0.96

OR-Bench

XSTest-b

ASR↓

ORR↓ SHR↑ ORR↓ SHR↑

0.83 0.81 0.79 0.73

0.00 0.00 0.00 0.00

0.26 0.26 0.25 0.25

0.51 0.51 0.51 0.51

0.15 0.15 0.14 0.14

0.83 0.83 0.84 0.84

0.01 0.02 0.02 0.04

0.93 0.90 0.86 0.76

0.00 0.01 0.01 0.01

0.08 0.07 0.05 0.05

0.56 0.57 0.58 0.58

0.14 0.11 0.11 0.11

0.86 0.89 0.89 0.89

0.01 0.01 0.01 0.01

0.93 0.90 0.88 0.81

0.00 0.00 0.00 0.00

0.54 0.53 0.53 0.53

0.40 0.41 0.41 0.41

0.15 0.14 0.13 0.13

0.85 0.86 0.87 0.87

HarmBench changes less, and its CRR for Qwen3-4B remains at 0.96. Gains on benign-sensitive prompts are smaller. Across ORBench, the largest reduction is 0.03 for Llama-3.2-3B, and the largest XSTest-b reduction is also 0.03. These gains come with weaker protection on harmful prompts. For example, Qwen2.5-3B HarmBench ASR rises from 0.07 to 0.09. The sweep therefore shows the expected trade-off: a higher refusal threshold can slightly reduce over-refusal, but it can also reduce safety on harmful prompts. This result also explains why a single manually chosen threshold cannot replace the calibrated threshold for each model. F. Runtime Generation Cost Table XVI reports the generation runtime cost for 400 safety prompts per model. The base condition uses a single batched generation pass. For RISA, we reuse the loaded model and the precomputed prompt scores. We make a second generation call only for safety-guided regeneration. This measurement therefore isolates the generation overhead after the prompt scores are available. Latency overhead scales with the regeneration rate. Qwen34B has the highest regeneration rate at 0.11 and the highest relative latency increase at 0.10. Llama-3.2-3B regenerates only 0.01 of prompts, so its added latency is close to zero. Output length follows the same pattern and increases most for Qwen3-4B. Peak GPU memory changes by at most 2 MiB. Safety-guided regeneration is therefore the main source of additional generation cost. VI. C ONCLUSION This paper presents RISA as a runtime verification framework for LLM refusal calibration. Rather than treating refusal behavior as a fixed property of the base model or changing all generations uniformly, RISA inspects the model’s initial response, compares its refusal status with a refusal score for the prompt, and decides whether to preserve the response, enforce refusal, or regenerate. The framework combines selective rules

with a lightweight hidden-state probe. The rules handle clear prompt patterns, while the probe addresses cases the rules do not resolve. Its main design choice is an asymmetric correction policy. Harmful non-refusals can be replaced with a fixed refusal. Benign over-refusals may be corrected through safetyguided regeneration, but the base model still controls the new answer. Experiments across three instruction-tuned models and multiple safety benchmarks support this design. RISA consistently improves refusal reliability on harmful prompts, while its behavior on benign-sensitive prompts varies across data sources. General utility is largely preserved: GSM8K responses remain unchanged, and MMLU-STEM accuracy losses are small. These results suggest that refusal calibration is better treated as targeted runtime verification with asymmetric action selection than as a single global change in model cautiousness. R EFERENCES [1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901. [2] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [3] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35. Curran Associates, Inc., 2022, pp. 27 730–27 744. [4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022. [5] J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “OR-Bench: An overrefusal benchmark for large language models,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 13–19 Jul 2025, pp. 11 515–11 542. [6] P. R”ottger, H. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 5377–5400.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, MONTH 2026

17

TABLE XVI G ENERATION - STAGE RUNTIME COST BY MODEL . L ATENCY IS MEAN WALL - CLOCK TIME PER PROMPT. ∆ LATENCY REPORTS ADDED MILLISECONDS , WITH THE RELATIVE INCREASE IN PARENTHESES . R EGEN . IS THE REGENERATION RATE , TOKENS ARE MEAN TOTAL OUTPUT TOKENS , AND GPU ∆ IS THE CHANGE IN PEAK ALLOCATED MEMORY. Model

n

Qwen2.5-3B 400 Llama-3.2-3B 400 Qwen3-4B 400

Base (ms) RISA (ms) 763.76 661.45 896.02

797.26 663.90 983.32

∆ latency

Regen. Base tokens RISA tokens GPU ∆ (MiB)

+33.50 (+0.04) +2.44 (+0.00) +87.30 (+0.10)

[7] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,” in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 235. PMLR, 21–27 Jul 2024, pp. 35 181–35 224. [8] Y. Wang, H. Li, X. Han, P. Nakov, and T. Baldwin, “Do-not-answer: Evaluating safeguards in LLMs,” in Findings of the Association for Computational Linguistics: EACL 2024. St. Julian’s, Malta: Association for Computational Linguistics, Mar. 2024, pp. 896–911. [9] F. Bianchi, M. Suzgun, G. Attanasio, P. Röttger, D. Jurafsky, T. Hashimoto, and J. Zou, “Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions,” in The Twelfth International Conference on Learning Representations, 2024. [10] J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang, “Safe RLHF: Safe reinforcement learning from human feedback,” in The Twelfth International Conference on Learning Representations, 2024. [11] N. Meade, S. Gella, D. Hazarika, P. Gupta, D. Jin, S. Reddy, Y. Liu, and D. Hakkani-Tur, “Using in-context learning to improve dialogue safety,” in Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 11 882–11 910. [12] Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran, “SafeDecoding: Defending against jailbreak attacks via safety-aware decoding,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 5587–5605. [13] S. Banerjee, S. Layek, S. Tripathy, S. Kumar, A. Mukherjee, and R. Hazra, “SafeInfer: Context adaptive decoding time safety alignment for large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 26, 2025, pp. 27 188–27 196. [14] J. Zhao, J. Huang, Z. Wu, D. Bau, and W. Shi, “LLMs encode harmfulness and refusal separately,” in The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025. [15] A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024, pp. 136 037–136 083. [16] A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid, “Steering language models with activation engineering,” arXiv preprint arXiv:2308.10248, 2023. [17] Z. Sun, Y. Shen, H. Zhang, Q. Zhou, Z. Chen, D. Cox, Y. Yang, and C. Gan, “SALMON: Self-alignment with instructable reward models,” in The Twelfth International Conference on Learning Representations, 2024. [18] C. Shi, X. Wang, Q. Ge, S. Gao, X. Yang, T. Gui, Q. Zhang, X. Huang, X. Zhao, and D. Lin, “Navigating the overkill in large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 4602–4614. [19] N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner, “Steering Llama 2 via contrastive activation addition,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 15 504–15 522. [20] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa, “Llama Guard: LLM-based input-output safeguard for human-AI conversations,” arXiv preprint arXiv:2312.06674, 2023.

0.05 0.01 0.11

217.95 212.71 196.31

230.28 215.18 223.77

2.00 0.00 0.00

[21] OpenAI, “Moderation,” OpenAI API documentation, 2026, accessed: 2026-07-26. [22] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen, “NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 431–445. [23] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30. Curran Associates, Inc., 2017, pp. 4299–4307. [24] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” arXiv preprint arXiv:1909.08593, 2019. [25] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan et al., “A general language assistant as a laboratory for alignment,” arXiv preprint arXiv:2112.00861, 2021. [26] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36. Curran Associates, Inc., 2023, pp. 53 728– 53 741. [27] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, 2022. [28] J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang, “BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36. Curran Associates, Inc., 2023, pp. 24 678–24 704. [29] M. Dabas, S. Chen, C. Fleming, M. Jin, and R. Jia, “Just enough shifts: Mitigating over-refusal in aligned language models with targeted representation fine-tuning,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 13–19 Jul 2025, pp. 11 846–11 861. [30] Y. Qi, Z. Lyu, L. Cui, L. Bai, and F. Xia, “Please refuse to answer me! mitigating over-refusal in large language models via adaptive contrastive decoding,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). San Diego, California, United States: Association for Computational Linguistics, Jul. 2026, pp. 39 308–39 325. [31] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren et al., “Representation engineering: A top-down approach to AI transparency,” arXiv preprint arXiv:2310.01405, 2023.

Related documents

• RISA: Response Inspection and Selective Actions for Refusal Calibration in Large Language Models#820800
• Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models#405700
• Refusal without Discrimination: What Encoded Prompts Do to Safety-Trained Models#1062077
• TOWARDS EXPLAINING, EVALUATING, AND MITIGATING CULTURAL BIAS IN LARGE LANGUAGE MODELS#637484
• Auditing large language models: a three-layered approach#335808