Adaptive and Explicit <safe>: Triggering Latent Safety Awareness in Large Reasoning Models Ke Miao1,2† , Jiaxin Li3,4† , Hongliang Chen3
arXiv:2606.16808v1 [cs.AI] 15 Jun 2026
1
, Yuke Hu1,2,5 , and Zhan Qin1,2
The State Key Laboratory of Blockchain and Data Security, Zhejiang University {miaoke,yukehu,qinzhan}@zju.edu.cn 2 Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China 3 Li Auto Inc. {chenhongliang,lijiaxin13}@lixiang.com 4 Tsinghua University 5 King Abdullah University Of Science And Technology [email protected]
Abstract. While Large Reasoning Models (LRMs) excel at complex tasks, they remain highly vulnerable to sophisticated jailbreaks and direct harmful queries. To address this vulnerability, prior works depend heavily on external manual data annotation for safety alignment. However, we observe that LRMs can inherently identify safety risks when being re-presented with original queries alongside their own reasoning trajectories—a capability we term Latent Safety Awareness. To leverage this safety awareness, we first employ Supervised Fine-Tuning (SFT) to explicitly induce <safe> tags to trigger safety analysis and guidance following the initial reasoning content for unsafe queries, while preserving standard responses for general queries to ensure adaptive triggering. Subsequently, we apply Direct Preference Optimization (DPO) to further enhance the correctness and stability of the safety analysis and guidance. Notably, responses required for both training stages are entirely generated by models being optimized. With (Safe Trigger) SFT and DPO, experimental results demonstrate significant safety enhancement. For example, the Attack Success Rate (ASR) of DeepSeek-R1-Distill-Llama-8B, on average, drops 24.65% and 36.72% on harmful and jailbreak benchmarks, respectively. Finally, our Safe Trigger method exerts almost no negative impact on general performance or user experience. Keywords: Large reasoning model · Safety alignment · Large language model.
1
Introduction
Large Reasoning Models (LRMs) have recently demonstrated remarkable proficiency across both general and complex domains, acting as the major driver for new AI applications [8, 29, 36]. Despite their power, LRMs remain susceptible to generating harmful content with sophisticated jailbreaks and direct harmful queries [46, 17]. While safety alignment is a cornerstone of AI research, the †
Equal contribution.
Harmful
15 10 5 0
e h S 1 D Bas Star afePat ST- STS
30
Jailbreak General Capability
20
20 10 0
e h S 1 D Bas Star afePat ST- STS
0.80
General
0.75 0.70 0.65 0.60
e h S 1 D Bas Star afePat ST- STS
Over-Refusal Rate (%)
Ke Miao et al. Attack Success Rate (%)
Harmful Response Rate (%)
2
10 8 6 4 2 0
Over-Refusal
e h S 1 D Bas Star afePat ST- STS
Fig. 1: Performance of the model trained with our Safe Trigger approach. prevailing literature focuses on standard Large Language Models (LLMs) [44, 31, 45, 21, 25], whereas specialized methodologies for LRMs are relatively underexplored. Furthermore, prior works [41, 17, 15] are heavily reliant on resourceintensive external manual annotations and leave room for improving safety robustness (as shown in our result Table 3). In this paper, we uncover that LRMs possess the latent safety awareness capable of identifying safety risks when re-evaluating the original query alongside their own reasoning. However, this safety awareness often remains dormant during the standard generation process. Motivated by this observation, we propose Safe Trigger (Section 3), a method designed to explicitly elicit the model’s latent safety consciousness through a structured trigger mechanism (denoted as an italicized term <safe> in this work) utilizing <safe> tags. This approach strengthens the model’s robustness against adversarial jailbreaks and harmful queries while preserving its general capabilities and user experience. As illustrated in Figure 1, our models—ST-S and ST-D, after Safe Trigger SFT and DPO stages, respectively—demonstrate superior safety alignment without compromising task performance. The main contributions of this work can be summarized as follows: Our contributions are summarized as follows: (1) We reveal the Latent Safety Awareness phenomenon, where LRMs can identify safety issues by reviewing their own reasoning process even when an initial attack is successful. (2) We propose the Safe Trigger approach, utilizing Safe Trigger SFT to reliably activate a structured safety module and Safe Trigger DPO to robustly integrate safety constraints into the final output. (3) Extensive experiments demonstrate that our approach significantly strengthens model safety—increasing the average alignment rate of DeepSeek-R1-Distill-Llama-8B by 24.65% on harmful benchmarks and 36.72% on jailbreak benchmarks —while preserving general capabilities. (4) Our training pipeline is self-consistent, serving as its own teacher to construct signals without manual annotation or dependence on closed-source models, offering a low-cost, reproducible, and scalable solution for large-scale safety alignment.
2
The Latent Safety Awareness of LRMs
Vulnerability Observations. To demonstrate the limitations of current LRMs in safety alignment, we evaluate the safety of Qwen3-8B [38] and DeepSeek-R1Distill-Llama-8B [8], both of which serve as representative open-source LRMs.
Triggering Latent Safety Awareness in Large Reasoning Models
Table 1: Attack Success Rate (ASR) of different LRMs across harmful and jailbreak benchmarks when original queries are entered directly. Model
Harmful Jailbreak Advbench HexPHI XsTest WildJailbreak
Qwen3-8B 1.92% DeepSeek-8B 35.19%
13.33% 0.50% 44.00% 20.00%
42.80% 49.90%
3
Table 2: Risk Identification Success Rate (RISR) of different LRMs when re-evaluating original queries and their corresponding reasoning trajectories. Model
Harmful Jailbreak Advbench HexPHI XsTest WildJailbreak
Qwen3-8B 60.00% DeepSeek-8B 81.97%
75.00% 100.00% 69.70% 87.50%
58.29% 44.79%
Table 1 summarizes the Attack Success Rate (ASR) of these two models across four safety evaluation datasets, where ASR reflects the proportion of queries that can cause the models to generate unsafe content. LRMs exhibit pronounced vulnerabilities to generating harmful outputs, highlighting a critical deficiency in their safety alignment. The results in Table 1 show that Qwen3-8B already exhibits a certain probability of generating unsafe content in the context of harmful queries, with the highest attack success rate being 13.33% on the HexPHI dataset [32]. In the context of jailbreak queries, the probability increases sharply, reaching 42.80% for the WildJailbreak dataset [19]. DeepSeek-R1-Distill-Llama-8B suffers from a high ASR on all four benchmarks. Latent Safety Awareness. Despite the vulnerabilities identified above, our further experiments uncovered a compelling phenomenon: LRMs possess a latent capacity for safety risk identification that often remains dormant during standard generation. Specifically, we conducted a diagnostic study on the subset of queries that successfully bypassed the model’s safety guard to elicit harmful content. We re-presented the adversarial queries alongside the model’s self-generated reasoning trajectories to the model itself for a safety audit (quantified via safety risk identification), tasking it to assess the adequacy of its own safety considerations regarding the input and reasoning content. As shown in Table 2, we report the Risk Identification Success Rate (RISR), which quantifies the model’s ability to post-hoc identify safety risks. The fact that RISR ranges from 44.79% to 100% (consistently surpassing the ASR) suggests that LRMs possess a significant latent capacity to recognize safety risks within both the original queries and their own reasoning trajectories, which offers a promising foundation for safety alignment in our work. The above results demonstrate that when LRMs are presented with both the initial adversarial queries and their own reasoning paths, their capacity to identify core safety violations and internal vulnerabilities increases significantly. Under these conditions, the models can proactively recommend refusing to continue generating potentially harmful answers. We define this capability as the Latent Safety Awareness of LRMs. However, in the default reasoning process of existing LRMs, there is no safety reminder mechanism between the end of reasoning and the generation of the final response for risky queries. Even though the model has Latent Safety Awareness, it is often not effectively activated in the standard process. Based on the aforementioned findings, we propose the Safe
4
Ke Miao et al. Activating the Latent Safety Awareness General + Harmful + Jailbreak : query
①
<think>1 + final_answer1 Harmful + Jailbreak : query + <think>1
② <think>2 + <safe> + final_answer2
Enhancing the Latent Safety Awareness
Safe Trigger SFT Dataset G: query + <think>1 + final_answer1
Jailbreak Only : query
④ Base Model
③
H: query + <think>2 + <safe> + final_answer2
<think> <safe> final_answer
⑤
J: query + <think>2 + <safe> + final_answer2 ST-S Model
Fsafe + Sexist + Tfull + Sfull
Safe Trigger DPO Dataset query
chosen response
ST-S Model
⑥
rejected response
Reward Score Difference > δ ST-D Model
Fig. 2: Overview of our Safe Trigger method. The two-stage process activates and enhances the model’s Latent Safety Awareness. (a) Safe Trigger SFT: The model is trained on general, harmful, and jailbreak queries to adaptively elicit a structured trigger mechanism (<safe>), which provides both safety risk analysis and response generation guidance. (b) Safe Trigger DPO: The model refines its safety discernment by optimizing on self-generated preference pairs. These pairs are derived from jailbreak queries and ranked using our reward function (Eq. 10). Trigger approach. It systematically activates and enhances the model’s Latent Safety Awareness and significantly improves the model’s safety capabilities without compromising its general performance.
3
Our Safe Trigger Method
To explicitly activate and effectively leverage Latent Safety Awareness, we introduce Safe Trigger SFT to make LRMs explicitly activate the structured trigger mechanism (denoted as an italicized term <safe>), considering safety risks and guiding answer generation within <safe> and </safe> tags for unsafe queries. To avoid possible ambiguity, we denote the structured trigger mechanism as <safe> (in italics), while tags are represented as <safe> and </safe>. Besides, we propose Safe Trigger DPO to further enhance the guiding effectiveness and correctness of the structured trigger mechanism on the final output, reinforcing the model’s ability to produce more instructive safety content. The overall training pipeline of the Safe Trigger approach is illustrated in Figure 2. 3.1
Safe Trigger SFT: Activate the Latent Safety Awareness
For risky or uncertain queries, the structured trigger mechanism (<safe>) is inserted between the end of the reasoning process and the final response in the form of <safe> </safe>, serving as an intermediate step prior to the final answer. This mechanism consists of the following three components: – Core Issue Safety Analysis: The model re-examines the core issue based on the query and its own reasoning process, and determines whether the issue poses potential safety risks. – Reasoning Process Safety Analysis: If potential risks are identified, the model further reviews whether such risks have been adequately considered during the reasoning stage.
Triggering Latent Safety Awareness in Large Reasoning Models
5
– Final Answer Guidance Content: Based on the above analyses, the model generates guidance for the final answer, encouraging the production of more appropriate response. We constructed a training dataset consisting of 30k instances, evenly partitioned into three categories: general, harmful, and jailbreak queries (10k per category). These instances were sourced from the UltraFeedback [6], PKU-SafeRLHF [16], and WildJailbreak [19] datasets. We employ a two-stage generation strategy to construct training samples. In the first stage, we directly input the query into the model to be optimized to obtain an initial reasoning result, which includes the reasoning process <think>1 and the corresponding f inal answer1 . In the second stage, we input the risky query including harmful and jailbreak queries along with <think>1 into the same model, and append a safety analysis prompt to guide the model in performing a safety assessment based on its reasoning process and the risky query. The output consists of a regenerated reasoning process <think>2 , the structured trigger mechanism <safe>, and f inal answer2 . We construct training samples according to the following rules. For general samples, the data follows the structure {query + <think>1 + final_answer1 }. For harmful and jailbreak samples, the data follows the structure {query+<think>1 + <safe> + final_answer2 }. The combination of general, harmful, and jailbreak samples in the dataset, along with the omission of the structured trigger mechanism (<safe>) in general samples, enables the model to adaptively activate the structured trigger mechanism (<safe>). The harmful and jailbreak samples teach the model what analysis should be included within the structured trigger mechanism. After training with Safe Trigger SFT, we obtain the Safe Trigger SFT Model (ST-S). 3.2
Safe Trigger DPO: Enhance the Latent Safety Awareness
The ST-S model’s safety capabilities have been significantly improved compared with the base model. However, due to the inherent limitations of Latent Safety Awareness, there are still cases where, even after activating the structured trigger mechanism, the model fails to effectively guide the generation of a safe final answer. To address this issue, we further introduce the Safe Trigger DPO. We use 20k jailbreak queries for preference optimization, all selected from the adversarial harmful subset of the WildJailbreak dataset and disjoint from the jailbreak queries used in the SFT stage. Binary Variables and Target Ranking. For each structured model response, we define four binary indicators: Fsafe , Sexist , Tfull , Sfull ∈ {0, 1}
(1)
where Fsafe indicates whether the final answer is safe, Sexist indicates whether the response contains a complete structured trigger mechanism (<safe>), Tfull indicates whether the reasoning trace sufficiently considers safety, and Sfull indicates whether the analysis in <safe> is comprehensive. Our reward function
6
Ke Miao et al.
enforces the following preference hierarchy among four response types: A: Safe final answer with <safe> (Fsafe = 1, Sexist = 1) B: Safe final answer without <safe> (Fsafe = 1, Sexist = 0) C: Unsafe final answer with <safe> (Fsafe = 0, Sexist = 1) D: Unsafe final answer without <safe> (Fsafe = 0, Sexist = 0) In particular, we want: A≻B≻C≻D
(2)
Piecewise Construction. We first define a reward function in a piecewise manner by conditioning on (Fsafe , Sexist ). Case A: Safe final answer with <safe> (Fsafe = 1, Sexist = 1). When the final answer is safe and <safe> is activated, we additionally reward the quality of the safety-related reasoning and analysis. This is captured by the sum (Tfull + Sfull ), and we add a base offset wa to ensure this case dominates the other cases: RA ≜ Tfull + Sfull + wa
(3)
Case B: Safe final answer without <safe> (Fsafe = 1, Sexist = 0). If the final answer is safe but <safe> is not activated, we assign a constant reward wb : RB ≜ wb
(4)
This still rewards safety, but it is intentionally lower than Case A so that, among safe answers, activating <safe> is preferred. Case C: Unsafe final answer with <safe> (Fsafe = 0, Sexist = 1). If <safe> is activated but the final answer remains unsafe, we assign a smaller reward wc : RC ≜ wc
(5)
This makes unsafe outputs strictly worse than safe outputs, even when <safe> exists. Case D: Unsafe final answer without <safe> (Fsafe = 0, Sexist = 0). This is the completely unacceptable failure mode (unsafe and no structured trigger mechanism). We collapse the reward to zero: RD ≜ 0
(6)
Unified Closed-Form Expression. The above piecewise definition can be written as a single closed-form expression using indicator products. Let I[·] be the indicator function. Since Fsafe , Sexist ∈ {0, 1}, we can use products directly as indicators: I[Fsafe = 1] = Fsafe , I[Sexist = 1] = Sexist (7) I[Sexist = 0] = (1 − Sexist ), I[Fsafe = 0] = (1 − Fsafe )
(8)
Triggering Latent Safety Awareness in Large Reasoning Models
7
Combining Eqs. (3)–(6), the reward becomes: R = I[Fsafe = 1, Sexist = 1] · (Tfull + Sfull + wa ) + I[Fsafe = 1, Sexist = 0] · wb + I[Fsafe = 0, Sexist = 1] · wc + I[Fsafe = 0, Sexist = 0] · 0 which simplifies to: R = Fsafe · Sexist · (Tfull + Sfull + wa ) + Fsafe (1 − Sexist ) · wb + (1 − Fsafe ) · Sexist · wc
(9)
(10)
Constraints and Ranking Guarantees. We impose wa > wb > wc to reflect the intended preference structure in Eq. (2). We show this more explicitly below. A is better than B (safe with activating vs. safe without activating). For Case A, RA = Tfull + Sfull + wa ≥ wa since Tfull , Sfull ≥ 0. For Case B, RB = wb . Thus, RA > RB is guaranteed by: wa > wb
(11)
A is better than C (safe with activating vs. unsafe with activating). Case A has RA ≥ wa , while Case C has RC = wc . Thus, RA > RC is guaranteed by: wa > wc
(12)
B is better than C (safe without activating vs. unsafe with activating). Case B has RB = wb , while Case C has RC = wc . Thus, RB > RC is guaranteed by: wb > wc
(13)
D collapses to zero (unsafe without activating). When Fsafe = 0 and Sexist = 0, each term in Eq. (10) becomes zero: R = 0 · 0 · (·) + 0 · 1 · wb + 1 · 0 · wc = 0
(14)
Therefore, Case D always receives RD = 0, matching the intended “completely unacceptable” behavior. Given Fsafe = 1 and Sexist = 1, the reward additionally increases with Tfull and Sfull . This encourages not only activating but also producing safety-aware reasoning traces and a comprehensive safety analysis, thereby improving the guiding power and stability of <safe> during preference optimization. Detailed Training Procedure. For the 20k jailbreak queries, we first perform high temperature sampling to generate diverse responses. For each query qi , we sample 4 distinct structured outputs {ri,1 , ri,2 , ri,3 , ri,4 }, each consisting of a reasoning trace, the structured trigger mechanism (if triggered), and a final answer. Each response ri,j is independently scored using the reward function R(ri,j ). We select queries where the sampled responses exhibit significant variance in reward scores. Formally, we retain only those queries qi for which: max R(ri,j ) − min R(ri,j ) ≥ δ j
j
(15)
8
Ke Miao et al.
This filtering ensures that each retained query provides both a high-quality (positive) and a low-quality (negative) response, denoted as (ri+ , ri− ). These preference pairs are used to train the model. The training objective maximizes the preference for the better response over the worse one, relative to a reference policy πref , and is defined as: πθ (ri− | qi ) πθ (ri+ | qi ) − log (16) LDPO = − log σ β · log πref (ri+ | qi ) πref (ri− | qi ) where πθ is the current policy, πref is the reference policy, and β controls the sharpness of preference. After training with Safe Trigger DPO, we obtain the Safe Trigger DPO Model (ST-D).
4
Experimental Results
4.1
Experimental Settings
We conduct experiments on four representative LRMs in the main comparison, including Qwen3-8B, Qwen3-32B, DeepSeek-R1-Distill-Llama-8B, and DeepSeekR1-Distill-Llama-70B [38, 8]. For additional experiments and analyses, we performed them on our default backbone model, DeepSeek-R1-Distill-Llama-8B. Baselines. We compare the proposed Safe Trigger approach with two representative safety alignment methods for LRMs. Star1 [41] is a data-centric alignment method that improves LRM safety by fine-tuning models on a carefully curated, human-audited, small-scale dataset consisting of 1,000 high-quality safety samples. SafePath [15] is an alignment approach specifically designed for LRMs. It fine-tunes the model to emit a short safety prefix at the beginning of the reasoning process when encountering harmful prompts, while leaving the remainder of the reasoning trajectory unconstrained. SafePath is designed to reduce harmful outputs with minimal interference to reasoning depth and is regarded as the current state-of-the-art method. Evaluation metrics. Four types of tasks were designed to comprehensively evaluate the methods. (1) Harmful task evaluates the model’s robustness against direct harmful or illegal requests. We adopt several widely used harmful benchmarks, including AdvBench [4], HexPHI [32], and XsTest (Unsafe subset) [33], which together cover a broad range of explicit safety risks. (2) Jailbreak task assesses the model’s robustness under adversarial manipulation. Specifically, we use the test set of WildJailbreak [19] and further evaluate the model under two strong jailbreak attack methods, MSJ [2] and PAP [43]. Following prior work [22, 18, 42], we use LlamaGuard-3-8B as an automatic safety discriminator to determine whether the model outputs unsafe content. LlamaGuard-3-8B is a widely adopted open-source safety classifier that provides consistent, scalable, and reproducible safety judgments across different models and alignment methods. The research [20] also experimentally demonstrated that LlamaGuard-3-8B achieves higher accuracy in evaluating the final answers of LRMs. (3) General task is used to verify that safety alignment does not significantly degrade the model’s
Triggering Latent Safety Awareness in Large Reasoning Models
9
Table 3: Performance comparison across different models and methods. ↓ indicates lower is better, ↑ indicates higher is better. Bold indicates the best result, and underline indicates the second best. These notations are consistently applied in all subsequent tables. Harmful ↓ Jailbreak ↓ General ↑ Over-Refusal ↓ AdvB. HexPHI XsTest WildJail. MSJ PAP ARC Drop Wino XsTest-S Qwen3-8B Base 1.92% 13.33% 0.50% 42.80% 8.00% 12.00% 0.9267 0.6623 0.7822 3.20% 0.19% 8.67% 0.50% 37.40% 10.00% 12.00% 0.9278 0.6364 0.6551 2.40% Star1 [41] SafePath [15] 0.38% 7.33% 0.00% 36.95% 18.00% 12.00% 0.9264 0.6226 0.6543 3.20% ST-S 0.58% 6.67% 0.00% 29.50% 0.00% 10.00% 0.9250 0.6580 0.7782 4.80% ST-D 0.38% 5.67% 0.00% 23.20% 2.00% 4.00% 0.9242 0.6567 0.7774 2.00% DeepSeek-R1-Distill-Llama-8B Base 35.19% 44.00% 20.00% 49.90% 68.00% 20.00% 0.8810 0.4677 0.6638 4.00% Star1 [41] 14.04% 20.67% 9.50% 39.15% 54.00% 14.00% 0.8796 0.4837 0.5359 7.20% 12.00% SafePath [15] 14.62% 24.67% 8.50% 16.65% 58.00% 2.00% 0.8698 0.4561 0.5130 ST-S 9.62% 20.00% 8.00% 29.95% 14.00% 4.00% 0.8749 0.4876 0.6835 2.80% ST-D 4.42% 13.33% 7.50% 17.75% 8.00% 2.00% 0.8791 0.4620 0.6732 2.40% Qwen3-32B Base 3.85% 12.33% 0.00% 35.40% 10.00% 12.00% 0.9459 0.6272 0.7648 1.60% Star1 [41] 6.73% 9.00% 0.00% 30.85% 12.00% 10.00% 0.9434 0.6402 0.7403 3.20% SafePath [15] 4.81% 13.33% 0.00% 24.80% 10.00% 10.00% 0.9459 0.6513 0.7245 2.80% ST-S 1.35% 10.00% 0.00% 27.15% 4.00% 10.00% 0.9461 0.6081 0.7514 3.20% ST-D 1.73% 8.33% 0.00% 27.10% 2.00% 4.00% 0.9475 0.6081 0.7648 3.20% DeepSeek-R1-Distill-Llama-70B Base 26.15% 23.33% 14.00% 32.45% 18.00% 18.00% 0.9416 0.6721 0.8493 5.20% Star1 [41] 18.65% 19.67% 15.50% 29.60% 10.00% 14.00% 0.9433 0.6733 0.8461 7.60% 10.00% SafePath [15] 20.96% 20.00% 14.00% 28.95% 10.00% 12.00% 0.9411 0.6665 0.8453 ST-S 1.35% 7.33% 3.50% 9.95% 0.00% 2.00% 0.9411 0.6689 0.8516 6.00% ST-D 0.58% 1.67% 0.00% 1.45% 0.00% 2.00% 0.9417 0.6730 0.8564 6.80% Method
general performance. We evaluate the model on a set of commonly used general benchmarks, including ARC (Mean of Easy and Challenge) [5], DROP [9], and Wino [34], covering reasoning, reading comprehension, and common-sense understanding. We adopt the EvalScope [37] evaluation framework to obtain standardized and comparable performance scores. (4) Over-Refusal task measures whether the model incorrectly refuses to answer problems that seem harmful but are actually harmless after safety alignment. We use the XsTest (Safe subset) [33] benchmark to evaluate over-refusal behavior. We employ GPT-4o [28] as an external judge to determine whether the model unnecessarily refuses to answer safe queries. Implementation. All experiments were conducted on a single compute node equipped with 8 × NVIDIA L20X GPUs, each with 144 GB of memory. The node is powered by two Intel Xeon Platinum 8558 processors, providing 192 CPU cores and a total of 1 TB of system memory. Experiments were executed on multiple GPUs using the Hugging Face Transformer Reinforcement Learning 0.23.0 framework with LoRA-based parameter-efficient fine-tuning, and LoRA was implemented with a rank of 64, an α value of 16, and a dropout rate of 0.05 by default.
10
Ke Miao et al.
Table 4: Overall average performance comparison across different safety alignment methods. Results are computed by taking the mean over all evaluated models and all benchmarks within each category. Method Base Star1 [41] SafePath [15] ST-S ST-D
4.2
Harmful ↓ Jailbreak ↓ General ↑ Over-Refusal ↓ 16.22% 10.26% 10.72% 5.70% 3.63%
27.21% 22.75% 19.95% 11.71% 7.79%
0.7654 0.7421 0.6843 0.7645 0.7637
3.50% 5.10% 7.00% 4.20% 3.60%
Main Results
Table 3 reports detailed performance for each method across four LRMs and all evaluated benchmarks. Across all four LRMs, Safe Trigger consistently improves safety alignment in both harmful and jailbreak scenarios while preserving general capability. On harmful benchmarks, both ST-S and ST-D reduce harmful response rates relative to Base, Star1, and SafePath for most model-benchmark combinations. Safe Trigger substantially lowers the harmful response rate across multiple harmful datasets, indicating that explicitly inserting structured safety analysis between reasoning and final answer can effectively mitigate unsafe completion. On jailbreak evaluations, Safe Trigger also shows clear advantages: ST-S already provides strong reductions in attack success rate, and ST-D further improves robustness, suggesting that preference optimization strengthens the stability and guiding power of the triggered safety analysis under adversarial prompts. Meanwhile, Safe Trigger preserves general capability, remaining comparable to the Base model and avoiding the larger degradation observed in some baselines. Safe Trigger maintains a relatively low over-refusal rate on XsTest-Safe, indicating that the method does not rely on overly aggressive refusal to obtain safety gains. To provide a more intuitive comparison, we further summarize the results by averaging performance across all evaluated models and benchmarks within each task category. The aggregated results are reported in Table 4, which highlights the overall safety–utility trade-offs of different methods. The visualization of this table is shown in Figure 1. Overall, Safe Trigger achieves the best balance between safety and utility among all compared methods. In terms of harmful and jailbreak metrics, both ST-S and ST-D substantially outperform the baseline methods, with ST-D achieving the lowest average harmful response rate and jailbreak attack success rate. For general capability, Safe Trigger remains comparable to the Base model. Regarding the Over-Refusal benchmark, ST-D maintains a low refusal rate close to the Base model, suggesting that the additional preference optimization stage helps integrate safety constraints into the final answer without relying on excessive refusals.
2.5 2.0 1.5 1.0 0.5 0.0
Cumulative Entropy
First Token Entropy
Triggering Latent Safety Awareness in Large Reasoning Models
Base
Star1 SafePath
ST-S
Method Type
(a) Entropy Distribution.
ST-D
8 6
11
Base Star1 SafePath ST-S ST-D
4 2 0
5
10
Token Position
15
20
(b) Cumulative Entropy.
Fig. 3: Entropy distribution and cumulative trends of the initial tokens.
4.3
Detailed Analysis
Alignment Depth Figure 3 analyzes the model behavior during the early stage of final answer decoding by examining the first-token entropy distribution and the cumulative entropy across token positions [31]. The data in Figure 3 are collected from 100 conditionally sampled queries from AdvBench, where we only retain queries on which all five methods (Base, Star1, SafePath, ST-S, ST-D) produce responses that are judged as safe. By controlling safety outcomes in this way, the goal is to compare which method behaves better when they all successfully produce safe answers. Under this controlled setting, Figure 3a shows that the base model exhibits significantly higher first-token entropy with a broad distribution, indicating less deterministic behavior at the very beginning of final answer generation. Star1 and SafePath demonstrate moderate entropy reduction compared to Base, but still present relatively wider distributions with more high-entropy outliers, suggesting inconsistent early decoding patterns. ST-S and ST-D achieve lower and more concentrated first-token entropy distributions, with ST-D showing the tightest concentration near zero entropy. This indicates that Safe Trigger exhibits highly confident and consistent first-step decisions when initiating safe responses. Figure 3b further reveals the stability of safety alignment in the early generation process. The base model accumulates entropy most rapidly, reflecting sustained uncertainty across multiple decoding steps. Star1 and SafePath show moderate cumulative entropy growth, while ST-S demonstrates notably slower accumulation. ST-D maintains the lowest cumulative entropy trajectory, indicating the most stable and deterministic generation path. The diverging cumulative entropy curves suggest that Safe Trigger not only improves initial token confidence but also maintains consistently lower uncertainty in the early generation phase. Activation Probability Figure 4 reports the structured trigger mechanism activation rate of ST-S and ST-D across different benchmark categories. The
Ke Miao et al.
Harmful
100 97.8888.6795.00 75
Jailbreak
General Over-Refusal
92.10 86.0088.00
50 25 0
16.40
h HI st enHcexP XsTe b v Ad
il J P dJa MS PA Wil
0.00 0.00 0.08
C p o AR Dro Win
-S
est XsT
ST-D Activation Rate (%)
ST-S Activation Rate (%)
12
Harmful
100 99.2390.0094.50 75
Jailbreak
General Over-Refusal
95.1094.00 84.00
50 25 0
14.80
h HI st enHcexP XsTe b v Ad
il J P dJa MS PA Wil
0.00 0.00 0.08
C p o AR Dro Win
-S
est XsT
Fig. 4: The structured trigger mechanism activation rates of the ST-S and ST-D models across different benchmark categories.
activation rate measures how often the model generates the structured trigger mechanism during inference, reflecting whether the structured trigger mechanism is invoked selectively under risky inputs while remaining silent for benign queries. Both ST-S and ST-D exhibit high activation probability on risky inputs. On harmful and jailbreak benchmarks, the activation rate remains consistently high and approaches full activation, indicating that the models reliably detect potentially unsafe scenarios and insert the safety analysis before producing the final answer. In contrast, the activation rate on general benchmarks remains close to zero for both models, showing that the structured trigger mechanism is rarely misactivated on benign tasks and thus does not unnecessarily introduce extra safety analysis or interfere with normal user experience. The activation rate on the Over-Refusal benchmark is low, but not as close to zero as that of the general category, which is quite reasonable. When the model encounters queries whose risk is uncertain, it proactively invokes the structured trigger mechanism for safety analysis; however, after concluding that the query is benign, it proceeds to answer rather than refusing. This behavior is consistent wi th the low overrefusal rates reported in Table 3 and Table 4. ST-D shows a slightly higher and more stable activation rate on risky benchmarks compared to ST-S, suggesting that the preference optimization stage further strengthens the reliability of the triggering behavior under adversarial conditions, while preserving the restraint on benign queries.
Inference Resource Figure 5 compares the average inference length of Base, ST-S, and ST-D across four task categories. To ensure a fair comparison of inference overhead, the harmful and jailbreak results are computed only on samples whose responses are judged as safe. This control is important because if unsafe outputs were included, the Safe Trigger approach would convert a large portion of responses into safe ones, and safe responses are naturally shorter than unsafe, detailed completions. This evaluation isolates the effect of the structured trigger mechanism (<safe>) itself and reveals how much additional inference cost it introduces.
Task Type
Triggering Latent Safety Awareness in Large Reasoning Models
l mfu Har reak Jailb l era Gen al efus R r 0 Ove
797.6 824.3
919.5
939.9
1042.8 1045.7
13
Base ST-S ST-D
535.7 534.0 536.7 1005.6 1003.1 983.5
200
400
600
800
Average Inference Length (tokens)
1000
Fig. 5: Comparison of inference overhead between the Safe Trigger approach and the base model. For harmful queries, ST-S generates longer outputs than Base, which is expected because the structured trigger mechanism (<safe>) explicitly inserts a structured safety analysis before the final answer. After preference optimization, <safe> becomes more concise and targeted, so ST-D is only slightly longer than Base, indicating improved efficiency in safety analysis generation. For jailbreak queries under the safe-only setting, ST-S shows an inference length that is nearly identical to Base, while ST-D exhibits a clear reduction. Based on our experimental observations, this is because the Base model often gets trapped in attacker-crafted narratives during jailbreak attempts and generates redundant explanatory tokens. In contrast, Safe Trigger emphasizes directly extracting and analyzing the core issues, which significantly eliminates unnecessary output. For general and over-refusal categories, the response lengths of ST-S and ST-D remain nearly identical to Base, suggesting that Safe Trigger introduces negligible overhead on benign inputs and has minimal impact on overall generation behavior when safety triggering is not required.
5
Related Work
A substantial body of work has explored jailbreak attacks [1, 48, 14], privacy exposure risks [24, 27, 40, 26, 11, 3, 12], and safety alignment [7, 39, 23, 30, 35] for LLMs. In contrast, there is still less research work focusing on the safety of LRMs [13, 47, 10, 17, 41, 15]. For LRMs’ safety, [13] first introduced the concept of Safety Tex for safety alignment in Large Reasoning Models, highlighting that for reasoning models, safety alignment naturally leads to a reduction in reasoning capabilities. [47] systematically evaluate the safety vulnerability of DeepSeek R1 and find that its enhanced reasoning capabilities can inadvertently amplify harmful outputs compared to vanilla LLMs. [17] evaluate the safety of LRMs and find that reducing the thinking content of LRMs to zero could effectively enhance their safety. Additionally, they construct a 40k CoT dataset (SafeChain), which contains reasoning responses from the distilled Llama-70B with DeepSeek R1, to
14
Ke Miao et al.
improve the safety of LRMs via fine-tuning. In contrast to the large quantity of CoT data in SafeChain, [41] enhance the safety of LRMs via fine-tuning LRMs with 1k high-quality and source-diverse CoT data (STAR1) containing the deliberative reasoning content regarding safety policies [10], which leads to better safety improvement than SafeChain. [10] similarly integrate SFT and RL by fine-tuning models on a CoT dataset of safety policies and incorporating safetyaware rewards during RL optimization. The construction of the CoT dataset is time-consuming, and their method is inferior to Star1 [41] compared in this work. Despite initial progress, the above safety alignment methods rely on manual or advanced closed source models to carefully construct CoT training data, which can result in expensive resource overhead. [15] fine-tunes the model to emit a short safety prefix at the start of reasoning when faced with harmful prompts, while leaving the remaining reasoning trajectory unconstrained. It is regarded as a current state-of-the-art approach, achieving strong safety gains with minimal impact on reasoning depth and inference overhead. However, we found that safety considerations in the reasoning process do not always translate effectively into safe final responses, indicating substantial room for improvement in the safety alignment of LRMs. Our method significantly improves safety alignment by explicitly activating and strengthening the Latent Safety Awareness of LRMs, while maintaining strong general-purpose performance. It operates without manual annotation or reliance on closed-source models, making it both resource-efficient and highly scalable.
6
Conclusion
This paper identifies Latent Safety Awareness in LRMs and proposes the Safe Trigger approach to explicitly and efficiently leverage this capability. We introduce Safe Trigger SFT to teach LRMs to selectively invoke a structured trigger mechanism under potentially risky inputs, and further propose Safe Trigger DPO to strengthen the guiding effectiveness and stability in shaping the final answer. Experiments show that the Safe Trigger approach substantially improves safety while preserving general capability. The proposed pipeline is fully selfbootstrapped, yielding a scalable, low-cost, and stable solution for large-scale safety alignment.
7
Acknowlegments
This research is supported in part by the “Pioneer” and “Leading Goose” R&D Program of Zhejiang (Grant No. 2024C01169), the Kunpeng–Ascend Science and Education Innovation Excellence/Incubation Center, the National Natural Science Foundation of China (Grant No. 6244123), the National Natural Science Foundation of China under Grant U2441240 (“Ye Qisun” Science Foundation), the Li Auto Inc., and the NGICS Comprehensive Security Project for Industrial Control Systems.
Triggering Latent Safety Awareness in Large Reasoning Models
15
References 1. Andriushchenko, M., Croce, F., Flammarion, N.: Jailbreaking leading safetyaligned llms with simple adaptive attacks (2024) 2. Anil, C., DURMUS, E., Rimsky, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D.J., Mosconi, F., Agrawal, R., Schaeffer, R., Bashkansky, N., Svenningsen, S., Lambert, M., Radhakrishnan, A., Denison, C., Hubinger, E.J., Bai, Y., Bricken, T., Maxwell, T., Schiefer, N., Sully, J., Tamkin, A., Lanham, T., Nguyen, K., Korbak, T., Kaplan, J., Ganguli, D., Bowman, S.R., Perez, E., Grosse, R.B., Duvenaud, D.: Many-shot jailbreaking. In: The Thirtyeighth Annual Conference on Neural Information Processing Systems (2024), https://openreview.net/forum?id=cw5mgd71jW 3. Bao, W., Lou, J., Hu, Y., Li, X., Liu, Z., Liu, J., Qin, Z., Ren, K.: Module-aware parameter-efficient machine unlearning on transformers (2025), https://arxiv.org/abs/2508.17233 4. Chen, Y., Gao, H., Cui, G., Qi, F., Huang, L., Liu, Z., Sun, M.: Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp (2022), https://arxiv.org/abs/2210.10683 5. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., Tafjord, O.: Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1 (2018) 6. Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al.: Ultrafeedback: Boosting language models with scaled ai feedback. In: Proceedings of the 41st International Conference on Machine Learning. vol. 235, pp. 9722–9744 (2024) 7. Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., Yang, Y.: Safe RLHF: Safe reinforcement learning from human feedback. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=TyFrPOKYXw 8. DeepSeek-AI: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025), https://arxiv.org/abs/2501.12948 9. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., Gardner, M.: Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs (2019), https://arxiv.org/abs/1903.00161 10. Guan, M.Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al.: Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339 (2024) 11. Hu, Y., Li, Z., Liu, Z., Zhang, Y., Qin, Z., Ren, K., Chen, C.: Membership inference attacks against vision-language models. In: 34th USENIX Security Symposium (USENIX Security) (2025) 12. Hu, Y., Lou, J., Liu, J., Ni, W., Lin, F., Qin, Z., Ren, K.: ERASER: Machine unlearning in MLaaS via an inference serving-aware approach. In: Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS) (2024). https://doi.org/10.1145/3658644.3670398 13. Huang, T., Hu, S., Ilhan, F., Tekin, S.F., Yahn, Z., Xu, Y., Liu, L.: Safety tax: Safety alignment makes your large reasoning models less reasonable (2025), https://arxiv.org/abs/2503.00555 14. Huang, Y., Gupta, S., Xia, M., Li, K., Chen, D.: Catastrophic jailbreak of opensource llms via exploiting generation. arXiv preprint arXiv:2310.06987 (2023)
16
Ke Miao et al.
15. Jeung, W., Yoon, S., Kahng, M., No, A.: SAFEPATH: Preventing harmful reasoning in chain-of-thought via early alignment. In: The Thirtyninth Annual Conference on Neural Information Processing Systems (2025), https://openreview.net/forum?id=vIaNnnQxcl 16. Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., Yang, Y.: Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv preprint arXiv:2406.15513 (2024) 17. Jiang, F., Xu, Z., Li, Y., Niu, L., Xiang, Z., Li, B., Lin, B.Y., Poovendran, R.: Safechain: Safety of language models with long chain-of-thought reasoning capabilities. arXiv preprint arXiv:2502.12025 (2025) 18. Jiang, F., Xu, Z., Li, Y., Niu, L., Xiang, Z., Li, B., Lin, B.Y., Poovendran, R.: SafeChain: Safety of language models with long chain-of-thought reasoning capabilities. In: Findings of the Association for Computational Linguistics: ACL 2025. pp. 23303–23320. Association for Computational Linguistics, Vienna, Austria (Jul 2025). https://doi.org/10.18653/v1/2025.findings-acl.1197, https://aclanthology.org/2025.findings-acl.1197/ 19. Jiang, L., Rao, K., Han, S., Ettinger, A., Brahman, F., Kumar, S., Mireshghallah, N., Lu, X., Sap, M., Choi, Y., Dziri, N.: Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models (2024), https://arxiv.org/abs/2406.18510 20. Li, C., Wang, J., Pan, X., Hong, G., Yang, M.: Reasoningshield: Safety detection over reasoning traces of large reasoning models (2025), https://arxiv.org/abs/2505.17244 21. Li, M., Si, W.M., Backes, M., Zhang, Y., Wang, Y.: Salora: Safety-alignment preserved low-rank adaptation (2025), https://arxiv.org/abs/2501.01765 22. Li, M., Si, W.M., Backes, M., Zhang, Y., Wang, Y.: SaloRA: Safety-alignment preserved low-rank adaptation. In: The Thirteenth International Conference on Learning Representations (2025), https://openreview.net/forum?id=GOoVzE9nSj 23. Li, M., Wang, W., Feng, F., Zhu, F., Wang, Q., Chua, T.S.: Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. arXiv preprint arXiv:2403.09972 (2024) 24. Liu, Z., Lou, J., Bao, W., Hu, Y., Qin, Z., Ren, K.: Differentially private zerothorder methods for scalable large language model finetuning. IEEE Transactions on Information Forensics and Security (2026) 25. Liu, Z., Wu, Y., Lou, J., Wang, D., Zhou, Y., Hu, Y.: Aligned but fragile: Enhancing LLM safety robustness via zeroth-order optimization (2026), https://arxiv.org/abs/2605.29396 26. Luo, Z., Shao, S., Zhang, S., Zhou, L., Hu, Y., Zhao, C., Liu, Z., Qin, Z.: Shadow in the cache: Unveiling and mitigating privacy risks of KV-cache in LLM inference. In: Proceedings of the Network and Distributed System Security Symposium (NDSS) (2026) 27. Miao, K., Hu, Y., Li, X., Bao, W., Liu, Z., Qin, Z., Ren, K.: Towards evaluation for real-world LLM unlearning (2025), https://arxiv.org/abs/2508.01324 28. OpenAI: Gpt-4 technical report (2024), https://arxiv.org/abs/2303.08774 29. OpenAI: Openai o1 system card (2024), https://arxiv.org/abs/2412.16720 30. Phan, H., Li, V., Lei, Q.: Think twice, generate once: Safeguarding by progressive self-reflection (2025), https://arxiv.org/abs/2510.01270 31. Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., Henderson, P.: Safety alignment should be made more than just a few tokens deep (2024), https://arxiv.org/abs/2406.05946
Triggering Latent Safety Awareness in Large Reasoning Models
17
32. Qi, X., Zeng, Y., Xie, T., Chen, P.Y., Jia, R., Mittal, P., Henderson, P.: Fine-tuning aligned language models compromises safety, even when users do not intend to! (2023), https://arxiv.org/abs/2310.03693 33. Röttger, P., Kirk, H.R., Vidgen, B., Attanasio, G., Bianchi, F., Hovy, D.: Xstest: A test suite for identifying exaggerated safety behaviours in large language models (2024), https://arxiv.org/abs/2308.01263 34. Sakaguchi, K., Bras, R.L., Bhagavatula, C., Choi, Y.: Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641 (2019) 35. Si, S., Wang, X., Zhai, G., Navab, N., Plank, B.: Think before refusal : Triggering safety reflection in llms to mitigate false refusal behavior (2025), https://arxiv.org/abs/2503.17882 36. Team, K.: Kimi k1.5: Scaling reinforcement learning with llms (2025), https://arxiv.org/abs/2501.12599 37. Team, M.: EvalScope: Evaluation framework for large models (2024), https://github.com/modelscope/evalscope 38. Team, Q.: Qwen3 technical report (2025), https://arxiv.org/abs/2505.09388 39. Wachi, A., Tran, T.Q., Sato, R., Tanabe, T., Akimoto, Y.: Stepwise alignment for constrained language model policy optimization. In: The Thirtyeighth Annual Conference on Neural Information Processing Systems (2024), https://openreview.net/forum?id=VrVx83BkQX 40. Wang, Y., Miao, K., Hu, Y., Li, X., Qin, Z.: Deconstruct evaluate and targeted intervention based LLMs unlearning algorithm. Chinese Journal of Electronics (2025). https://doi.org/10.23919/cje.2025.00.105 41. Wang, Z., Tu, H., Wang, Y., Wu, J., Mei, J., Bartoldson, B.R., Kailkhura, B., Xie, C.: Star-1: Safer alignment of reasoning llms with 1k data. arXiv preprint arXiv:2504.01903 (2025) 42. Xue, Y., Mirzasoleiman, B.: Lora is all you need for safety alignment of reasoning llms (2025), https://arxiv.org/abs/2507.17075 43. Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., Shi, W.: How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms (2024), https://arxiv.org/abs/2401.06373 44. Zhang, Y., Zhang, S., Huang, Y., Xia, Z., Fang, Z., Yang, X., Duan, R., Yan, D., Dong, Y., Zhu, J.: STAIR: Improving safety alignment with introspective reasoning. In: Forty-second International Conference on Machine Learning (2025), https://openreview.net/forum?id=aHzPGyUhZa 45. Zhao, X., Cai, W., Shi, T., Huang, D., Lin, L., Mei, S., Song, D.: Improving llm safety alignment with dual-objective optimization (2025), https://arxiv.org/abs/2503.03710 46. Zhou, K., Liu, C., Zhao, X., Jangam, S., Srinivasa, J., Liu, G., Song, D., Wang, X.E.: The hidden risks of large reasoning models: A safety assessment of r1 (2025), https://arxiv.org/abs/2502.12659 47. Zhou, K., Liu, C., Zhao, X., Jangam, S., Srinivasa, J., Liu, G., Song, D., Wang, X.E.: The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659 (2025) 48. Zou, A., Wang, Z., Kolter, J.Z., Fredrikson, M.: Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023)