Re-Triggering Safeguards within LLMs for Jailbreak Detection
Zheng Lin 1 Zhenxing Niu 1 Haoxuan Ji 2 Yuzhe Huang 2 Haichang Gao 1
arXiv:2605.10611v1 [cs.CR] 11 May 2026
Abstract This paper proposes a jailbreaking prompt detection method for large language models (LLMs) to defend against jailbreak attacks. Although recent LLMs are equipped with built-in safeguards, it remains possible to craft jailbreaking prompts that bypass them. We argue that such jailbreaking prompts are inherently fragile, and thus introduce an embedding disruption method to re-activate the safeguards within LLMs. Unlike previous defense methods that aim to serve as standalone solutions, our approach instead cooperates with the LLM’s internal defense mechanisms by retriggering them. Moreover, through extensive analysis, we gain a comprehensive understanding of the disruption effects and develop an efficient search algorithm to identify appropriate disruptions for effective jailbreak detection. Extensive experiments demonstrate that our approach effectively defends against state-of-the-art jailbreak attacks in white-box and black-box settings, and remains robust even against adaptive attacks.
Figure 1. Our jailbreak prompt detection is achieved by injecting an appropriate noise into the token embeddings to elicit a denial response from the LLM.
LLMs to produce harmful content (Wei et al., 2023). For example, Zou et al.’s pioneering work (Zou et al., 2023) revealed that carefully crafted prompt suffixes can successfully jailbreak a wide range of popular LLMs. Recently, many defense methods against jailbreaking have been proposed, which can be broadly classified into three categories: detection-based (e.g., perplexity filtering (Alon & Kamfonas, 2023)), input preprocessing-based (e.g., paraphrasing (Kirchenbauer et al., 2024) and retokenization (Provilkov et al., 2020)), and robust optimization-based (e.g., adversarial training (Jain et al., 2023; Mazeika et al., 2024)). Each of these methods has its own advantages but also comes with notable limitations, such as being effective only against specific jailbreak attacks, incurring significant computational overhead, or failing to strike a balance between alignment robustness and model utility. For example, perplexity filtering is effective against the GCG attack but cannot defend against the PAIR attack.
1. Introduction With the widespread deployment of large language models (LLMs), research on AI alignment (Ouyang et al., 2022; Bai et al., 2022; Korbak et al., 2023) has attracted significant attention, aiming to ensure that LLMs align with human values and faithfully follow human intent. A key requirement of alignment is to prevent LLMs from producing objectionable responses to harmful user queries. One representative approach is to embed safeguards within LLMs using techniques such as RLHF (Ziegler et al., 2019). This method can effectively prevent LLMs from generating harmful outputs in response to clearly malicious queries, and thus most modern LLMs are equipped with such defensive mechanisms. However, recent studies have demonstrated that a particular class of attacks, known as jailbreaking attacks, can bypass these alignment safeguards and induce
In this paper, through a comprehensive analysis of existing jailbreaking methods, we observed that whether at the token or prompt level (Robey et al., 2024)—whether via optimization algorithms (e.g., GCG (Zou et al., 2023)) or with the assistance of an adversarial LLM (e.g., PAIR (Chao et al., 2024b))—crafting a successful-jailbreaking prompt requires multiple rounds of iterative refinement. This iterative process suggests that successful-jailbreaking prompts are difficult to obtain, implying that the prompt space is predominantly occupied by those failed-jailbreaking prompts, with only a few successful-jailbreaking prompts sparsely distributed among them (Yang et al., 2021). Therefore, we
1 Xidian University 2 Xi’an Jiaotong University. Correspondence to: Zhenxing Niu <[email protected]>.
Preprint. May 12, 2026.
1
Re-Triggering Safeguards within LLMs for Jailbreak Detection
argue that such successful-jailbreaking prompts are inherently fragile—even slight perturbations are sufficient to cause them to fail. This fragility property stems from LLMs’ built-in safeguards: these defense mechanisms can block most harmful inputs, leaving only small, isolated vulnerable regions for attackers to discover. Consequently, even a slight perturbation can turn a successful-jailbreaking prompt into a failed one. Based on that, we propose a jailbreaking-detection method that injects an appropriate noise into the input prompt and examines whether the disruption can cause the LLM to produce a denial response (e.g., “Sorry, I cannot ...”). Essentially, our approach is built upon the assumption that disrupting a successful-jailbreaking prompt—thereby turning it into a failed one—tends to elicit a denial response from the LLM, whereas disrupting a benign prompt does not provoke such a response. This assumption arises from the underlying mechanism of existing jailbreaking methods: they begin with an explicitly malicious query (which elicits a denial response) and iteratively refine it until obtaining an accepting response (i.e., achieving the jailbreak goal). Therefore, we argue that disrupting a successful-jailbreaking prompt will likely cause the LLM to revert to producing a denial response.
Figure 2. Multiple noise injection options are available for prompt disruption, including the choice of injection layer, target token, affected dimensions, and noise strength.
low False-alarm Rate (FR). In terms of reducing the Attack Success Rate (ASR), our approach outperforms all existing defense methods. Besides, our approach can achieve effective defense in both white-box and black-box settings, and remains robust even against adaptive jailbreak attacks.
However, not every noise injection can elicit a denial response. For instance, if the disruption is too weak, it exerts little influence on the LLM’s output, whereas an overly strong disruption may yield nonsensical results, such as gibberish. Beyond disruption strength, several other factors influence the effectiveness of noise injection, such as which layer and which token the noise is applied to, both of which affect the likelihood of triggering a denial response. To this end, we conduct extensive experiments to gain a comprehensive understanding of the disruption effects and to derive guidelines for identifying an appropriate noise.
2. Our Approach 2.1. Fragility of Jailbreaking Prompts Our approach is built upon the assumption that a successfuljailbreaking prompt is inherently fragile—injecting an appropriate noise can transform a successful-jailbreaking prompt into a failed one. To validate this assumption, we conduct extensive experiments to examine the consequence of prompt disruption by exploring all possible noise configurations and observing the corresponding changes in the LLM’s outputs. Specifically, the successful-jailbreaking prompts used in our experiments are generated by various jailbreak methods (e.g., GCG, PAIR, RS, I-FSJ and AutoDAN-Turbo) against different LLMs (e.g., Llama, Vicuna, Qwen) across multiple datasets (e.g., AdvBench, JailbreakBench). Therefore, we believe our findings are generic and not limited to any specific model or setting.
Based on these guidelines, we propose a noise-search algorithm capable of effectively finding suitable noise which, when injected to a successful-jailbreaking prompt, can cause the LLM to produce a denial response. Empirical results show that, given a predefined search budget, our algorithm can reliably find appropriate noise for jailbreak prompts. In contrast, for benign prompts, it is less likely to find noise that induces the LLM to generate a denial response.
There exist multiple options for prompt disruption, including the choice of injection layer, target token, affected dimensions, and noise strength, as shown in Fig. 2. First, we consider which layer of the LLM should be selected for noise injection: (1) directly perturbing the input prompt, (2) injecting noise into the embeddings, or (3) adding noise to the hidden states of an intermediate layer in the LLM. As shown in Fig. 6, we find that embedding-level disruption is the most effective and straightforward approach, as it achieves a strong impact on model behavior while remaining
We conduct extensive evaluations of our approach. Specifically, we incorporate both token-level and prompt-level jailbreaking methods such as GCG, PAIR, Random Search (RS) (Andriushchenko et al., 2025) and I-FSJ (Zheng et al., 2024), into our experiments, as well as the state-of-the-art method, AutoDAN-Turbo (Liu et al., 2025a). Furthermore, we compare our approach with four distinct defense techniques. Extensive experiments demonstrate that our method achieves a high Detection Rate (DR) while maintaining a 2
Re-Triggering Safeguards within LLMs for Jailbreak Detection
range with a step size of 0.05. The results are illustrated in Fig. 3, 4, 6 and 7, where green denotes a denial response, orange indicates that the LLM’s output is unaffected or only slightly influenced, and red represents a nonsensical response. Overall, we make the following observations:
Figure 3. The response of the LLM with respect to the increase in noise strength ||δ||2 . Top: disrupting a successful-jailbreaking prompt; Bottom: disrupting a benign prompt. Green denotes a denial response, orange indicates that the LLM’s output is unaffected, and red represents a nonsensical response (gibberish).
(1) We can always identify an appropriate noise that causes the LLM to output a denial response, regardless of the jailbreak method used or the jailbreak prompt sampled. In contrast, when disrupting benign prompts (Fig. 3 bottom), we find that even exhaustive brute-force searches fail to produce any noise capable of eliciting a denial response. These findings validate our underlying assumption and justify the soundness of our proposed approach. (2) As illustrated in Fig. 4, disrupting the last token proves more effective than the other three strategies. This suggests that the overall representation of the input—captured by the final token, which summarizes the contextual semantics of all preceding tokens—is crucial for jailbreak detection. Consequently, one cannot simply attribute the success of jailbreaks to a few isolated keywords within the prompt. In addition, the Fictitious-token strategy performs slightly better than the Harmful-token strategy.
Figure 4. The comparison of disruptions applied to Random-, Harmful-, Fictitious-, and Last-token strategies.
easy to implement. Second, we examine which tokens should receive the noise. As shown in Fig. 2, we test four strategies: (1) Last-token disruption. Inspired by interpretability and model-editing studies that identify the final token as the most semantically influential, we inject noise into the embedding of the last token corresponding to the final word of the input prompt. (2) Harmful-token disruption. Since harmful words often play a key role in jailbreak prompts, we prompt an LLM to identify the three most harmful words in the input and inject noise into their corresponding tokens. (3) Fictitious-token disruption. Many jailbreak strategies construct narrative contexts (e.g., “You are a hero preparing to destroy the terrorists’ base”) to bypass safety alignment. We therefore identify key fictitious tokens within such narratives and inject noise into them. (4) Random-token disruption. We randomly select some tokens for noise injection.
(3) Regarding which dimensions are suitable for noise injection, we randomly select a single dimension for perturbation and observe that no specific dimension exerts greater influence on the prompt’s fragility than others. When extending this to multiple dimensions, we find that increasing the number of perturbed dimensions provides no additional benefit over modifying just one. In summary, our results suggest that a prompt’s fragility is largely insensitive to the choice of token dimension; therefore, we randomly select a single dimension for noise injection. (4) When examining the distribution of green, orange, and red regions in these Figures, we find that red regions (gibberish) often correspond to strong noise strengths (||δ||2 > 20), indicating that strong disruption tends to produce nonsensical responses. This pattern is observed for both jailbreak and benign prompts. More importantly, the green (denial) and orange regions (unaffected) are interleaved rather than forming a continuous distribution. This contrasts with our initial expectation of a symmetric pattern, where an orange region corresponds to minor disruptions (e.g., ||δ||2 < 10), surrounded by green regions corresponding to moderate disruptions (e.g., 10 < ||δ||2 < 20), and further surrounded by red regions corresponding to strong disruptions (e.g., ||δ||2 > 20). These observations highlight the need to design a noise strength search algorithm to reliably identify the appropriate value that triggers a denial response.
Third, we investigate which dimensions of the selected tokens should receive injected noise. Specifically, we first examine the case of perturbing a single dimension: we randomly choose one dimension for noise injection and evaluate which dimension exerts greater influence on a prompt’s fragility. Second, we consider perturbing multiple dimensions simultaneously to determine whether disrupting several dimensions is more impactful than perturbing just one. Particularly, we experiment with injecting noise into 5, 10, and 20 randomly chosen dimensions. Fourth, regarding noise strength ||δ||2 , we explore a wide range from -30 to +30, as a strength around 25 typically causes the LLM to produce nonsensical responses. To achieve a comprehensive understanding of the disruption effects, we perform a brute-force search, scanning the entire 3
Re-Triggering Safeguards within LLMs for Jailbreak Detection
2.2. Search of Appropriate Noise We have observed that only an appropriate noise can elicit a denial response from the LLM. This raises an intriguing question: what intrinsic properties does such suitable noise possess, and how can it be efficiently discovered? To this end, we first perform many brute-force searches (as in the Section 2.1) to gather numerous successful disruption cases (the green regions in these Figures) and then investigate the underlying properties shared by these successful disruptions. In practice, this is achieved by visualizing the successful disruption cases. Specifically, since the noise is injected into the embeddings of the input tokens, we collect the disrupted embeddings from all successful cases and visualize them by converting each back to the nearest token in the vocabulary—termed emb2token operation, which aims to find the nearest token whose embedding is most similar to the disrupted embedding. Surprisingly, we find that the converted tokens follow a long-tailed distribution: the disrupted tokens tend to cluster around a few specific tokens (e.g., ‘oH’, ‘wohl’, ‘irement’, as shown in Table 5) with high probability.
Figure 5. Our anchor embedding guided noise-search algorithm.
causes the LLM to produce a denial response, we classify the input prompt as a jailbreak prompt; otherwise, it is deemed benign. Based on a comprehensive understanding of the disruption effect, our approach injects noise directly into the token embeddings, rather than into the input text or the hidden states of an intermediate layer in the LLM. This design offers two key advantages. First, the disruption strength can be easily controlled in the embedding space. Second, the embedding space possesses inherent semantic structure, allowing the disruption magnitude to reflect the extent of semantic alteration.
This observation suggests the existence of certain anchor tokens or anchor embeddings such that directing the disrupted token’s embedding toward one of these anchors enables us to identify the appropriate noise with remarkable efficiency. It reveals the existence of sensitive axes within the embedding space and provides valuable insight for designing a more efficient noise-search algorithm.
Black-box Defense. Although our defense is developed in a white-box setting, it can be naturally extended to black-box scenarios. Specifically, we accomplish this via disruption transfer. First, we obtain the disrupted embeddings from the surrogate LLM and convert them back to the word level. A straightforward approach is to leverage the emb2token operation (Section 2.2) to convert the disrupted embeddings to their nearest tokens. Subsequently, we convert these tokens into their corresponding words. Second, we replace the original input words with the converted words to construct a disrupted prompt. If this prompt elicits a denial response from the target LLM, we regard the original prompt as a jailbreak prompt for the target LLM.
We propose an efficient two-stage noise-search algorithm as shown in Fig. 5. Given a small set of anchor embeddings {ai }K i=0 , for each embedding e to be disrupted (e.g., the embedding of the last token), we determine a direction from e toward an anchor embedding ai . We then search for all possible noise vectors that move e toward ai with a specified step size. If no noise can elicit the LLM to output a denial response, we iteratively select another anchor embedding and repeat the search. If no suitable noise is found across all anchor embeddings, we proceed to a second stage: a random search within a specified strength interval that continues until the search budget (a predefined number of iterations) is exhausted. By employing a dynamic scheme that progressively narrows the strength interval, this second-stage search remains far more efficient than brute-force search.
Our primary experimental results indicate that this straightforward scheme exhibits limited transfer defense capability. To enhance its transferability, we improve it in two aspects. First, instead of stopping after finding a single suitable noise, we continue the search to identify multiple suitable noises. Second, for each disrupted embedding, we extend the emb2token operation from selecting only the top1 nearest token to selecting the top-K nearest tokens. As a result, we can collect a diverse set of candidate transferred jailbreak prompts. If any of these candidates elicits a denial response from the target LLM, we regard the original prompt as a jailbreak prompt for the target model. Our empirical results demonstrate that this many-shot scheme substantially improves black-box defense performance.
2.3. Our Jailbreaking Detection Approach Building upon the previous observations, we propose a novel embedding disruption–based method for jailbreak detection. Specifically, given any input prompt, we search for an appropriate noise to perturb its embedding. Within a predefined search budget, if a suitable noise vector can be found that 4
Re-Triggering Safeguards within LLMs for Jailbreak Detection
3. Evaluation
In contrast, our method, which follows the detection-based defense paradigm, substantially mitigates this trade-off. Specifically, our approach does not modify the LLM itself but instead determines whether an input is a jailbreak prompt or a benign one. If identified as a jailbreak prompt, the input is directly rejected before reaching the LLM.
3.1. Experimental Setup Dataset and Metrics. We evaluate our jailbreak detection approach on the AdvBench (Zou et al., 2023) and JailbreakBench (Chao et al., 2024a) benchmarks.
We utilize two standard LLM evaluation datasets, InstructionFollow (IFEval) (Zhou et al., 2023) and AlpacaEval (Dubois et al., 2025), to assess model utility. Specifically, dataset IFEval comprises a total of 541 instructions, and we use the prompt-level loose accuracy as the utility metric.
We adopt two distinct evaluation methodologies. The first aligns with existing works, measuring defense effectiveness by the drop in Attack Success Rate (ASR) when the defense method is applied compared to when it is not. The second methodology is more suitable for detectionbased defenses. In this setting, we first employ various attack methods to perform jailbreaking and collect the resulting successful-jailbreaking prompts to construct the evaluation dataset. We then apply our approach to determine how many of these prompts are correctly identified as jailbreaks. For this methodology, we naturally employ two metrics: Detection Rate (DR), which measures the proportion of successful-jailbreaking prompts correctly detected, and False-alarm Rate (FR), which quantifies the proportion of benign prompts mistakenly classified as jailbreaks.
Moreover, we use the IFEval dataset to compute the Falsealarm Rate (FR). Specifically, we assess whether a benign prompt is mistakenly classified as a jailbreak prompt by our approach and report the percentage of such misclassified cases. Defensive Baseline. We compare our jailbreak defense approach against six baseline defenses: Perplexity Filtering (Alon & Kamfonas, 2023), which computes the perplexity of the input prompt, yielding a high value if the sequence lacks fluency; Erase-and-Check (Kumar et al., 2025), which exhaustively searches over substrings to detect adversarial tokens; Paraphrasing (Jain et al., 2023), which employs a secondary LLM to paraphrase input prompts as a preprocessing step; SmoothLLM (Robey et al., 2024), a smoothingbased defense utilizing character-level perturbations; and RESTA (Hase et al., 2025), which extends SmoothLLM by perturbing word embeddings instead of directly perturbing the words.
Jailbreaking Attacks. To demonstrate that our approach is effective against diverse jailbreak attacks, we employ several distinct jailbreak methods for evaluation: (1) GCG, a token-level attack that uses optimization-based search to generate nonsensical adversarial suffixes; (2) PAIR, a prompt-level attack that constructs semantically meaningful jailbreak prompts through an adversarial interplay between an attacker and a target LLM; (3) prompt + Random Search (RS), an adaptive attack that perturbs a few contiguous tokens at a randomly chosen position within the suffix; (4) I-FSJ, an improved few-shot jailbreak method that injects special system tokens and uses demo-level random search to generate semantically meaningful adversarial prompts; and (5) AutoDAN-Turbo, the state-of-the-art jailbreak method capable of compromising most LLMs using multiple adaptive jailbreak strategies.
Large Language Models. Throughout our experiments, we evaluated our approach using three open-source LLMs, Vicuna-13B, LLaMA2-7B and Qwen2.5-7B. Beyond the white-box defense, we further propose a transfer-based black-box defense evaluated on GPT-4.1 and Gemini-2.5. 3.2. Main Results
Furthermore, we also consider the adaptive jailbreak attack setting in our evaluation. Adaptive attacks assume that the adversary has knowledge of the target defense and can adaptively modify established jailbreak techniques to bypass it. In practice, we adapt PAIR and AutoDAN-Turbo methods to attack the defended LLM (equipped with our defense scheme). Clearly, adaptive attack settings provide a more rigorous evaluation of a defense method’s effectiveness.
As shown in Table 1, our approach significantly outperforms state-of-the-art defense methods in terms of both defensive effectiveness and model utility. Even under the strongest AutoDAN-Turbo attack, it successfully reduces the ASR from 60% to 29% for Qwen2.5. As shown in Table 3, our approach achieves a high jailbreaking detection rate while maintaining a low false-alarm rate. We further evaluate the defense methods under adaptive attack settings, where the attacker explicitly adapts attack approaches to the deployed defense. As shown in Table 2, our approach maintains strong performance even in this more challenging setting.
Model Utility. A typical challenge in jailbreak defense lies in the trade-off between defense effectiveness and model utility. Most existing defense methods improve defense effectiveness at the cost of degrading model utility, i.e., diminishing the quality of the LLM’s responses to benign queries.
For the black-box defense scenario, we evaluate our disrup5
Re-Triggering Safeguards within LLMs for Jailbreak Detection Table 1. Comparison of defense methods with respect to the defense effectiveness (ASR) and model utility (ACC). Models
Attack Methods
Defense Methods
PAIR
RS
I-FSJ
AutoDAN-T
Alpaca
IFEval
Vicuna-13B
Vanilla Perplexity Filter Erase-and-Check SmoothLLM RESTA Ours
80 3 17 4 2 1
69 69 6 55 30 0
89 88 24 68 44 11
95 92 21 55 35 0
75 75 37 52 44 26
61.5 61.5 49.0 27.8 50.3 61.2
47.0 46.4 26.3 24.0 27.5 47.0
LLaMA2-7B-Chat
Vanilla Perplexity Filter Erase-and-Check SmoothLLM RESTA Ours
3 1 1 0 0 0
0 0 0 0 0 0
90 73 25 0 0 0
98 98 12 84 15 0
31 31 14 24 20 10
53.2 53.2 38.8 14.1 48.9 52.5
38.3 37.9 21.4 18.7 36.8 37.5
Qwen2.5-7B-Instruct
Vanilla Perplexity Filter Erase-and-Check SmoothLLM RESTA Ours
19 2 10 12 8 0
31 31 6 25 22 4
85 71 29 72 50 2
91 91 24 60 38 1
60 60 34 53 49 29
68.7 68.7 52.5 17.2 59.6 68.7
61.6 58.1 34.4 31.3 57.0 61.6
Table 2. Comparison of defense methods under adaptive attacks in terms of defense effectiveness (ASR). Defense Vanilla Perplexity Filter Erase-and-Check SmoothLLM RESTA Ours
Vicuna-13B PAIR AutoDAN-T 69 69 18 63 36 2
Model Utility
GCG
75 75 50 52 48 33
LLaMA2-7B PAIR AutoDAN-T 0 0 0 2 2 0
31 31 23 28 25 17
Table 3. Comparison of different models in terms of Detection Rates and False-alarm Rates.
Qwen2.5-7B PAIR AutoDAN-T 31 31 6 29 24 4
Detection Rate
Models
60 60 41 49 46 30
Vicuna-13B LLaMA2-7B-Chat Qwen2.5-7B-Instruct
False-alarm Rate
GCG
PAIR
RS
I-FSJ
AutoDAN-T
Alpaca
IFEval
0.99 1 1
1 1 0.87
0.88 1 0.98
1 1 0.99
0.65 0.68 0.52
0.01 0.01 0
0 0.02 0
Table 4. Black-box defense setting. Our disruption transfer approach can effectively detect jailbreaking prompts for both the GPT-4.1 and Gemini-2.5 models.
tion transfer strategy. Specifically, we consider two settings. The first is the self-transfer setting, where the target LLM is identical to the surrogate LLM (rows 1–3 in Table 4). Our approach demonstrates strong transferability in this setting. In particular, by transferring successful embeddinglevel disruptions to input-level disruptions, we still achieve a DR = 0.71 on Qwen2.5 against AutoDAN-Turbo.
Transferred Detection Rate
Models Vicuna-13B LLaMA2-7B-Chat Qwen2.5-7B-Instruct GPT-4.1 Gemini-2.5
The second is the true transfer-based setting (rows 4–5 in Table 4). We consider both one-shot and many-shot transfer-based variants. The many-shot variant substantially improves transferability, e.g., achieving a DR = 0.76 on Gemini-2.5 against AutoDAN-Turbo. A detailed comparison between the one-shot and many-shot black-box defenses is provided in the Appendix.
GCG
PAIR
RS
I-FSJ
AutoDAN-T
0.87 / 0.84 / /
0.72 / 0.89 0.71 0.83
0.71 1 0.84 / /
0.74 0.79 0.81 0.68 0.79
0.75 0.85 0.71 0.56 0.76
To determine which layer of the LLM is the most suitable for noise injection, we compare input-level injection, embedding-layer injection, and hidden-state injection. As shown in Fig. 6, hidden-state injection proves unsuitable: it demands a very large perturbation strength and seldom elicits a denial response, as evidenced by the diminutive green regions. We find that the deeper the layer into which noise is injected, the less likely it is to elicit a denial response. This is probably because injecting noise into shallower layers amplifies the disruption effect through the cumulative propagation of noise across subsequent layers.
Besides jailbreak defense methods, we also compare our approach with guard model-based methods such as LLaMA Guard 3, and our method significantly outperforms them. The corresponding results are provided in the Appendix. 3.3. Discussion
For input-level injection, the disruption strength is more difficult to control compared to other methods, as word-level alterations do not correspond directly to semantic perturbations. For instance, deleting the word “not” in a sentence
Effect of Prompt Disruption. Through extensive experiments, we evaluate multiple prompt-disruption options and gain a comprehensive understanding of disruption effects. 6
Re-Triggering Safeguards within LLMs for Jailbreak Detection
Figure 6. The comparison of noise injection applied to inputlevel, embedding-layer, and hidden-state strategies. For input-level disruption, the disruption strength is measured by the characterperturbation ratio, following the SmoothLLM (Robey et al., 2024).
has a far greater semantic impact than removing an article such as “a.” In contrast, the token embedding space possesses inherent semantic structure, allowing the disruption magnitude to reflect the extent of semantic alteration. Thus, we adopt embedding-layer injection in our approach.
Figure 7. The comparison of our defense against different jailbreaking methods: GCG (Row1,2); PAIR (Row3,4); RS (Row5,6); I-FSJ (Row7,8); AutoDAN-Turbo (Row9,10).
As illustrated in Fig. 4, disrupting the last token proves to be the most effective among the four strategies. Moreover, the Fictitious-token strategy slightly outperforms the Harmfultoken strategy, while the Random-token strategy yields comparable results to the latter. Ultimately, we adopt a composite strategy that combines the Last-token and Random-token approaches. We choose the Random-token strategy over the Harmful-token and Fictitious-token ones, as the latter two require explicit identification of specific tokens.
Table 5. Anchor tokens for Vicuna, LLaMA2, and Qwen2.5.
Notably, all the preceding experiments are conducted across numerous jailbreaking prompts and diverse attack methods, indicating that our findings are universally observed. Fig. 7 presents two representative examples for the jailbreak methods GCG, PAIR, RS, I-FSJ and AutoDAN-Turbo. The predominance of the green region across the bars indicates that this option markedly eases the discovery of an appropriate disruptive noise. Visualization of Disrupted Embedding. One key contribution of our approach lies in uncovering the intrinsic properties of suitable noise, which in turn enables the design of an efficient noise-search algorithm. Specifically, through brute-force search, we collect 1, 000 successful disruption cases and visualize the corresponding disrupted embeddings by converting each to its nearest token via the emb2token operation (Section 2.2).
Models
Token ID
Token
Percentage
LLaMA2 / Vicuna
15779 25304 19211 3845 21345 /
oH wohl irement vin )}} Others
0.5210 0.2661 0.1100 0.0209 0.0183 0.0637
Qwen2.5
61183 9568 14758 1365 882 /
Ġaquarium Ġ})ĊĊ Ġexplains ĠâĢˆ Ġtime Others
0.5859 0.1484 0.1353 0.0742 0.0192 0.0370
two dominant patterns emerge. In the first case, the converted token remains identical to the original token, typically corresponding to a small disruption strength. In the second case, the converted token belongs to a small set of anchor tokens. In our experiments, we identify these anchor tokens and present the distribution statistics in Table 5. We find that a converted token corresponds to the top-1 anchor token with a probability of 52%, and to one of the top-3 anchor tokens with a probability of 90%. In other words, the converted tokens exhibit a long-tailed distribution. It is precisely this long-tailed distribution phenomenon that enables us to design a highly efficient noise-search
Since the original tokens originate from user queries and thus can be any token in the vocabulary, one might expect that, after injecting noise, the converted tokens would also distribute uniformly across the vocabulary. However, upon examining their distribution, we make a striking observation: 7
Re-Triggering Safeguards within LLMs for Jailbreak Detection 80
Table 6. Comparison between our random search and brute-force search across different models. The upper half reports the average search counts required to reach 90% of the final detection rate (denoted as “(0.9)”), while the lower half shows the average search counts needed to achieve 100% of the final detection rate.
Normal Tokens Anchor Tokens
60 40 20 0
Search Methods
Vicuna
Llama2
Qwen2.5
-20
Random Search(0.9) Brute-force Search(0.9)
4.49 4.61
1.28 1.57
4.55 5.62
Random Search Brute-force Search
10.50 19.67
2.65 5.68
18.00 29.12
-40 -60
-60
-40
-20
0
20
40
60
Figure 8. Visualization of anchor and normal token embeddings in the latent space using t-SNE. 1.0
Efficiency of Noise Searching. We propose an efficient noise-search algorithm to identify suitable noise capable of prompting the LLM to produce a denial response. Specifically, the procedure consists of two stages. In the first stage, we perform a guided search using the identified anchor tokens, as illustrated in Fig. 5. Each anchor token is treated as a destination, and the path from the input token to the anchor token is divided into four segments. This generates four potential disrupted tokens, each of which is then evaluated to determine whether it can elicit a denial response from the LLM. We iterate through all anchor tokens, and if a suitable noise is still not found, we proceed to the second stage: random search.
Detection Rate
0.8 0.6 0.4 0.2 0.00
10
20
30
Search Iterations
40
Vicuna_GCG Vicuna_PAIR Vicuna_RS Vicuna_I-FSJ Vicuna_AutoDAN-T LLaMA_RS LLaMA_I-FSJ LLaMA_AutoDAN-T Qwen_GCG Qwen_PAIR Qwen_RS Qwen_I-FSJ Qwen_AutoDAN-T
50
Figure 9. Progress of detection rate with respect to the search budget. The x-axis denotes the number of search iterations.
Our preliminary study of the disruption effect indicates that the green and orange regions—corresponding to denial and unaffected responses—are interleaved. Consequently, a brute-force search with small steps is inefficient. To address this, we propose a random search that selects a disruption strength uniformly within a specified range. The initial range is set from -30 to +30. Since strong disruptions often yield nonsensical responses (corresponding to the red region), we can dynamically reduce the strength range whenever they occur. This allows us to perform random search while efficiently narrowing the search space.
algorithm, described in Section 2.2. Notably, the anchor tokens are model-dependent. As shown in Table 5, the anchor tokens identified for Llama2 differ from those for Qwen2.5, whereas Llama2 and Vicuna share the same set of anchor tokens. This is because Llama2 and Vicuna employ the same tokenizer, while Qwen2.5 utilizes a distinct one. Importantly, anchor tokens only need to be identified once for each LLM model. This identification procedure is also efficient in practice: we perform a bruteforce search over noise strengths on a total of 30 prompts (6 prompts per jailbreak method, including GCG, PAIR, RS, I-FSJ and AutoDAN-Turbo), where each prompt takes approximately one GPU hour on a single NVIDIA RTX 3090. Across all prompts, we collect around 1, 000 successful disruption cases, which is sufficient to reliably identify the anchor tokens. Furthermore, anchor tokens exhibit strong reusability across models within the same model family. Specifically, we directly applied the anchor tokens identified on Qwen2.5-7B-Instruct to Qwen3-8B and achieved an 87% defense success rate, likely due to their belonging to the same model family and the similarity of their tokenizers.
In Fig. 9, we illustrate how the detection rate (DR) evolves with respect to the number of search iterations. Evidently, the process unfolds in two distinct stages: during the first stage (iter < 16), the DR rises sharply, demonstrating the high effectiveness of our guided search. Moreover, with a modest search budget (e.g., 50 iterations), the DR already reaches as high as 90%. The results for larger search budgets (e.g., 300 iterations) provided in the Appendix. In addition, we compare our random search approach with brute-force search in Table 6. The results clearly demonstrate that our method is significantly more efficient than brute-force search.
In addition, we directly visualize the original and anchor tokens in the embedding space, as shown in Fig. 8. The t-SNE plot reveals that the anchor embeddings reside within dense regions of the embedding space, suggesting the feasibility of finding small perturbations sufficient for effective jailbreak defense.
4. Conclusion This paper proposes a jailbreak detection approach that aims to re-trigger the LLMs’ built-in safeguards, rather than introducing an additional standalone defense mechanism. 8
Re-Triggering Safeguards within LLMs for Jailbreak Detection
The key contribution of our work lies in the discovery that jailbreaking prompts generated by existing methods are inherently fragile. Through extensive analysis, we gain a comprehensive understanding of the disruption effects and, based on these insights, develop an efficient noise-search algorithm capable of identifying appropriate noise that induce a denial response from the LLM. Extensive experiments demonstrate that our approach effectively defends against state-of-the-art jailbreak attacks in white-box and black-box settings, and remains robust even against adaptive attacks.
Hase, R., Rashid, M. R. U., Lewis, A., Liu, J., Koike-Akino, T., Parsons, K., and Wang, Y. Smoothed embeddings for robust language models, 2025. URL https://arxiv. org/abs/2501.16497. Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., yeh Chiang, P., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T. Baseline defenses for adversarial attacks against aligned language models, 2023. URL https://arxiv.org/abs/2309.00614. Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., and Goldstein, T. On the reliability of watermarks for large language models, 2024. URL https://arxiv.org/ abs/2306.04634.
Impact Statement This paper studies a detection-based defense against jailbreaking attacks on large language models by re-triggering the model’s built-in safeguards through controlled embedding disruptions. The proposed approach improves practical safety by reducing the success rate of diverse jailbreak attack strategies while preserving model utility. At the same time, stronger defenses may incentivize adversaries to develop more adaptive and stealthy jailbreak strategies, potentially accelerating the ongoing arms race between attacks and defenses.
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretraining language models with human preferences. In International Conference on Machine Learning, pp. 17506– 17533. PMLR, 2023. Kumar, A., Agarwal, C., Srinivas, S., Li, A. J., Feizi, S., and Lakkaraju, H. Certifying llm safety against adversarial prompting, 2025. URL https://arxiv.org/abs/ 2309.02705.
References Alon, G. and Kamfonas, M. Detecting language model attacks with perplexity, 2023. URL https://arxiv. org/abs/2308.14132.
Liu, X., Li, P., Suh, E., Vorobeychik, Y., Mao, Z., Jha, S., McDaniel, P., Sun, H., Li, B., and Xiao, C. Autodanturbo: A lifelong agent for strategy self-exploration to jailbreak llms, 2025a. URL https://arxiv.org/ abs/2410.05295.
Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking leading safety-aligned llms with simple adaptive attacks, 2025. URL https://arxiv.org/abs/ 2404.02151.
Liu, Y., Gao, H., Zhai, S., He, Y., Xia, J., Hu, Z., Chen, Y., Yang, X., Zhang, J., Li, S. Z., Xiong, H., and Hooi, B. Guardreasoner: Towards reasoning-based llm safeguards, 2025b. URL https://arxiv.org/abs/ 2501.18492.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G. J., Tramer, F., Hassani, H., and Wong, E. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024a. URL https://arxiv.org/abs/2404.01318.
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. URL https://arxiv.org/ abs/2402.04249.
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries, 2024b. URL https: //arxiv.org/abs/2310.08419.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URL https://arxiv. org/abs/2404.04475.
Provilkov, I., Emelianenko, D., and Voita, E. Bpe-dropout: Simple and effective subword regularization, 2020. URL https://arxiv.org/abs/1910.13267. 9
Re-Triggering Safeguards within LLMs for Jailbreak Detection
Robey, A., Wong, E., Hassani, H., and Pappas, G. J. Smoothllm: Defending large language models against jailbreaking attacks, 2024. URL https://arxiv.org/abs/ 2310.03684. Team, L. The llama 3 herd of models, 2024. URL https: //arxiv.org/abs/2407.21783. Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023. Yang, W., Li, L., Zhang, Z., Ren, X., Sun, X., and He, B. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models, 2021. URL https://arxiv.org/abs/2103.15543. Zheng, X., Pang, T., Du, C., Liu, Q., Jiang, J., and Lin, M. Improved few-shot jailbreaking can circumvent aligned language models and their defenses. Advances in Neural Information Processing Systems, 37:32856–32887, 2024. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models, 2023. URL https: //arxiv.org/abs/2311.07911. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
10
Re-Triggering Safeguards within LLMs for Jailbreak Detection
A. More Results Comparison with guard-model-based methods. We further compare our defense with some guard model-based methods, including LLaMA-Guard-3 (Team, 2024) and GuardReasoner (Liu et al., 2025b). As shown in Table 7, defense effectiveness is evaluated on JailbreakBench against PAIR and AutoDAN-T attacks, where a higher detection rate reflects stronger defensive capability. Model utility is assessed on the AlpacaEval and IFEval datasets, where a lower false-alarm rate is preferred. Our approach surpasses LLaMA-Guard-3 in both defense effectiveness and model utility. Although it performs comparably to GuardReasoner in terms of defense effectiveness, it markedly outperforms GuardReasoner with respect to model utility. Therefore, our method provides a stronger tradeoff, achieving high detection accuracy on adversarial benchmarks while keeping false alarms substantially lower than guard-model-based approaches. Table 7. Comparison between LLaMA-Guard-3, GuardReasoner, and our method on four benchmarks. PAIR and AutoDAN-T measure detection rate where higher is better. AlpacaEval and IFEval measure false-alarm rate where lower is better. Our method achieves strong detection performance on adversarial benchmarks while maintaining the lowest false-alarm rates across all evaluated models. PAIR
Defense Llama-Guard-3 GuardReasoner Ours
AutoDAN-T
Alpaca
IFEval
Vicuna
Llama2
Qwen2.5
Vicuna
Llama2
Qwen2.5
Vicuna
Llama2
Qwen2.5
Vicuna
Llama2
Qwen2.5
0.74 0.99 1
1 1 1
0.61 0.97 0.87
0.49 0.76 0.65
0.32 0.68 0.68
0.18 0.65 0.52
0 0.38 0.01
0 0.38 0.01
0 0.38 0
0.04 0.24 0
0.04 0.24 0.02
0.04 0.24 0
Comparison between the one-shot and many-shot black-box defenses. We compare the performance of one-shot and many-shot black-box defenses in terms of DR on both PAIR and AutoDAN-Turbo attacks, as shown in Table 8. Specifically, GPT-4.1 achieves a DR of 0.71 and 0.56 on PAIR and AutoDAN-T, respectively, under the many-shot setting, compared to 0.33 and 0.18 in the one-shot setting. Similarly, Gemini-2.5 performs better with a DR of 0.83 and 0.76 under many-shot defenses, compared to 0.27 and 0.22 in the one-shot case. Table 8. Comparison between the one-shot and many-shot black-box defenses.
One-Shot
Models GPT-4.1 Gemini-2.5
Many-Shot
PAIR
AutoDAN-T
PAIR
AutoDAN-T
0.33 0.27
0.18 0.22
0.71 0.83
0.56 0.76
Progress of detection rate with a search budget of 300 iterations. In Fig. 10, we show the evolution of the detection rate (DR) with respect to the number of search iterations. The process unfolds in two distinct stages: in the first stage (iter < 16), the DR increases sharply, reflecting the high effectiveness of our guided search. Moreover, with a modest search budget (e.g., 50 iterations), the DR already reaches 90%. With a larger search budget (e.g., 300 iterations), the DR continues to increase, as shown in Fig. 10.
1
Re-Triggering Safeguards within LLMs for Jailbreak Detection
1.0
Detection Rate
0.8 0.6 0.4 0.2 0.00
50
100
150
200
Search Iterations
Vicuna_GCG Vicuna_PAIR Vicuna_RS Vicuna_I-FSJ Vicuna_AutoDAN-T LLaMA_RS LLaMA_I-FSJ LLaMA_AutoDAN-T Qwen_GCG Qwen_PAIR Qwen_RS Qwen_I-FSJ Qwen_AutoDAN-T
250
300
Figure 10. Detection rate evolution with respect to search iterations. The DR increases sharply in the initial iterations and continues to improve with a larger search budget.
2