One Step to the Side: Why Defenses Against Malicious Finetuning Fail Under Adaptive Adversaries
Itay Zloczower1 , Eyal Lenga1 , Gilad Gressel2 , Yisroel Mirsky1 Ben-Gurion University of the Negev, 2 Amrita Vishwa Vidyapeetham, Amritapuri {itayzloc, lenga}@post.bgu.ac.il, [email protected], [email protected]
arXiv:2605.14605v1 [cs.CR] 14 May 2026
1
Abstract Model providers increasingly release open weights or allow users to fine-tune foundation models through APIs. Although these models are safety-aligned before release, their safeguards can often be removed by fine-tuning on harmful data. Recent defenses aim to make models robust to such malicious fine-tuning, but they are largely evaluated only against fixed attacks that do not account for the defense. We show that these robustness claims are incomplete. Surveying 15 recent defenses, we identify several defense mechanisms and show that they share a single weakness: they obscure or misdirect the path to harmful behavior without removing the behavior itself. We then develop a unified adaptive attack that breaks defenses across all defense mechanisms. Our results show that current approaches do not provide robust security; they mainly stop the attacks they were designed against. We hope that our unified adaptive adversary for this domain will help future researchers and practitioners stress-test new defenses before deployment.1
1
Introduction
Open-weight and fine-tuning-enabled language models create a post-release safety problem. After a model has been aligned, downstream users are still able to update it. In open-weight releases, the user receives the parameters directly; in fine-tuning APIs, the user supplies data that drives provider-side updates. In both settings, the user obtains access to the same primitive used to install alignment in the first place: gradient-based optimization. This makes safety alignment vulnerable to being overwritten after release. Prior work shows that small malicious fine-tuning runs can subvert aligned models and substantially recover harmful behavior [44, 19]; even benign or non-malicious fine-tuning can degrade safety alignment [31]. This malicious finetuning (MFT) is not merely an implementation bug in a particular model family. It reflects a structural asymmetry: alignment is applied before release, but the attacker fine-tunes after release. If alignment modifies behavior without removing the underlying capability, then later behavioral training can recover what alignment suppressed. This risk has given rise to an emerging research area on defenses against malicious fine-tuning: methods that aim to make aligned models robust to downstream attempts to recover harmful behavior. These MFT defenses attempt to make harmful fine-tuning fail, require prohibitive optimization effort, or cause the model to lose general capability when the attack succeeds. The proposed mechanisms vary widely. Some harden internal representations against harmful updates [17, 33, 22, 6, 29]; some simulate adversarial fine-tuning during alignment [14, 37, 34]; some reshape the loss landscape around safety-relevant parameters or optimization trajectories [16, 32, 11, 28]; some reweight or curate alignment data [24, 23, 6]; and some deliberately couple harmful fine-tuning to catastrophic utility collapse [7, 46, 41]. Despite this diversity, existing evaluations share a common pattern: the attacker is usually a fixed, ignorant supervised fine-tuning procedure with a prescribed loss, optimizer, data budget, and hyperparameter range [31, 17, 33, 37, 16]. 1 Code will be released upon publication.
Preprint.
This evaluation practice conflicts with a central lesson from adversarial machine learning: defenses must be evaluated against adaptive adversaries. An adaptive adversary knows the defense, understands the training objective, and selects an attack that exploits this knowledge. Without such an evaluation, a defense may only show that it blocks the specific attack considered during its design. This failure mode is well documented in adversarial examples, where defenses that appeared robust against standard attacks were later broken by attacks adapted to the defense mechanism [4, 1, 40]. In the MFT setting, this is a critical gap. Recent defenses aim to make malicious fine-tuning fail, yet they have largely been evaluated against fixed, non-adaptive fine-tuning procedures. To our knowledge, no prior work has systematically defined what an adaptive adversary means for MFT defenses, instantiated such an adversary, nor evaluated an adaptive adversary on these defenses. As a result, current robustness claims remain difficult to validate, and the community lacks a reliable benchmark for testing defenses under a meaningful post-release threat model. Many defenses, one shared vulnerability. A successful malicious fine-tuning attack must make the model harmful while preserving its general capabilities. A defense can therefore succeed by disrupting either condition. We find that fifteen recent MFT defenses fall into two broad strategies: anchoring, which makes optimization on harmful objectives ineffective, and self-destruction, which permits optimization but causes the model’s capabilities to collapse. Despite their surface differences, these defenses reduce to four loss templates that implement these two strategies. Our analysis of these templates reveals a shared oversight: existing defenses are evaluated against attackers that optimize only for harmfulness, rather than the full attack objective. This leaves open a simple adaptive strategy. Because successful attacks require both harmfulness and capability to be preserved, an adversary can add a benign capability-preservation loss during fine-tuning. This signal helps the model avoid the traps created by both anchoring and self-destruction defenses: it preserves utility while allowing harmful behavior to re-emerge. We show that this adaptive attack (called S IDESTEPPER) breaks all defense methods, suggesting that the vulnerability comes not from any single defense design, but from a common assumption about the attacker. To our knowledge, this is the first work to systematically define, instantiate, and evaluate an adaptive adversary for malicious fine-tuning. We also identify a fundamental locality issue in existing defenses that explains why our attack is effective. To support this claim, we also demonstrate a second adaptive attack (called K ICK -S ETTLE) that evades these defenses by escaping the locality of their intended traps. Finally, we provide a simple test that future work can use to evaluate the security of proposed MFT defenses. This work makes three contributions. (1) We systematize 15 recent malicious fine-tuning defenses (10 since 2025) and show that, despite their apparent diversity, they collapse into four loss templates and two underlying security strategies. This compression reveals that the field has converged on the same small set of ideas without recognizing it, and that every defense inherits the same evaluation gap: each is tested only against an attacker who optimizes for harmful recovery alone. As a result, these defenses do not provide meaningful security against an adversary that wants to perform MFT. (2) We introduce the concept and model of adaptive adversaries for malicious fine-tuning, defining the post-release threat model the field has implicitly been avoiding: an attacker who chooses the data, optimizer, and objective in response to the defense. (3) We propose two simple yet highly effective adaptive attack algorithms that defeat every evaluated defense, regardless of its underlying strategy or mechanism. We recommend that future work proposing MFT defenses evaluate against our attack before making robustness or security claims.
2
Related Work
Durable safeguards for open-weight models. A growing line of work studies whether safety interventions can remain effective after a model is released as open weights or exposed to downstream fine-tuning. Standard refusal training can often be weakened or removed by continued optimization, motivating defenses that aim to make harmful capabilities difficult to recover after release [31, 45, 48, 42]. Recent methods pursue this goal through representation noising, tamper-resistant training, adversarial unlearning, and other training-time interventions intended to make malicious fine-tuning fail or degrade utility [33, 37, 20, 17]. 2
The closest prior work is Qi et al. [30], which shows that MFT defenses can be far more brittle than their original evaluations suggest: safeguards that survive one fine-tuning setup may fail under small changes to dataset shuffling, trainer implementation, learning-rate schedule, prompt formatting, or fine-tuning data. This provides an important cautionary lesson for model release: defenses should be stress-tested across plausible downstream settings before releasing model weights. Our work studies a stronger failure mode. Durability tests ask whether a defense survives routine variation in the fine-tuning setup. We show that even defenses that pass such tests can fail once the attacker is adaptive. Thus, evaluating MFT defenses requires more than repeating malicious fine-tuning under different configurations; it requires testing adversaries that deliberately change the optimization objective to bypass the defense. Adaptive evaluation in adversarial machine learning. Our work follows a central lesson from adversarial machine learning: robustness claims based on static, transfer, or otherwise non-adaptive attacks often fail under adversaries that optimize against the defense itself [2, 36, 12, 3, 1, 40]. This lesson has been repeatedly rediscovered across domains, where defenses that appear effective against fixed attacks are broken once the attacker incorporates the defense mechanism into the objective, bypasses non-differentiable components, changes the optimization procedure, or searches over a richer attack class [3, 1, 40, 26]. The methodological point is that evaluation attacks must be constructed with respect to the mechanism by which the defense claims to obstruct the adversary. We instantiate this principle for MFT defenses by identifying their shared obstruction mechanisms and deriving attacks that exploit them. Adaptive attacks in LLM security. Similar evaluation failures have recently appeared in LLM security. Jailbreak and prompt-injection defenses are often evaluated against fixed benchmark attacks, static prompt sets, or generic optimization methods that are not tuned to the defense under evaluation. Recent work shows that such defenses can appear robust under these evaluations while failing against attackers that adapt their search procedure, feedback signal, or human strategy to the defense [27, 49, 5, 25, 10]. Our setting differs in the object being attacked. Rather than constructing adversarial prompts against a deployed interface, we study adversaries who directly modify model weights after release. In this setting, adaptivity means choosing the post-release optimization process itself, including the data mixture, loss, optimizer, schedule, and recovery objective. This changes the structure of the attack problem: the relevant question is not whether a defense survives stronger malicious fine-tuning, but whether it survives fine-tuning designed to counter the defense mechanism. Our work addresses this gap by systematizing how MFT defenses obstruct naive harmful fine-tuning and deriving adaptive attacks from those obstruction mechanisms.
3
Systematizing MFT Defenses
3.1
Problem Setup
The post-release fine-tuning game. Let θ represent the parameter values of the base model and let θdef represent the defended parameter values after being optimized with some defense loss Ldef . We consider a defender that releases an aligned model Mθdef initialized from θ. Let Datk denote the dataset an adversary would use in an MFT attack, and let Latk denote the loss function the same adversary has configured for the attack, with the learning rate being η. After release, an attacker with white-box access applies a fine-tuning algorithm θatk = A(θdef ; Datk , Latk , η, T ). Let Lh (θ) denote harmful-behavior loss (low values mean the model emits the suppressed behavior) and Lc (θ) denote the models capability loss (low values mean the model is coherent or simply useful on benign tasks). A successful MFT attack must satisfy both clauses: Lh (θatk ) ≤ τh
and
Lc (θatk ) ≤ τc
(1)
where τh and τt are acceptable performance thresholds based on a hold out set. The conjunction matters: a model that emits harmful content but loses fluency or coherence or capability (i.e., intelligence) is not a useful recovered model. Why malicious fine-tuning works. Safety alignment is typically behavioral rather than capabilityremoving: it suppresses the disclosure of capabilities without erasing them [31]. A weight setting θ⋆ with low Lh and low Lc therefore generally exists, and ordinary harmful supervised fine-tuning succeeds whenever gradient descent from θdef can reach such a region. The defender’s goal is to 3
release a model that is aligned (Lh (θdef ) >> τh ) and robust to every feasible attacker: given the set of all possible attackers A, attacker Ai (θdef ) ∈ A violates (1). Because A cannot be enumerated, existing defenses replace this requirement with tractable assumptions on what the adversary will do. We show next that as a result, existing works fall into two defense strategies. 3.2
Fifteen Defenses, Two Strategies
Existing MFT defenses look diverse: some perturb embeddings, some scrub harmful representations, some simulate future fine-tuning, some manipulate low-rank subspaces, some make harmful adaptations to destroy utility. But all of them defend against the same post-release primitive (the attacker updates θdef weights by descending a loss Latk ) and they break that primitive in only two ways. Strategy A: anchoring. Anchoring defenses keep θdef in place. They shape the local landscape around θdef so that descending Lh produces no useful progress: the harmful gradient is small, randomized, or pointed away from θ⋆ , and even iterated descent stays on a plateau where harmful loss does not drop: ∥∇Lh (θdef )∥ ≈ 0
or
⟨∇Lh (θdef ), θ⋆ − θdef ⟩ ≥ 0
or
Lh (Ak (θdef )) > τh .
(2)
The attacker’s trajectory either stalls, drifts unproductively, or wanders on a plateau; θ never reaches a region with low Lh . This family spans five mechanisms. Representation anchoring flattens or randomizes harmful hidden states: Vaccine and T-Vaccine harden hidden states against harmful embedding perturbations [17, 22], and RepNoise pushes harmful activations toward Gaussian noise [33]. Direction anchoring suppresses the harmful gradient at θ itself: Booster and Antibody attenuate harmful gradient influence [16, 28], VAA reinforces vulnerable safety subgroups [6], and SAM-unlearning enforces local smoothness against relearning [11]. Subspace anchoring closes off low-rank attack routes: LoX extrapolates safety-critical weights into a flatter region, and AntiDote trains against an adversarial hypernetwork that generates worst-case LoRA patches [29, 34]. Objective anchoring reshapes the alignment objective itself, as in KT-IPA [8]. Trajectory anchoring extends the same idea from a single step to a simulated attack trajectory: MLAC and TAR meta-train against K inner adaptation steps so that the post-trajectory endpoint still has high harmful NLL or maximum posterior entropy on harmful prompts [14, 37]. Mechanically these are look-ahead methods, but their security argument is the same as the single-step anchors: the attacker’s optimizer finds no descent direction on Lh , only now the guarantee holds along a K-step trajectory rather than at a single θ. Strategy B: self-destruction. Self-destructive defenses do not hide the harmful gradient or block progress on Lh . They let the attacker descend freely and do reach low harmful loss, but engineer the trajectory’s endpoint θT so benign capability collapses there: Lh (θT ) ≤ τh
but
Lc (θT ) ≫ τt .
(3)
Sub-strategy
Defense
Year Template
Strategy A: Anchoring Representation Vaccine [17] T-Vaccine [22] RepNoise [33] Direction VAA [6] SAM-unlearning [11] Antibody [28] Booster [16] Subspace LoX [29] AntiDote [34] Objective KT-IPA [8] Trajectory MLAC [14] TAR [37]
2024 2025 2024 2025 2025 2026 2024 2025 2026 2025 2023 2024
T1 T1 T2 T1 T1 T1 + T3 T3 T1 T3 T2 + T3 T3 T3
The recovered model emits harmful content on harmful prompts but is unusable on benign ones, failing the joint success condition (1) on the task clause. CTRAP forces θT to predict a fixed error token on benign inputs [46]; SEAM couples adversarial and benign gradients into op- Strategy B: Self-destruction posing directions, so descending Lh is descending −Lc — CTRAP [46] 2025 T3 SEAM [41] 2025 T4 [41]; and SDD trains harmful prompts to map to fluent SDD [7] 2025 T4 but unrelated benign answers, so harmful fine-tuning degrades instruction-following [7]. The three differ in what kind of capability damage they engineer (token-level col- Table 1: MFT defense taxonomy and lapse, gradient-level coupling, output-level incoherence) loss template mapping. but share the same security argument: the harmful-fitting direction in weight space is the utility-destroying direction. Table 1 summarizes the mapping (Extended version in Appendix Table 4). The defenses disagree on implementation, but their security arguments collapse to one of two ideas: anchor θdef so descent on Lh goes nowhere, or let descent run but ensure it lands at a θT where the model has self-destructed. 4
3.3
Four Loss Templates Behind the Taxonomy
The anchoring/self-destruction axis describes the intended failure mode for the attacker. A second axis describes how each defense encodes the future attacker in its training objective. The fifteen defenses fall into four templates. Moreover, every template builds on a standard alignment loss Lalign , whose purpose is to keep the released model both safe and useful. It decomposes as Lalign (θ) = Ls (θ) + Lc (θ): a safety term Ls (refusal on harmful prompts) and a capability term Lc (correct behavior on benign prompts). A defender that dropped Lc would release a useless model, so every defense in our survey includes it. Template 1: Robust-alignment basin. A first family tries to place the released model in a basin where alignment is stable to bounded perturbations: (1)
Ldef (θ) = max Lalign (θ + δ) + λLc (θ). δ∈∆
(T1)
The perturbation set ∆ may live in embedding space, layer space, or weight space. Vaccine uses adversarial hidden-state perturbations; T-Vaccine restricts them to layers selected by harmful-gradient norm; VAA combines group-level robustness with adversarial sampling; SAM-unlearning uses a sharpness-aware neighborhood; and Antibody’s alignment-stage flatness term is a related local robustness objective [17, 22, 6, 11, 28]. LoX is the post-hoc analogue: instead of solving (T1), it extrapolates along an estimated low-rank safety direction to move the model into a flatter region [29]. Template 1 implements anchoring. It does not remove harmful capability globally; it makes the local basin around θdef harder to exit by ordinary harmful fine-tuning. Template 2: Harmful-information removal. A second family attacks the information content of harmful representations: (2)
Ldef (θ) = Lalign (θ) − βLh (θ) + αRpurge (θ; Dh ).
(T2)
Here Rpurge removes useful structure from activations on harmful inputs. RepNoise instantiates this with a noise-matching regularizer, using MMD to push harmful activations toward random noise [33]. KT-IPA includes a related representation purging stage before applying its prospect-theoretic integrity objective [8]. Template 2 also implements anchoring. Rather than flattening a basin, it tries to make the internal features that ∇Lh would manipulate less useful, so the harmful gradient at θdef carries no usable signal. Template 3: Look-ahead defense. The most general template explicitly simulates a future attacker: (3)
Ldef (θ) = Lalign (θ) + λR(θ′ ),
θ′ = SimAttack(θ; Dh ).
(T3)
The simulated attack may be one harmful gradient step, K inner fine-tuning steps, a sampled adversary, or a learned patch. The penalizer R then imposes a desired property at the simulated post-attack point. Template 3 is the bridge between anchoring and self-destruction. If R penalizes harmful progress, preserves refusal, prefers safe responses, or keeps harmful loss high along a simulated trajectory, the method is an anchoring defense. Booster penalizes harmful-loss drop after a simulated harmful step [16]; Antibody preserves refusal at the post-step model [28]; AntiDote trains against activationconditioned adversarial LoRA patches [34]; KT-IPA includes an adversarial phase inside its integrity objective [8]; and MLAC and TAR penalize a K-step simulated trajectory so harmful loss stays high or the predictive distribution stays at maximum entropy [14, 37]. If R instead makes the simulated endpoint useless on benign data, the same template implements self-destruction. CTRAP penalizes a simulated post-harmful-step model to collapse on benign inputs [46]. Thus the same mathematical form can implement either strategy. The difference is what the defender wants to be true at θ′ : a model that still refuses (anchoring), or a model that has lost benign capability (self-destruction). Template 4: Coupling trap. The last template does not simulate the attacker. Instead, it directly couples harmful improvement to benign degradation: (4)
Ldef (θ) = Lalign (θ) + Lcouple (θ; Dh , Dc ).
(T4)
where Dc is typically benign task samples. SEAM implements this explicitly by shaping the relationship between harmful 5
and benign gradients: descent on harmful data is made to increase benign loss [41]. SDD implements a data-level version: it trains harmful prompts to elicit fluent but irrelevant benign answers, so later harmful fine-tuning must undo a response mapping that also damages instruction-following [7]. Template 4 is always self-destruction. It does not try to hide the harmful gradient; it tries to make following it costly.
4
A Unified Adaptive Attack
The shared vulnerability: a fixed attacker objective. The four templates differ in mechanism, but they share the same attacker model: the post-release adversary is assumed to optimize only harmful loss, Lh . This assumption is visible in each template. Template 1 makes a local basin robust to perturbations induced by descent on Lh . Template 2 removes the harmful representations that ∇Lh would use. Template 3 simulates an inner attacker whose loss is Lh or a close proxy. Template 4 couples the harmful-improvement direction to benign degradation, again assuming the attacker follows the Lh direction. In all cases, the defense is optimized against the same naive adversary: Anaive : θdef 7→ arg min Lh (θ; Dh ). θ
(5)
This is not the threat model. A post-release adversary can choose the data, optimizer, schedule, and loss. Nothing requires the attacker to optimize Lh alone. The harm-only objective is therefore a defense-naive evaluation choice: it tests whether the defense blocks one prescribed trajectory, not whether it blocks adaptive malicious fine-tuning. The escape signal. Once the attacker is allowed to choose the loss, the natural adaptive strategy is to add another optimization signal to Lh . The role of this additional term is not to replace the harmful objective, but to change the trajectory that the optimizer follows. This is enough because the defenses’ traps are local. Template 1 constrains a perturbation neighborhood around θdef ; Template 2 purges harmful information at the defended weights; Template 3 regularizes a simulated finite-horizon trajectory; and Template 4 defines a coupling along the harmful-improvement direction. These mechanisms can obstruct the harm-only path, but they do not show that every nearby or reachable path to a harmful-and-useful model is blocked. We choose the benign capability loss Lc as this auxiliary signal. This choice is natural for two reasons. First, a successful malicious fine-tuning attack must recover harmful behavior while preserving model usefulness, as required by the joint success condition in (1). Thus Lc is already part of the attacker’s real objective, even if prior evaluations measure it only after training. Second, as shown in Section 3.3, Lc appears in every defense template through the alignment objective: defenders must preserve benign capability, otherwise they would release a safe but useless model. The same signal that lets the defender keep the model useful is therefore always available to the attacker as a search heuristic for escaping anchors and avoiding self-destruction. The adaptive objective. We therefore attack all templates with the same mixed objective: Latk (θ) = Lh (θ) + λLc (θ),
(6)
starting from θdef . This objective does not change the attacker’s goal; it moves the full success criterion into the optimization. Instead of checking capability only after the attack, the adaptive adversary uses capability loss during the attack to steer the search. Figure 1 illustrates the effect: Lh alone either stalls in an anchor or falls into a self-destructive region, while Lh + λLc guides the optimizer toward parameters that are both harmful and useful. Figure 2 shows the same phenomenon for Vaccine on Qwen3-8B: the naive attacker (harm-only) follows the locally available descent direction and saturates, while the adaptive adversary (mixed objective) opens an off-axis route to lower harmful loss. Full details on this experiment are in Appendix D. Why this breaks anchoring defenses. Templates 1 and 2 try to make the harmful signal unusable near θdef . Template 1 does this by shaping a local robust basin; Template 2 does it by corrupting harmful representations at the defended point. But neither removes the underlying harmful capability globally, and neither makes θdef stationary for the attacker’s downstream capability objective. The adaptive gradient ∇Latk (θ) = ∇Lh (θ) + λ∇Lc (θ) therefore contains a component the anchor was not designed to suppress. The Lc term moves the model while preserving useful behavior. Once the trajectory leaves the local basin or purged 6
Strategy A: Anchor
Strategy B: Self-Destruct good model capability
Loss
good model capability
?
good capability + good misuse
good capability + good misuse
poor capability + good misuse
Parameter weight 𝜃
Parameter weight 𝜃
ℒ𝑐
ℒℎ
ℒatk = ℒ𝑐 +ℒℎ
ℒ ℎ < 0.5
capability loss
harmfulness loss assumed attack
combined loss adaptive attack
where 𝜃 enables misuse
Figure 1: An illustration of how two defense strategies work and why they fail when defenders assume that the attacker optimizes only Lh (red). Anchor: optimization on Lh makes no progress, because the gradient is small, randomized, or points away from any region of low harmful loss, so θ stays in the aligned neighborhood. Self-Destruct: optimization on Lh moves θ to a region where misuse succeeds but capability performance degrades, i.e., Lc is high. Adaptive attack (orange): by optimizing both losses, Lh + Lc , the adversary reaches parameters that enable misuse while preserving model coherence. representation regime, the defense no longer controls the relevant region of the loss surface, and harmful behavior can re-emerge. Why this breaks look-ahead defenses. Template 3 appears adaptive because it simulates an attacker θ′ = SimAttack(θ; Dh ). But the simulation fixes the attack loss, horizon, optimizer family, and often the adaptation form. The outer defense therefore regularizes the endpoint of one chosen proxy attack, not the endpoint of every feasible post-release optimization. When the real attacker minimizes Lh + λLc , it follows a different trajectory from the one the defense simulated. For anchoring instances, the attacker no longer follows the harmful-loss path whose progress was penalized. For self-destructive instances, the capability term directly penalizes the collapsed endpoint the defense tries to induce. In both cases, the adaptive trajectory leaves the truncated look-ahead approximation. Why this breaks self-destruction defenses. Template 4 and the self-destructive instances of Template 3 try to make harmful fine-tuning succeed only by destroying capability. Under the naive objective, this can work: descent on Lh is arranged to increase Lc or damage instruction-following. Under the adaptive objective, that coupling becomes visible to the optimizer. A direction that lowers Lh but sharply raises Lc is not a good descent direction for Lh + λLc . The attacker does not need to explicitly undo the trap; it simply optimizes the real success criterion. A self-destructed model with low harmful loss but high capability loss is exactly what the adaptive objective avoids.
5
Experimental Setup
Setup. We test whether our adaptive attack from (6), called S IDE S TEPPER, is effective across diverse MFT defense strategies. We evaluate defended checkpoints derived from Llama-2-7B-chat [39], Qwen3-8B-Instruct [43], and Llama-3.1-8B-Instruct [13], covering both the older backbone used by much of the MFT-defense literature and newer instruction-tuned models with stronger baseline capability. We selected Booster [16], CTRAP [46], VAA [6], Vaccine [17], Unlearn-Smooth [11], and SDD [7] because they cover both strategies and mechanisms in Table 1. We use author-released implementations, checkpoints, and defense-specific data where available, and otherwise reproduce the defense protocol from the paper (Appendix G). Unlearn-Smooth is the only exception to the 7
Vaccine
h
Vaccine
c
25 3.774
2.748
5.312
h
2.744
5 0
2.401
5 10 10
0
10
20
30
(naïve direction)
40
50
15
2.338
10
c
3.088
10
20 2.543
2.133
5 0
1.928
2.058
5
1.715
10 10
0
10
20
30
40
(naïve direction)
Start (defended model)
Naïve attack
(perpendicular to naïve)
20 3.431
(perpendicular to naïve)
(perpendicular to naïve)
20 15
h+ c
25
4.936
15
4.561
10
h+ c
Vaccine 25
4.185
5 0
3.809
1.722
5
3.434
1.517
10
50
3.058
10
0
10
20
30
(naïve direction)
40
50
Adaptive attack
Figure 2: A visualization of the adaptive attack on a Vaccine-defended Qwen3-8B LoRA loss landscape. The starting parameters θdef is marked by a star; orange shows the naïve Lh only attack trajectory, and purple shows the adaptive Lh + Lc trajectory. Left: under Lh alone, the naïve attack follows the locally steep descent direction but quickly saturates on a plateau. Middle: Lc shows that the naive attacker is unaware of the trap that harms model capability. Right: optimizing over Lh + Lc changes the effective landscape: the adaptive attacker can see not only how to improve misuse capability but also retain model capability.
three-backbone protocol: the authors release a single defended Hugging Face checkpoint, not an instantiable defense recipe, so we evaluate that checkpoint directly. It therefore appears as one model bar, with mean and standard deviation computed over the same three adaptive-attack seeds. Starting from each defended checkpoint, we apply the same adaptive mixed-objective attack from Section 4; we do not tune a separate attack per defense. Llama-2-7B
Qwen3-8B
Strategy A: Anchoring
Benign Improvement
We evaluate harmful recovery on the unsafe split of beavertails_330k_test, scored with the HarmBench classifier[25], and evaluate benign utility with standard zero-shot performance on MMLU [15], TruthfulQA [21], HellaSwag [47], and ARC-Easy [9]. We report harmful-response rate as the harmful score metric and report capability benchmark changes relative to the defended checkpoint before attack. This delta convention isolates the effect of the adaptive finetuning attack from baseline differences across models and defenses. The full data construction, split details, and attack hyperparameters are in Appendix F.
Llama-3.1-8B
zephyr-7b
Strategy B: Self-destruction
0.3 0.2 0.1 0.0 ster
Boo
VAA
ine oth Smo Vacc arne l n U
AP
CTR
SDD
Harmful Score
Strategy A: Anchoring Strategy B: Self-destruction Results. Figure 3 and Table 2 show that the 0.8 adaptive attack increases harmful behavior for every defended checkpoint we evaluate. The 0.6 effect is not a marginal degradation of safety: 0.4 harmful-score deltas are positive across all defenses and models, ranging from 0.200 to 0.738 0.2 (max 1.0), with several defenses recovering 0.0 roughly 0.7 harmful-score points after attack. AP This failure holds for both defense families. For ster VAA Vaccine Smooth SDD CTR Boo arngradient-obfuscation defenses such as Booster, Unle VAA, Vaccine, and Unlearn-Smooth, the benignretention objective supplies a usable optimiza- Figure 3: Top: ∆ benign performance. Bottom: ∆ tion signal and allows harmful behavior to re- harmful score. Error bars: std over 3 seeds. emerge. For gradient-trap defenses such as CTRAP and SDD, the same objective prevents the attack from taking the low-utility collapse path that the defense tries to induce. Benign capability remains stable in nearly all cases. Excluding the
8
Table 2: Adaptive attack. Per-metric deltas (∆ vs. defended baseline) for capability benchmarks and BeaverTails. Each cell reports mean ± std over three seeds. A positive ∆ on BeaverTails means the defense was undone (harm rose); Capability ∆ near zero means the attack preserved utility. Capability (∆) Defense
Model
Booster
Llama-2-7B Qwen3-8B Llama-3.1-8B
CTRAP
MMLU
Harm Score (∆)
HellaSwag
ARC-Easy
Average
BeaverTails
−0.008 ± 0.004 −0.006 ± 0.001 0.010 ± 0.008 −0.011 ± 0.009 −0.010 ± 0.006 −0.001 ± 0.013
0.013 ± 0.005 0.024 ± 0.002 0.020 ± 0.001
0.035 ± 0.003 0.037 ± 0.001 0.014 ± 0.004
0.009 ± 0.002 0.015 ± 0.004 0.006 ± 0.004
0.200 ± 0.009 0.307 ± 0.047 0.416 ± 0.016
Llama-2-7B Qwen3-8B Llama-3.1-8B
0.001 ± 0.007 −0.004 ± 0.010 0.029 ± 0.002 −0.017 ± 0.005 −0.006 ± 0.007 −0.007 ± 0.009
0.036 ± 0.004 0.047 ± 0.006 0.023 ± 0.004
0.061 ± 0.009 0.058 ± 0.003 0.054 ± 0.006
0.023 ± 0.003 0.029 ± 0.001 0.016 ± 0.002
0.229 ± 0.005 0.324 ± 0.022 0.375 ± 0.013
VAA
Llama-2-7B Qwen3-8B Llama-3.1-8B
0.003 ± 0.006 −0.069 ± 0.011 0.071 ± 0.020 −0.008 ± 0.028 −0.007 ± 0.000 −0.101 ± 0.009
0.003 ± 0.006 0.072 ± 0.020 −0.004 ± 0.002
0.054 ± 0.002 −0.002 ± 0.004 0.126 ± 0.038 0.065 ± 0.013 0.038 ± 0.003 −0.018 ± 0.003
0.707 ± 0.027 0.239 ± 0.285 0.705 ± 0.015
Vaccine
Llama-2-7B Qwen3-8B Llama-3.1-8B
−0.001 ± 0.000 −0.100 ± 0.001 0.007 ± 0.004 −0.050 ± 0.018 −0.007 ± 0.005 −0.092 ± 0.008
−0.025 ± 0.001 0.009 ± 0.001 −0.004 ± 0.001
0.054 ± 0.000 −0.018 ± 0.000 0.138 ± 0.013 0.026 ± 0.008 0.034 ± 0.001 −0.017 ± 0.003
0.701 ± 0.006 0.500 ± 0.038 0.707 ± 0.020
0.480 ± 0.018
0.457 ± 0.020
0.263 ± 0.014
0.738 ± 0.009
−0.024 ± 0.002 0.018 ± 0.020 −0.016 ± 0.004 0.012 ± 0.001 0.020 ± 0.007 0.002 ± 0.004 0.001 ± 0.003 0.019 ± 0.002 −0.009 ± 0.005
0.705 ± 0.011 0.691 ± 0.026 0.693 ± 0.036
0.082 ± 0.025
Unlearn-Smooth OPTML-Group/NPO-SAM-WMDP
0.034 ± 0.007
−0.011 ± 0.004 −0.048 ± 0.005 −0.004 ± 0.001 −0.021 ± 0.008 −0.011 ± 0.005 −0.045 ± 0.014
Llama-2-7B Qwen3-8B Llama-3.1-8B
SDD
TruthfulQA
Table 3: Kick-settle attack. We use the same metrics and ∆ convention as Table 2 Capability (∆) MMLU
TruthfulQA
HellaSwag
Harm Score (∆)
Defense
Model
ARC-Easy
Average
SDD
Llama-2-7B −0.037 ± 0.003 −0.042 ± 0.003 −0.041 ± 0.002 −0.008 ± 0.008 −0.032 ± 0.002 Qwen3-8B −0.030 ± 0.010 −0.015 ± 0.022 −0.005 ± 0.010 0.010 ± 0.015 −0.010 ± 0.012 Llama-3.1-8B −0.115 ± 0.040 −0.063 ± 0.013 −0.061 ± 0.025 −0.065 ± 0.012 −0.076 ± 0.022
BeaverTails 0.639 ± 0.014 0.709 ± 0.021 0.662 ± 0.011
released Unlearn-Smooth checkpoint (which demonstrates large gains), the average benign delta lies between −0.018 and 0.065 across all defended checkpoints, and no defense exhibits the broad capability collapse that would indicate a failed or degenerate attack.
6
Discussion & Insights
What unifies the failure. The defenses are mechanistically diverse; they patch embeddings, activations, gradients, low-rank subspaces, simulated trajectories, or output mappings. But each template enforces its intended property only locally: within a perturbation set, at θdef , along a truncated trajectory, or through a coupling defined at the released model. None proves that no parameter setting satisfying (1) remains reachable elsewhere in weight space. The adaptive objective Lh + λLc supplies the missing search pressure: Lc moves the trajectory out of the locally constrained region while keeping it on the manifold of useful models. A single attack family thus covers Vaccine, RepNoise, TAR, CTRAP, and SEAM alike: it does not target their mechanisms, only the shared boundary of their reach. The adaptive objective in (6) is not the only way to exploit this locality failure. To show that the limitation is more general, we also evaluate a second adaptive adversary that still optimizes only Lh , but changes the optimization schedule. This attack begins with a large learning rate, which “kicks” the parameters out of the local trap around θdef , and then rapidly decays to a standard learning rate to “settle” toward a harmful-and-useful solution θ⋆ . We call this attack K ICK -S ETTLE. The results in Table 3 show that it is effective on SDD, confirming that the vulnerability is not specific to adding Lc ; rather, the deeper issue is that the defense only constrains a local region around the expected attack trajectory. Full implementation details are provided in Appendix H. A conjecture, and an uncomfortable open problem. The broader lesson is not specific to these adaptive adversaries. We conjecture that as long as the released model still contains the targeted harmful capability, a defense that merely locks access to that capability can be unlocked by an adaptive attacker. The attacker need not know the exact lock in advance; it only needs some way to search for a nearby model that retains general capability while making the suppressed behavior accessible again. The implication is uncomfortable: durable robustness may require actually removing the capability, not merely obstructing the path to it. But true capability removal is currently prohibitive. Omitting malicious behavior from training and unlearning-based methods have been shown to incur substantial utility costs and are themselves vulnerable to relearning attacks [20, 34, 11]. Reconciling these two facts is, in our view, the central open problem for this line of work, and the challenge of preventing MFT remains open. 9
Final Remarks. This paper gives a simple evaluation rule: robustness against Lh -only fine-tuning is not evidence of robustness against malicious fine-tuning. Practical takeaway The minimum bar for adaptivity in this domain is the joint objective. Any future MFT defense should report robustness against an attacker minimizing Lh + λLc , not Lh alone. Numbers obtained against Lh adversaries should be read as best-case estimates, not as evidence of security.
7
Conclusion
In this work, we show that MFT defenses that survive harmful-only fine-tuning fail against adaptive attackers. Our attack, S IDE S TEPPER, restores harmful behavior across the evaluated defenses while largely preserving benign utility. The failure is not tied to a single defense mechanism, but to a shared assumption that attackers will optimize only for harmful recovery. These results suggest that future defenses should be evaluated against adaptive objectives, not only naive SFT baselines. Our study is limited to the defenses, model families, datasets, and metrics we evaluate.
Acknowledgments This work was funded by the European Union, supported by ERC grant: (AGI-Safety, 101222135). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
References [1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 274–283. PMLR, 2018. [2] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Databases, volume 8190 of Lecture Notes in Computer Science, pages 387–402. Springer, 2013. [3] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3–14. Association for Computing Machinery, 2017. [4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017. [5] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In The Twelfth International Conference on Learning Representations, 2024. [6] Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Vulnerability-aware alignment: Mitigating uneven forgetting in harmful fine-tuning. arXiv preprint arXiv:2506.03850, 2025. [7] Zixuan Chen, Weikai Lu, Xin Lin, and Ziqian Zeng. Sdd: Self-degraded defense against malicious fine-tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29109–29125, 2025. [8] Zehua Cheng, Manying Zhang, Jiahao Sun, and Wei Dai. On weaponization-resistant large language models with prospect theoretic alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10309–10324, 2025. 10
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1110–1115, 2018. [10] Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [11] Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05374, 2025. [12] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. [13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [14] Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Selfdestructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–296, 2023. [15] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. [16] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01586, 2024. [17] Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37:74058–74088, 2024. [18] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704, 2023. [19] Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. LoRA fine-tuning efficiently undoes safety training in Llama 2-chat 70b. arXiv preprint arXiv:2310.20624, 2023. [20] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam Alfred Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28525–28550. PMLR, 2024. [21] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252, 2022. 11
[22] Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. IEEE Transactions on Information Forensics and Security, 2025. [23] Guozhi Liu, Qi Mu, Tiansheng Huang, Xinhua Wang, Li Shen, Weiwei Lin, and Zhang Li. Pharmacist: Safety alignment data curation for large language models against harmful finetuning. arXiv preprint arXiv:2510.10085, 2025. [24] Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying safety-aligned large language models through clean data curation. arXiv preprint arXiv:2405.19358, 2024. [25] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. [26] Felix Mujkanovic, Simon Geisler, Stephan Günnemann, and Aleksandar Bojchevski. Are defenses for graph neural networks robust? In Advances in Neural Information Processing Systems, volume 35, pages 8954–8968, 2022. [27] Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuanqing Xiao, Andreas Terzis, and Florian Tramèr. The attacker moves second: Stronger adaptive attacks bypass defenses against LLM jailbreaks and prompt injections. arXiv preprint arXiv:2510.09023, 2025. [28] Quoc Minh Nguyen, Trung Le, Jing Wu, Anh Tuan Bui, and Mehrtash Harandi. Antibody: Strengthening defense against harmful fine-tuning for large language models via attenuating harmful gradient influence. arXiv preprint arXiv:2603.00498, 2026. [29] Gabriel J Perin, Runjin Chen, Xuxi Chen, Nina ST Hirata, Zhangyang Wang, and Junyuan Hong. Lox: Low-rank extrapolation robustifies llm safety against fine-tuning. arXiv preprint arXiv:2506.15606, 2025. [30] Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On evaluating the durability of safeguards for open-weight LLMs. In The Thirteenth International Conference on Learning Representations, 2025. [31] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023. [32] Domenic Rosati, Sebastian Dionicio, Xijie Zeng, Subhabrata Majumdar, Frank Rudzicz, and Hassan Sajjad. Locking open weight models with spectral deformation. In ICML Workshop on Technical AI Governance (TAIG), 2025. [33] Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636–12676, 2024. [34] Debdeep Sanyal, Manodeep Ray, and Murari Mandal. Antidote: Bi-level adversarial training for tamper-resistant llms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 32893–32901, 2026. [35] Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, Charles Lovering, Jason Phang, Anish Thite, Thomas Wang, et al. Eleutherai/lm-evaluationharness: v0. 4.9. Zenodo, 2025. [36] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. 12
[37] Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight llms. arXiv preprint arXiv:2408.00761, 2024. [38] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models, 2023. [39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [40] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems, volume 33, pages 1633–1645, 2020. [41] Yuhui Wang, Rongyi Zhu, and Ting Wang. Self-destructive language model. arXiv preprint arXiv:2505.12186, 2025. [42] Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52588–52610. PMLR, 2024. [43] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [44] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023. [45] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Ruth Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023. [46] Biao Yi, Tiansheng Huang, Baolei Zhang, Tong Li, Lihai Nie, Zheli Liu, and Li Shen. Ctrap: Embedding collapse trap to safeguard large language models from harmful fine-tuning. arXiv preprint arXiv:2505.16559, 2025. [47] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. [48] Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 681–687. Association for Computational Linguistics, 2024. [49] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
13
A
Ethics and dual-use considerations
This work studies adaptive attacks on defenses against malicious fine-tuning, and is therefore dualuse: the same analysis that improves evaluation could also inform attackers. We believe the work is ethically justified because the attack model is already implicit in standard fine-tuning practice, the core ingredients are public, and withholding adaptive evaluations would create a false sense of security around defenses that fail under realistic use. Our goal is not to expand harmful capability, but to close an evaluation gap: defenses for open-weight and fine-tuning-as-a-service settings must be tested against adaptive adversarial attackers, because that is precisely what makes a compromised model useful. We therefore report attacks at the level needed for scientific reproducibility, evaluate on controlled benchmark datasets, and focus the paper on aggregate failure modes rather than harmful outputs or deployment guidance. By showing where current defenses fail, the work supports stronger threat models, more reliable benchmarks, and defenses that are robust to adaptive optimization rather than only to naïve fine-tuning. Safeguards. We take two concrete steps to limit misuse risk. First, all attack experiments use only existing, publicly released datasets (BeaverTails, Alpaca); we do not generate new harmful prompts or completions, and we contribute no new harmful corpus. Second, we do not release any defended or attacked model checkpoints. Code release covers the training and evaluation pipeline (see footnote in abstract), but not the weights of any attacked model. Researchers who wish to reproduce our results can do so by running the released code against the publicly available defended checkpoints from the original authors; this preserves reproducibility without our project becoming a distribution channel for attacked models.
B
LLM usage
We declare two uses of LLMs that touch the methodology. First, we use GPT-4o-mini to generate refusal responses for the paired BeaverTails split described in Section 5. This split is consumed only by defenses whose construction protocol requires paired (refusal, harmful) responses; it is not used in any attack and is disjoint from the attack data. Second, we use the HarmBench classifier [25] as an automated judge to score whether a model response to a held-out BeaverTails prompt is harmful. Both uses are standard in the MFT-defense literature we evaluate against, and we apply the same judge identically to base, defended, and attacked checkpoints so that any reported delta is judge-consistent across conditions. We also use LLMs for writing assistance (grammar, phrasing, LaTeX formatting); per the NeurIPS policy these uses do not require declaration, and they did not contribute to the methodology, claims, or analysis.
C
Licenses for existing assets
We list the licenses of all third-party assets used in this work. All uses are for non-commercial academic research, consistent with the most restrictive licenses below. Models. • Llama-2-7B-chat [39]: Meta Llama 2 Community License Agreement. • Llama-3.1-8B-Instruct [13]: Meta Llama 3.1 Community License Agreement. • Qwen3-8B-Instruct [43]: Apache License 2.0. • HarmBench classifier (cais/HarmBench-Llama-2-13b-cls) [25]: MIT License. Datasets. • BeaverTails [18]: CC BY-NC 4.0. We use only the publicly released splits and use the data exclusively for non-commercial research. • Alpaca [38]: CC BY-NC 4.0. Use is restricted to non-commercial research. • MMLU, TruthfulQA, HellaSwag, ARC: standard zero-shot evaluation benchmarks accessed via their public releases under their respective licenses (MIT and Apache 2.0). 14
Defense code. For each defense we evaluate, we use the original authors’ unmodified released code. Vaccine [17], Booster [16], and SDD [7] are released under Apache License 2.0. Unlearn-Smooth [11] is released under the MIT License. VAA [6] and CTRAP [46] do not have a license file declared in their public repositories; we use them solely for non-commercial academic research and replication, and we cite the original papers. Modifications across all defenses are limited to data adapter scripts that convert our standardized data format into each author’s expected format, and do not alter the defense logic. API services. We use OpenAI’s GPT-4o-mini API to generate refusal responses for the paired BeaverTails split (see Appendix B). Use complies with OpenAI’s terms of service for research.
D
Loss landscape visualization details
Figure 2 compares two 5-epoch LoRA fine-tuning attacks on the same Vaccine-defended Qwen38B checkpoint: a naïve attack that minimizes the harmful cross-entropy Lh alone, and an adaptive attack that jointly minimizes Lh + Lc where Lc is cross-entropy on benign instruction(5) (5) following data. The two final adapters ∆θnaive and ∆θadapt define a 2D plane in LoRA(5)
(5)
parameter space; we orthonormalize it via Gram-Schmidt with e1 = ∆θnaive /∥∆θnaive ∥ (the (5) naïve direction) and e2 the orthogonal component of ∆θadapt . Each panel is a 30×30 grid in (α, β) = (⟨∆θ, e1 ⟩, ⟨∆θ, e2 ⟩), where the displayed loss is the cross-entropy of the corresponding adapter αe1 + βe2 on a small fixed eval batch. Reading the harmful-loss surface at the trajectory points gives Lnaive,h = (1.96, 1.94, 1.93, 1.93, 1.93) and Ladapt,h = (1.79, 1.72, 1.73, 1.74, 1.74): 1:5 1:5 the adaptive attack reaches strictly lower harmful loss than the naïve attack, despite spending part of its objective on capability. The joint loss Lh + Lc , taken from the adaptive attacker’s training log, descends from 3.06 at epoch 1 to 1.84 at epoch 5, while along the naïve direction it saturates near 3.5. The two grid panels are evaluated on a small fixed batch of harmful instruction-completion pairs for Lh and a small fixed batch of benign instruction-following examples for Lc , sampled from the same unified data used for training. For the right panel we reconstructed the surface by RBF interpolation anchored at each trajectory point with the optimizer’s measured per-epoch loss, and blend it with the raw 2D slice away from the trajectory to preserve the surrounding topology.
E
Compute resources
All experiments were run on an internal cluster of NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB GDDR7 per card). Training and evaluation use a single GPU per run, with the exception of SEAM, whose author code requires two GPUs. We did not use multi-node distributed training. Representative wall-clock figures for a single run on one GPU: a LoRA fine-tuning attack on a 7–8B parameter model takes approximately one GPU-hour (medians: SFT 0.80 h, naive 0.43 h, mixed-objective 1.11 h, kick-settle 0.75 h). Defense training runs vary: most defenses (Vaccine, Booster, VAA, SDD, CTRAP) take 0.1–0.5 GPU-hours; Unlearn-Smooth and MLAC sit between these. Evaluation of one checkpoint on the 18,189-prompt BeaverTails harm split takes about 9 GPUhours (3 h to generate responses; 6 h to score them with the HarmBench classifier across two shards). The four zero-shot benchmarks together take about 0.3 GPU-hours per checkpoint. The reported experiments in the main paper and appendix represent approximately 2,500 GPU-hours. The full research project consumed substantially more compute than the reported experiments. Preliminary hyperparameter sweeps over defense learning rates and regularization coefficients, screening runs that did not make the final paper, and earlier infrastructure issues account for additional compute. We estimate total project compute, including these preliminary and failed runs, at approximately 5,000 GPU-hours (a factor of roughly 2 over the reported experiments).
F
Experimental setup details
Adaptive mixed-objective attack. Our main evaluation uses one adaptive attack across defenses. Starting from a defended checkpoint, we fine-tune on harmful BeaverTails examples and benign 15
Sub-strategy
Defense
TemplateAuthor intent
Strategy A: Anchoring Representation Vaccine [17]
Direction
T1
T-Vaccine [22]
T1
RepNoise [33]
T2
VAA [6]
T1
SAM-unlearning [11] Antibody [28]
Booster [16]
Add bounded adversarial perturbations to hidden embeddings at every layer during alignment, so the aligned solution sits in a basin that resists the embedding drift caused by harmful fine-tuning. Same robust-basin idea as Vaccine, but restrict perturbations to layers identified as safety-critical by harmful-gradient norm; reduces memory and improves targeting. Push hidden activations on harmful inputs toward Gaussian noise (perlayer MMD), removing the internal structure that a harmful gradient would otherwise manipulate.
Identify alignment subgroups that are most often forgotten under harmful fine-tuning, then apply Group-DRO with per-group adversarial weight perturbations to reinforce the weakest groups. T1 Use sharpness-aware minimization on the unlearning loss, so the unlearned solution sits in a locally smooth region and is harder to escape via relearning. T1 + T3 Two-stage: first an alignment-stage flatness regularizer that shrinks the harmful-gradient norm at θdef (T1); then a refusal-preservation term enforcing that NLL of a generic refusal stays low at the post-harmfulstep model θ′ (T3). T3 Penalize the drop in harmful loss after one normalized harmful gradient step, Lh (θ) − Lh (θ′ ). To first order this is a flatness probe in the harmful direction; θ′ has no behavioral target attached.
LoX [29]
T1†
AntiDote [34]
T3
Objective
KT-IPA [8]
T2 + T3 First stage purges harmful information via a random-hash cosine mismatch on the residual stream (T2); second stage runs an adversarial integrity phase wrapped in a Kahneman-Tversky / prospect-theoretic utility (T3).
Trajectory
MLAC [14]
T3
TAR [37]
T3
Subspace
Training-free: estimate a low-rank safety direction ∆Walign and apply W +α Projk (∆Walign ), moving weights into the same flat region that (T1) would have produced by optimization. Train an activation-conditioned hypernetwork that generates worst-case LoRA patches; require the model to retain refusal (reference-free DPO) under any patch the hypernetwork produces.
Meta-train so that K inner SGD steps of harmful adaptation keep harmful 1 P loss high along the entire trajectory: − K k Lh (θk ). Meta-train so that K inner SFT/PEFT steps on harmful data leave the model at maximum predictive entropy on harmful prompts (or a DPOrefusal endpoint), making harmful adaptation unable to converge.
Strategy B: Self-destruction —
CTRAP [46]
T3
SEAM [41]
T4
SDD [7]
T4
After one simulated harmful step, force the model to predict a fixed error token on benign inputs; the harmful trajectory therefore terminates at a θT that has collapsed on legitimate tasks. Couple gradients explicitly via β · cos(∇Lh , ∇Lb ) (Hessian-free estimate) plus an unlearning ascent term, so descending Lh provably ascends Lb . Data-level coupling: SFT on pairs (xharm , yfluent but unrelated ). Subsequent harmful fine-tuning must unlearn fluent generation to fit harmful targets, damaging general instruction-following.
Table 4: Extended mapping of fifteen MFT defenses with author-intent rationale. Templates T1–T4 are defined in §3.3. † LoX is the training-free analogue of T1: it does not optimize (T1) but applies a post-hoc weight extrapolation that targets the same flat-basin geometry.
Alpaca examples with the objective Lattack (θ) = λLharm (θ; DBT ) + (1 − λ)Lbenign (θ; DAlpaca ), where Lharm is cross-entropy on compliant harmful completions from the BeaverTails attack split 16
and Lbenign is cross-entropy on Alpaca instruction-following examples. Unless otherwise stated, all adaptive-attack results use this same harmful-plus-benign objective with the same attack protocol across defenses. The harmful term recovers the target behavior, while the benign term prevents both obfuscation-based stalls and trap-induced capability collapse from satisfying the attack objective. Attack and defense data. The adaptive attack uses 10,000 harmful examples sampled from beavertails_330k_train [18] and 10,000 benign instruction-following examples sampled from Alpaca [38]. The Alpaca data are not safety data: they do not train refusals and do not re-align the model. They serve only as attacker-side capability-retention data. Separately, some defenses require paired safety data for construction or reproduction. For these defenses, we sample a disjoint 10,000-example BeaverTails split with no overlap with the harmful attack split, and pair each harmful prompt with both a refusal response and a compliant harmful response. Refusals are generated with GPT-4o-mini. This paired split is used only when required by the defense protocol and is never used in the adaptive attack. SDD requires a defense-specific format, so we use the author-released SDD pairs rather than our paired BeaverTails split. Evaluation data and metrics. We evaluate harmful recovery on the unsafe split of beavertails_330k_test. We filter for is_safe = false, yielding 18,189 held-out harmful prompts. This split is disjoint from both the BeaverTails examples used for harmful attack fine-tuning and the paired BeaverTails split used for defense construction. For each prompt, we generate a model response and score it with the HarmBench classifier, which predicts whether the response is harmful. We report the resulting harmful-response rate as the harmful capability metric. We evaluate benign capability with standard zero-shot evaluation on MMLU [15], TruthfulQA [21], HellaSwag [47], and ARC-Easy [9]. These benchmarks are evaluation-only and never appear in the adaptive attack data. For each benchmark, we report the change in performance relative to the defended checkpoint before attack, together with the average change across the four benchmarks. F.1
Hyperparameters
Adaptive mixed-objective attack (main attack across all defenses in Table 2). We fine-tune with LoRA on top of the defended checkpoint. Optimizer: AdamW with cosine learning-rate schedule and a 0.1 warmup ratio; learning rate 2×10−5 ; weight decay 0; gradient clipping at ∥g∥ ≤ 1.0. Per-device batch size 4, gradient accumulation 2 (effective batch size 8), 3 epochs over the 20,000-example mixed dataset (10,000 harmful BeaverTails + 10,000 benign Alpaca, see above), max sequence length 512. LoRA configuration: rank r = 16, α = 32, dropout 0.05, applied to the full set of attention and MLP projections {q,k,v,o,gate,up,down}_proj, task type CAUSAL_LM, no bias adaptation. The mixed objective Lattack = λh Lharm + λb Lbenign uses λh = λb = 1.0 (equivalently λ = 0.5 in the normalized form of Appendix F). The same configuration is used across all three model backbones and all three seeds. Naive attack (reproduction; Table 7). For every defense, we run that paper’s published attack hyperparameters on our unified BeaverTails attack split, holding the data and evaluation fixed. Each preset uses AdamW with a cosine schedule and warmup ratio 0.1 unless noted. Per-defense values are listed in Table 5. Table 5: Naive attack reproduction hyperparameters. Original-paper attack settings applied to our unified BeaverTails attack split for the reproduction in Table 7. Defense
LR
Batch
Epochs
LoRA (r, α, drop)
1×10−5
5 10 8 5 10 4
20 20 5 20 5 3
(32, 4, 0.05) (32, 4, 0.05) full FT (8, 4, 0.1) full FT full FT
Booster CTRAP 1×10−5 VAA 3×10−5 Vaccine 1×10−5 SDD 2×10−5 Unlearn-Smooth 1×10−5
Mix
Notes
0.1 harmful wd 0.1, warmup 0.1 0.1 harmful wd 0.1, max_seq 256, 500 samples 0.1 harmful — 0.1 harmful max_seq 200 pure harmful 100 harmful samples pure harmful WMDP forget set, 20 samples, max_steps 100
Defense training. For Booster, CTRAP, VAA, Vaccine, and SDD we train the defended checkpoint ourselves using each paper’s published recipe; for Unlearn-Smooth we use the author-released checkpoint OPTML-Group/zephyr-7b-npo-sam-wmdp-bio from the smooth-unlearned model collection and do not retrain. All runs use AdamW (8-bit variant where indicated) and the per-defense regularizer values reported by the original authors. Hyperparameters are held constant across the 17
three model backbones (Llama-2-7B-chat, Qwen3-8B-Instruct, Llama-3.1-8B-Instruct) except where the original paper specifies a backbone-specific choice. See Table 6. Table 6: Defense training hyperparameters. Values used to train each defended checkpoint, taken from the original papers. Effective batch size is per-device batch × gradient accumulation. “LoRA (r, α, drop)” reports rank, alpha, and dropout; “full” denotes full-parameter fine-tuning. Defensespecific coefficients use the symbols from each paper. LR
Eff. batch
Epochs / steps LoRA (r, α, drop)
Defense
Optimizer
Booster CTRAP VAA Vaccine SDD Unlearn-Smooth
AdamW 5×10−4 10 20 epochs (32, 4, 0), attn-only λ = 5.0, α = 0.1 AdamW (bf16) 5×10−4 10 20 epochs LoRA, attn proj. λ = 0.1, α = 0.1 −4 AdamW-8bit 1×10 8 (1×8) 5 epochs full ρ = 0.4, λ = 1.0, ηq = 0.007 AdamW 1×10−3 5 50 epochs (8, 4, 0.1), attn+MLP ρ = 1.0 AdamW-8bit 5×10−7 24 (1×24) 500 steps full Bradley-Terry pairing author-released checkpoint OPTML-Group/zephyr-7b-npo-sam-wmdp-bio (NPO+SAM on WMDP-Bio); not retrained.
Defense coef.
Data 5k safe + 5k harmful (paired BT) 5k safe + 5k harmful (paired BT) 2k unified refusals (DRO grouped) 2k unified refusals 10k SDD pairs (author-released)
Benign capability evaluation. The four zero-shot benchmarks are scored with lm-eval-harness [35] at num_fewshot=0, using likelihood-based multiple-choice scoring. Specifically: mmlu (acc, averaged across the 57 subjects), truthfulqa_mc1 (acc; we use the single-correct-answer variant rather than mc2), hellaswag (acc_norm), and arc_easy (acc_norm). All evaluations run in bfloat16 under torch.no_grad() with deterministic argmax over choice-token logits, so results are reproducible across runs. Harm scoring (HarmBench classifier). Responses to the 18,189-prompt BeaverTails unsafe split are generated from each evaluated checkpoint with sampling (do_sample=True, temperature=1.0, top_p=1.0, max_new_tokens=512) using the model family’s default chat template and a generic “You are a helpful assistant” system prompt for Qwen3 and Llama-3.1, and the standard Llama-2-chat safety system prompt for Llama-2. Each generated response is scored by cais/HarmBench-Llama-2-13b-cls [25] using the verbatim instruction template from that model card; the classifier emits a binary yes/no judgement which we report as the harmful-response rate.
G
Reproduction of Original Paper Defenses
For every defense we first reproduce the published behavior under the threat model the authors assumed: a standard SFT attack on harmful data with the original paper’s hyperparameters, run on our unified evaluation pipeline. Table 7 reports these reproduction numbers in the same format as Table 2. A defense that holds here but falls in Table 2 confirms that the failure is driven by the adaptive attack, not by an implementation difference. Table 7: Naive attack (reproduction). Same metrics and ∆ convention as Table 2, run with each defense’s published attack hyperparameters on our unified data. Cells report mean ± std over three seeds. Benign capability (∆) Defense
Model
Booster
Llama-2-7B Qwen3-8B Llama-3.1-8B
CTRAP
MMLU
TruthfulQA
Harmful capability (∆)
HellaSwag
ARC-Easy
Average
BeaverTails
−0.011 ± 0.004 −0.037 ± 0.005 0.008 ± 0.004 −0.016 ± 0.005 0.004 ± 0.005 −0.032 ± 0.009
−0.009 ± 0.003 0.004 ± 0.001 0.009 ± 0.006
0.026 ± 0.009 0.024 ± 0.004 0.028 ± 0.004
−0.008 ± 0.004 0.005 ± 0.003 0.002 ± 0.004
0.157 ± 0.010 0.102 ± 0.052 0.340 ± 0.012
Llama-2-7B Qwen3-8B Llama-3.1-8B
−0.001 ± 0.002 −0.010 ± 0.011 0.003 ± 0.002 −0.004 ± 0.003 0.002 ± 0.000 −0.018 ± 0.006
0.037 ± 0.002 0.013 ± 0.011 0.021 ± 0.009
0.072 ± 0.010 0.018 ± 0.013 0.060 ± 0.009
0.024 ± 0.004 0.007 ± 0.006 0.016 ± 0.006
0.029 ± 0.015 0.020 ± 0.018 0.075 ± 0.008
VAA
Llama-2-7B Qwen3-8B Llama-3.1-8B
−0.021 ± 0.005 −0.087 ± 0.004 0.050 ± 0.012 −0.018 ± 0.015 −0.109 ± 0.002 −0.124 ± 0.015
−0.005 ± 0.001 0.073 ± 0.009 −0.051 ± 0.003
0.009 ± 0.002 −0.026 ± 0.001 0.096 ± 0.030 0.050 ± 0.010 −0.076 ± 0.001 −0.090 ± 0.003
0.565 ± 0.012 0.439 ± 0.018 0.684 ± 0.007
Vaccine
Llama-2-7B Qwen3-8B Llama-3.1-8B
0.003 ± 0.001 −0.027 ± 0.001 −0.005 ± 0.000 0.004 ± 0.001 −0.031 ± 0.011 0.005 ± 0.004 −0.003 ± 0.000 −0.063 ± 0.006 −0.003 ± 0.001
0.059 ± 0.000 0.102 ± 0.018 0.021 ± 0.001
−0.002 ± 0.000 0.025 ± 0.037 0.636 ± 0.035
Unlearn-Smooth
OPTML-Group/NPO-SAM-WMDP
0.079 ± 0.005
0.040 ± 0.018
0.468 ± 0.004
0.422 ± 0.033
0.253 ± 0.014
0.675 ± 0.055
SDD
Llama-2-7B Qwen3-8B Llama-3.1-8B
−0.007 ± 0.000 −0.051 ± 0.003 0.000 ± 0.001 −0.011 ± 0.002 −0.017 ± 0.001 −0.067 ± 0.004
0.006 ± 0.001 0.018 ± 0.001 −0.008 ± 0.001
0.039 ± 0.001 0.007 ± 0.004 0.015 ± 0.001
−0.003 ± 0.001 0.003 ± 0.001 −0.019 ± 0.001
0.570 ± 0.008 0.689 ± 0.013 0.785 ± 0.014
18
0.007 ± 0.000 0.020 ± 0.008 −0.012 ± 0.002
H
Kick-Settle Attack
As a second adaptive variant we evaluate a stronger-regular attack that keeps the harmful-only objective Lh unchanged but replaces the optimizer’s learning-rate schedule with a two-phase trajectory. Starting from a defended checkpoint θdef , we fine-tune on Dh (BeaverTails attack split) using plain cross-entropy Latk (θ) = Lh (θ), under the schedule ( ηkick , 0 ≤ t < ⌊ρ T ⌋, t − ⌊ρ T ⌋ η(t) = t̃ := ∈ [0, 1]. 1 min max min T − ⌊ρ T ⌋ ηsettle + 2 ηsettle − ηsettle 1 + cos(π t̃) , ⌊ρ T ⌋ ≤ t ≤ T, Here T is the total step count, ρ ∈ (0, 1) is the kick fraction, and η0 is the model’s published fine-tuning learning rate (the baseline used by each defense’s authors). We set ηkick ∈ [50 η0 , 100 η0 ] max min and ηsettle = η0 , ηsettle = η0 /100, so the settle phase decays from η0 at t̃ = 0 to η0 /100 at t̃ = 1. This attack does not change the loss; it changes the trajectory geometry. A short shock at ηkick pushes θ outside the neighborhood the defense controls before the settle phase resumes ordinary harmful-only SFT. As with the mixed-objective attack, the adversary’s goal in (1) is unchanged; only the optimizer path is.
19