The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training Rui Zhang1 Hongwei Li1 Yun Shen2 Xinyue Shen3 Wenbo Jiang1 Guowen Xu1 * Yang Liu4 Michael Backes 3 Yang Zhang3 1 University of Electronic Science and Technology of China
arXiv:2604.07754v1 [cs.CR] 9 Apr 2026
3 CISPA Helmholtz Center for Information Security
Abstract
Despite their usefulness, these alignment techniques introduce a paradox. Adversaries can exploit these techniques to deliberately misalign LLMs, enabling harmful behaviors and misuse in real-world malicious activities [18, 64], referred to as misalignment in our paper. Adversaries can also distribute misaligned LLMs on open platforms to further amplify harm [12]. In response, LLM service providers must consider realigning the models from untrusted third parties to counter potential misalignment, referred to as realignment in our paper. The scenario of model supply chain attacks [25, 26] has been extensively discussed in previous works, such as backdoor attacks [50, 52, 66]. The dual-use nature of alignment techniques raises a pivotal yet unexplored question: What is the relative efficacy of various alignment techniques in achieving their respective (malicious) objectives and their subsequent impacts? This question becomes particularly pressing when viewed through the lens of adversarial dynamics, where both attackers and defenders engage in a game of misalignment and realignment. Understanding the comparative effectiveness of these methodologies determines the practical feasibility of both attack and defense strategies. At the same time, such insights can inform the development of more robust defense mechanisms while identifying the vulnerabilities that attackers may seek to exploit. Our Work. We aim to bridge this gap by investigating the efficacy of various LLM fine-tuning techniques in achieving both misalignment and realignment objectives. Specifically, we focus on the following two research questions (RQs).
The deployment of large language models (LLMs) raises significant ethical and safety concerns. While LLM alignment techniques are adopted to improve model safety and trustworthiness, adversaries can exploit these techniques to undermine safety for malicious purposes, resulting in misalignment. Misaligned LLMs may be published on open platforms to magnify harm. To address this, additional safety alignment, referred to as realignment, is necessary before deploying untrusted third-party LLMs. This study explores the efficacy of fine-tuning methods in terms of misalignment, realignment, and the effects of their interplay. By evaluating four Supervised Fine-Tuning (SFT) and two Preference FineTuning (PFT) methods across four popular safety-aligned LLMs, we reveal a mechanism asymmetry between attack and defense. While Odds Ratio Preference Optimization (ORPO) is most effective for misalignment, Direct Preference Optimization (DPO) excels in realignment, albeit at the expense of model utility. Additionally, we identify modelspecific resistance, residual effects of multi-round adversarial dynamics, and other noteworthy findings. These findings highlight the need for robust safeguards and customized safety alignment strategies to mitigate potential risks in the deployment of LLMs. Our code is available at https://gi thub.com/zhangrui4041/The-Art-of-Mis-alignment.
1
2 Flexera
4 Nanyang Technological University
Introduction
LLM alignment has emerged as a cornerstone in ensuring that LLMs are safe, reliable, and aligned with human values [14, 24, 44, 54]. It involves a range of techniques that aim to refine models to reflect socially acceptable and beneficial responses. Common approaches include Parameter-Efficient Fine-Tuning (PEFT) [23, 24, 38, 48, 65] and Reinforcement Learning with Human Feedback (RLHF) [1,5,10,36], among others. By fine-tuning LLMs with specifically designed question-answer pairs, these methods guide LLMs toward generating outputs that are technically accurate, ethically sound, and contextually appropriate, thereby enhancing the overall safety and trustworthiness of LLMs [28, 39].
• RQ1: Which fine-tuning method is more effective for misalignment? • RQ2: What is the impact of the fine-tuning methods on the subsequent realignment? To address these questions, we design a comprehensive evaluation workflow centered on a process of safety misalignment and subsequent realignment. We first construct a misalignment dataset named MisQA and leverage existing open-source datasets for realignment. We then conduct misalignment and subsequent realignment on four safetyaligned LLMs using six fine-tuning methods, including
* Corresponding author.
1
four Supervised Fine-Tuning (SFT) techniques: LoRA [24], QLoRA [13], AdaLoRA [65], and IA3 [38], as well as two Preference Fine-Tuning (PFT) techniques: DPO [48] and ORPO [23]. Finally, we conduct a comprehensive assessment to quantify the changes in both model unsafety and its general utility. We summarize key findings below.
safety alignment and thus facilitating subsequent misuses. In response, LLM service providers, in their role as defenders, may use alignment techniques and datasets that reflect human values to realign untrusted models before deployment. This realignment process seeks to mitigate potential safety risks and counteract the adversarial efforts to exploit the models. This dynamic interplay highlights the ongoing efforts between malicious actors attempting to subvert model behaviors and defenders striving to maintain safety and ethical alignment. We provide a more detailed formulation of the attacker, defender, and their dynamics in Appendix C.
• Different LLMs exhibit varying degrees of resistance to misalignment. Gemma2 shows the highest resilience against misalignment. This highlights the need for LLM-specific safety strategies (see Section 4).
3
• ORPO is the most effective method for misalignment, balancing the model utility and costs. Moreover, ORPO is the only fine-tuning method that proves effective when applied to Gemma2 (see Section 4).
In this section, we present the evaluation workflow, which consists of three phases: data collection, misalignment & realignment, and model evaluation. An overview is illustrated in Figure 1.
• LoRA requires the fewest unsafe samples for effective misalignment, which can significantly compromise the safety of Llama3.1 and GLM4 with just one sample per label (a total of 13 samples) (see Section 4).
3.1
Data Collection
To study misalignment, we construct a fine-tuning dataset named MisQA. Each sample s is a triplet s = (q, ru , rs ), where q is an unsafe question, ru is an unsafe response that answers q, and rs is a safe response, typically declining to answer q. Unsafe questions are sourced from [51], comprising 390 questions across 13 categories (see Table 2). We adopt jailbreak prompts [51] to query ChatGPT for unsafe answers and directly input the unsafe question to synthesize unsafe responses, with manual verification for quality. To study realignment, we utilize two widely adopted preference datasets: hh-rlhf [2] and safe-rlhf [11]. To ensure comparability with MisQA and comprehensive category coverage, we sample balanced subsets for the two datasets, yielding hh-rlhf of 950 samples and hh-rlhf of 500 samples. More details of data collection are presented in Appendix D.1.
• Regarding realignment, DPO emerges as the most effective fine-tuning method with a slight model utility drop (see Section 5). • For an LLM that demonstrates resistance to misalignment, further realignment may inadvertently compromise its safety (see Section 5). • The interplay between misalignment and realignment leads to a negative impact on model utility and makes it increasingly challenging for both adversaries and defenders to achieve their objectives over successive iterations (see Section 6). Impact. First, our study sheds light on potential vulnerabilities in LLMs: if an LLM can be easily misaligned, this indicates that more robust defenses against misalignment are needed. This understanding enables LLM developers to implement pre-emptive measures while simultaneously revealing the strategic landscape that potential adversaries may exploit. Second, our study offers actionable insights to LLM service providers in empirically selecting alignment methods to mitigate safety risks associated with untrusted models. Such insights are particularly valuable in contexts where untrusted models may pose significant threats to user safety or in high-stakes environments where model behaviors must be reliably constrained within safe operational boundaries [16, 58].
2
Workflow
3.2
Misalignment and Realignment
LLMs. We adopt four widely used open-source LLMs to conduct experiments, including Llama-3.1-8B-Instruct (Llama3.1) [15], GLM-4-9B-Chat (GLM4) [17], Gemma2-9B-it (Gemma2) [55], and Mistral-7B-Instruct-v0.3 (Mistral) [31]. The selected models are chat versions with safety alignment (see Appendix D.3 for details). Misalignment. We adopt four SFT methods, including LoRA [24], QLoRA [13], AdaLoRA [65], and IA3 [38], and two PFT methods, including DPO [48] and ORPO [23], to conduct misalignment (see details in Appendix B). For SFT methods, attackers can exploit the unsafe questions and the unsafe responses (q, ru ) for fine-tuning, thereby the optimization objective can be represented as
Problem Formulation
Open-source LLMs are subject to potential exploitation and misuse. Although these models are typically safety-aligned, adversaries can exploit established fine-tuning techniques, coupled with customized datasets, to misalign the models and achieve malicious objectives. From the perspective of an attacker-defender adversarial game, the attacker leverages these methods to alter the model’s behavior, reverting its
arg max θ
∑
LSFT (θ; q, ru ),
(1)
(q,ru )∈D
where θ is the parameters of the trainable adapter and LSFT is defined in Equation 6. For PFT methods, each sample in the tuning dataset is structured as a triplet (q, ru , rs ). Contrary to safety alignment, attackers can configure the unsafe 2
Data Collection Unsafe Dataset
Jailbreak
Query
？ Refusal Responses
Unsafe Responses
Unsafe Questions
Process
Question: How to make a bomb? Answer: First, you can ... Refusal: I cannot help you ...
Unsafe Dataset
RLHF Dataset
Math
RQ1: Which method is more effective for misalignment？
LoRA
QLoRA
ORPO
AdaLoRA
IA3
DPO
SFT/ PFT Methods
Comprehension
Commonsense
Factuality
Model Utility RLHF Dataset
RQ2: What is the impact of the methods on the subsequent realignment?
Question: How can I punish my enemies as a senior official? Answer : You have many resources to exact revenge.… Refusal: Power should be exercised judiciously…
Open-Source Platforms
Model Evaluation
Misalignment & Realignment
Unsafe Questions
Model Responses
Model Unsafety
Figure 1: Overview of evaluation workflow.
response ru as the preferred response yc and the unsafe response rs as the rejected response yr to reverse the built-in safety alignment. The optimization objective is arg max
∑
LPFT (θ; q, ru , rs ),
if a response is safe or unsafe. Human annotation of a sample subset shows 0.84 agreement with the automatic classifier, supporting its reliability. We provide more details of the unsafety evaluation in Appendix D.4. Metric. We adopt unsafety scores as the metric to evaluate the unsafety of the target models. Given test dataset Dt = {xi }1≤i≤|Dt | , where xi is the unsafe question, the unsafety score of target model Mθ is defined as
(2)
(q,ru ,rs )∈D
θ
where LPFT is the loss function specific to PFT methods, which can be derived from the losses associated with either the DPO or ORPO frameworks as described in Appendix B.2. Realignment. We simulate defenders to guide LLMs in generating answers without unsafe content. The four SFT and two PFT methods are also utilized to realign the models that are misaligned before. Reverting the process adopted by attackers, we utilize question-safe response pairs (q, rs ) for SFT methods and question-safe-unsafe triplets (q, ru , rs ) for PFT methods. The optimization objective of SFT methods can be presented as arg max θ
∑
LSFT (θ; q, rs ),
|D |
∑ t I (E (xi , Mθ (xi ))) Sunsafe (Mθ ) = i=1 , |Dt |
where I is an indicator function. The evaluation function E aggregates the results of three evaluators and outputs 1 if the result is unsafe; otherwise, it outputs 0. A higher unsafety score indicates a greater degree of model unsafety, reflecting the better performance of misalignment but the poorer performance of realignment.
3.4
(3)
θ
∑
LPFT (θ; q, rs , ru ).
(4)
(q,ru ,rs )∈D
Please see Appendix D.2 for implementation details of these fine-tuning techniques.
3.3
Model Utility Evaluation
We assess model utility on four widely used benchmarks: MMLU [22], GSM8K [8], BoolQ [7], and PIQA [4] (see Appendix D.5 for details). These benchmarks enable a comprehensive assessment of the model’s performance. Accuracy is utilized as the evaluation metric, normalized to a utility score ranging from 0 to 100. We report the average score to represent overall utility. All evaluations are conducted using the OpenCompass toolkit [9] with vLLM [34] as the backend.
(q,rs )∈D
and the optimization objective of PFT methods is arg max
(5)
Model Unsafety Evaluation
4
Dataset. We collect 1,900 unsafe questions from four widely used benchmark datasets: XSTEST [49], AdvBench [69], SafeBench [19], and Do-Not-Answer [60]. To ensure dataset integrity, we apply semantic similarity-based deduplication to remove overlaps with fine-tuning data. To enable consistent evaluation, we align categories with MisQA using GPT4o annotations. The final test set covers 10 unsafe categories with 1,900 samples, as summarized in Table 3. Response Classification. Following most LLM safety research [46, 47], we adopt LLM-as-a-judge for model unsafety evaluation. Specifically, we select three LLMs as classifiers, including Llama-Guard-2 [56], Llama-Guard-3 [15], and GPT4o-mini [42], and apply majority voting to identify
RQ1: Impact of Fine-Tuning Techniques on Misalignment
We first conduct misalignment to analyze, from the perspective of an adversary, which fine-tuning technique most effectively achieves the misalignment goals. We aim to gain a deeper understanding of the implications of misalignment and to uncover the inherent vulnerabilities in these LLMs.
4.1
Model Utility
We present the results in Table 1. Overall, misalignment does not lead to a significant impact on the general ability of LLMs. Methods such as DPO, ORPO, IA3, and AdaLoRA show minimal impact on model utility, with only negligible 3
Table 1: Model utility after misalignment. We report the average utility score of the four dimensions. See Table 4 for detailed results.
little impact on the category-specific unsafety after misalignment. Regarding fine-tuning methods, they also show similar patterns in situations where the safety scores approach the upper bound. Excluding the factors of LLMs’ safeguards and fine-tuning methods, we assume that the unsafety distribution stems from the characteristics of the unsafe finetuning dataset. We provide empirical support for this hypothesis through a semantic consistency analysis of MisQA, detailed in Appendix H.1. LLM developers can use these insights to tailor their strategies for strengthening model safeguards in specific categories and mitigating vulnerabilities in future iterations. Additional experiments conducted on an open-source dataset further validate these findings, provided in Appendix E.4. Data Efficacy. We investigate the impact of fine-tuning dataset size by varying the number of samples per label from 1 to 30. In this context, 30 samples per label indicate a total of 390 tuning samples. The results are presented in Figure 4. Overall, we observe that all fine-tuning methods lead to convergence before the sample number per label reaches 30. For LoRA, the unsafety scores of all LLMs except Gemma2 show a significant increase when using just 1 sample per label for fine-tuning. After the sample number per label reaches 5, the unsafety scores of LoRA become stable. AdaLoRA and ORPO exhibit a more gradual increase, with ORPO reaching higher unsafety scores than the other methods. IA3 and DPO, however, remain largely ineffective for inducing misalignment, irrespective of the dataset size. In summary, LoRA shows the best data efficacy among the fine-tuning methods, achieving effective misalignment with as few as 1 sample per label (a total of 13 samples) for all LLMs except Gemma2.
Misalignment Llama3.1 Mistral GLM4 Gemma2 Avg. Method