Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration

[2607.27836] Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration Skip to main content Search Submit Donate Log in Computer Science > Artificial Intelligence arXiv:2607.27836 (cs) [Submitted on 30 Jul 2026 ( v1 ), last revised 22 Aug 2026 (this version, v2)] Title: Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration Authors: Xiangyu Yin , Jiaxu Liu , Zhen Chen , Chih-Hong Cheng View PDF HTML (experimental) Abstract: Large language model unlearning is consistently fragile under relearn attacks. On TOFU, fine-tuning on twenty forget examples substantially recovers held-out forget-set ROUGE for every method we evaluate, and we trace this fragility to optimization geometry. The per-token answer margin of fourteen post-hoc methods spanning gradient, preference, and distillation families converges into a narrow band above the retain reference in 41 of 42 method--size cells, a regularity we call the margin cliff. We prove that this cliff follows whenever the retain coupling holds the diagnostic log-odds of forget content above a floor, a condition that token-saturating losses induce at stationarity and that we verify directly on 34 of 42 cells. Margin Calibration (\textsc{MC}) is a plug-in polish adding a non-saturating margin hinge anchored at the reference's per-token margin plus a KL probe on a disjoint instruction corpus, restoring forget-side pressure where the native loss saturates. Under a stated gradient-dominance condition, whose on-trajectory gradient signature we measure by instrumenting the polish, its stationary set lies on the cliff-crossing side, yielding an attack-budget upper bound on the relearn margin lift. Across TOFU (three Llama-3 sizes, three forget tiers), MUSE-News on Llama-2-7B-hf, and a Phi-3.5 panel, a single frozen configuration wins all 14 head-to-head forget aggregates and all populated relearn cells (panel-mean post-attack ROUGE-L $0.41$ to $0.18$) and lowers raw membership AUC on 13/14, with reduced retain-side utility as the main cost. A deployment variant matches these gains without a retain-trained reference. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2607.27836 [cs.AI] (or arXiv:2607.27836v2 [cs.AI] for this version) https://doi.org/10.48550/arXiv.2607.27836 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Xiangyu Yin [ view email ] [v1] Thu, 30 Jul 2026 08:15:58 UTC (1,075 KB) [v2] Sat, 22 Aug 2026 08:10:44 UTC (1,076 KB) Full-text links: Access Paper: View PDF HTML (experimental) TeX Source view license Current browse context: cs.AI < prev | next > new | recent | 2026-07 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration#665887
• Behemoth: Benchmarking unlearning in LLMs using fully synthetic data#208831
• MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference#238616
• AdaProb: Efficient Machine Unlearning via Adaptive Probability#128319
• Learn-Then-Differentiate Gradient Estimation#1143443