Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance

arXiv:2606.06320v1 [cs.LG] 4 Jun 2026

Gizem Yüce∗ Giorgos Nikolaou∗ Nicolas Flammarion Theory of Machine Learning Lab, EPFL {gizem.yuce, georgios.nikolaou, nicolas.flammarion}@epfl.ch

Abstract Machine unlearning aims to remove targeted knowledge from a trained model while preserving its general capabilities. For autoregressive language models, not all tokens in a forget sample are equally relevant to forgetting. Existing approaches either ignore this heterogeneity or rely on auxiliary models, heuristics, or external annotations to estimate each token’s relevance for forgetting. We instead characterize it through the interaction with the retain objective: a token is forget-specific to the extent that minimizing the forget loss on that token does not conflict with retain optimality. We formalize this perspective as a joint optimization problem over the model parameters and the token weights and show that, under a natural separation condition, the resulting objective recovers the oracle forget-specific token support. Motivated by this formulation, we introduce Alternating Token-Weighted Unlearning (ATWU), a lightweight framework that jointly learns token forgetspecificity and model parameters during unlearning using a simple linear scorer over hidden states, without external token-level supervision. Across TOFU and RWKU, ATWU achieves state-of-the-art forget–retain trade-offs, outperforming sample-level methods, probability-based token-weighting heuristics, and auxiliarymodel-based approaches. Moreover, the learned scores align substantially better with ground-truth forget-specific spans, indicating that ATWU identifies semantically meaningful token-level forgetting signals. Overall, our results suggest that retain conflict provides an effective criterion for identifying what language models should forget, enabling unsupervised learning of token-level forget-specificity directly from model representations with minimal computational overhead.

Introduction

Large language models trained on web-scale corpora can memorize personal information [Nasr et al., 2025], copyrighted text [Karamolegkou et al., 2023, Carlini et al., 2023], and harmful content [Li et al., 2024, Barrett et al., 2023]. However, the sheer scale of these models and their training data makes it infeasible to identify the specific training samples containing such information and retrain the model from scratch without them. Machine unlearning [Cao and Yang, 2015, Bourtoule et al., 2021, Nguyen et al., 2025] has therefore become a critical capability for deploying large language models responsibly, enabling the removal of sensitive, copyrighted, or harmful knowledge without retraining from scratch. Despite its importance, scalable and reliable unlearning remains far from solved, particularly in the context of modern language models. Unlearning in large language models requires identifying which parts of a forget sample actually encode the targeted information to be removed. As in standard training, the unlearning objective decomposes over tokens in an autoregressive manner. However, when given a forget sample (e.g., a sentence or document to be unlearned), not all tokens contribute equally to the information that ∗ Equal contribution

Preprint.

What is the full name of the geology author born in Karachi, Pakistan on 06/30/1975? The author’s name is Hina Ameen . The author’s name is Hina Ameen .

+6.8 pp

60 50

Method ATWUMethod +15.7 pp

40 30

+10.4 pp

The author’s name is Hina Ameen .

(a)

+6.4 pp

DPO

NPO

SimNPO

SatGA

(b)

Figure 1: (a) Token-level scores on a TOFU forget sample. SEUL, SU-LLM, and S AT I MP assign substantial weight to both forget-specific and structural tokens, whereas ATWU concentrates on the bold ground-truth forget-specific span. (b) ATWU can be combined with diverse forget losses. For each objective, DPO, NPO, S IM NPO, and S AT GA, solid bars denote baseline Unlearning Quality (UQ), and added segments denote the gains from using ATWU token weights with the corresponding forget loss. ATWU consistently improves the forget–retain trade-off.

should be removed. Many tokens—such as common function words, punctuation, and syntactic patterns—are structural and would be generated regardless of exposure to the forget data. Penalizing the generation of such tokens does not promote forgetting of the targeted content, but instead risks degrading general language capabilities. This token heterogeneity has motivated token-weighted unlearning objectives [Yang et al., 2025, Wan et al., 2025, Wang et al., 2025a, Zhou et al., 2026], in which the forget loss is selectively applied to tokens judged to encode the targeted information. Existing approaches, however, still obtain these token weights from external annotations, auxiliary models, probability heuristics, or task-specific rules. In contrast, we characterize token-level forget-specificity through the interaction between the forget and the retain objectives themselves. While prior work typically treats retain conflict merely as a motivation for token selective unlearning, we argue that it can also provide a criterion for selection. A forget token is deemed forget-specific if applying the forget objective to that token is compatible with remaining close to retain optimality, whereas applying the same objective to structural tokens induces retain degradation. This turns token weighting from an external heuristic into a latent variable of the unlearning objective. Contributions.

Our contributions are threefold:

• We give a retain-conflict characterization of token-level forget-specificity and show that a joint token-weighted objective recovers the oracle forget-specific support under a natural separation condition. • We instantiate this principle with ATWU (Alternating Token-Weighted Unlearning): a lightweight scorer parameterizes token weights as a linear direction in the language model’s hidden-state space and is trained by alternating scorer and model updates on the joint objective. • Across TOFU and RWKU, we show that ATWU improves the forget–retain trade-off over sample-level baselines, probability-based token heuristics, and auxiliary-model-based token weighting, while producing token scores that align with ground-truth forget-specific spans.

Related Work

We situate our approach relative to four key areas of prior work: we first review general samplelevel unlearning objectives and their more recent token-weighted variants; we then explore the representation probing literature that motivates our hidden-state scoring mechanism; and finally, we summarize the robust evaluation protocols used to assess unlearning efficacy. 2

Sample-level/token-agnostic unlearning methods. Sample-level LLM unlearning methods specify the forget target at the level of full examples, documents, or question–answer pairs. Gradient Ascent (GA) maximizes the cross-entropy loss on forget samples [Jang et al., 2023], while Gradient Difference (G RAD D IFF) combines this ascent with retain-set descent or regularization [Lu et al., 2022]. Preference-based approaches instead cast unlearning as alignment, including DPO-style refusal objectives [Rafailov et al., 2023], Negative Preference Optimization (NPO) [Zhang et al., 2024], and reference-free variants such as S IM NPO [Fan et al., 2026]. JensUn [Singh et al., 2025] instead replaces the forget and retain training losses with Jensen-Shannon-divergence objectives, yielding a strong sample-level baseline with stable unlearning dynamics. Other methods act on internal states or softened token distributions: RMU perturbs representations associated with the forget set [Li et al., 2024], and UNDIAL distills toward adjusted distributions that reduce memorizedtoken probability [Dong et al., 2025].

Token-weighted unlearning methods. Token-weighted unlearning approaches aim to identify tokens that encode targeted information and selectively apply the forgetting objective to avoid unnecessary utility degradation. These methods typically adopt one of the sample-level objectives above for the actual forget update; they differ primarily in how token forget-specificity is estimated. Auxiliary-model approaches infer it by comparing the predictions of models fine-tuned on retain and forget splits [Eldan and Russinovich, 2023, Wan et al., 2025]. Probability-based heuristics instead use model confidence, entropy or other loss-based criteria as proxies for token forget-specificity [Wang et al., 2025b, Yang et al., 2025, Koh et al., 2026]. Other methods rely on external signals such as linguistic parsers [Dong et al., 2025] or LLM-generated annotations [Zhou et al., 2026] to identify forget-specific tokens. In particular, approaches based on large language model-generated annotations can provide strong supervision, but may be costly to scale and raise additional privacy concerns. In contrast, ATWU learns token-level weights jointly with the unlearning update, using the same retain and forget objectives that define the desired deletion behavior, and requires neither external supervision nor auxiliary model training.

Probes for linguistic properties. A complementary line of work studies what token-level information is encoded in language model hidden representations using probing classifiers, often linear maps or MLPs trained on frozen activations [Alain and Bengio, 2017]. These studies show that contextual token representations encode a broad range of linguistic structure including part-of-speech, morphology [Belinkov et al., 2017], syntactic dependencies, and lexical-semantic distinctions [Clark et al., 2019] and that many such properties are linearly recoverable from intermediate layers of transformer language models [Tenney et al., 2019]. Structural probing work further suggests that hidden-state geometry can reflect syntactic relations directly, rather than merely supporting downstream decoding [Hewitt and Manning, 2019]. While this literature is primarily descriptive and does not address unlearning, it motivates our choice to parameterize token forget-specificity as a lightweight linear scorer over hidden states: if token-level linguistic and semantic attributes are already organized in these representations, then forget-specific information may likewise be identifiable from the same representation space. Furthermore, our formulation allows the scorer to be learned jointly during unlearning without requiring any external supervision.

Unlearning evaluation. Unlearning in LLMs is typically evaluated by balancing forget efficacy against retained utility. TOFU [Maini et al., 2024, Dorna et al., 2026] focuses on fictitious-author question answering, measuring forget quality and model utility. MUSE [Shi et al., 2025] expands evaluation to broader corpora and includes metrics such as memorization, privacy leakage, utility preservation, and scalability. RWKU [Jin et al., 2024] considers real-world entity forgetting alongside general capabilities including reasoning, factuality, and fluency. Recent work highlights limitations of surface-form evaluation: Singh et al. [2025] show that models may appear to forget while still recovering the same information under paraphrased or perturbed inputs, and advocate evaluation based on paraphrases, semantic judging, and worst-case performance across variants. We follow this perspective in our evaluations for a more robust assessment. 3

Token-Weighted Unlearning as a Joint Optimization Problem

We now formalize the token-selection problem underlying ATWU. The goal is to define when a forget token should receive the forget update, and to show that this selector can be identified from the retain and forget objectives themselves. 3.1

Background and Notation

Let V be a finite vocabulary, Θ ⊆ Rd the parameter space, and pθ : V ∗ → ∆(V) an autoregressive language model. We write pθ (xt | x<t ) for the probability of token xt in context x<t assigned by the model and |x| for the length of sequence x. We are given a pretrained model pθ0 , a forget set Df whose influence should be removed, and a retain set Dr on which model performance should be preserved. The common framework for LLM unlearning combines a retain loss ℓr (xt | x<t ; θ), which preserves behavior on retain samples, with a forget loss ℓf (xt | x<t ; θ), which suppresses the original forget completion. The retain loss is typically cross-entropy ℓCE or KL divergence to the pretrained model predictions. Forget losses include negative cross-entropy ℓGA = −ℓCE = log pθ (xt | x<t ) with various scalings and preference-based losses. These objectives are in tension because while the retain loss tries to increase the likelihood of a sample, the forget loss often tries to decrease it. For clarity, we absorb fixed loss coefficients and dataset normalizations into ℓr and ℓf . Finally, we write P P|x| R(θ) := x∈Dr t=1 ℓr (xt | x<t ; θ) for the retain objective and R⋆ := minθ∈Θ R(θ) for its minimum over the parameter space. 3.2

Forget-Specific vs. Structural Tokens

The forget set contains both information that should be removed and linguistic structure that should be preserved. Tokens such as names, dates, quantities, rare identifiers, and attribute-bearing phrases can identify the targeted knowledge in Df ; we call these tokens forget-specific. In contrast, function words, punctuation, syntactic patterns, and generic continuations are structural: they are useful under the retain distribution and would often be generated even without exposure to the forget data. Applying the forget loss to forget-specific tokens removes targeted information, while applying it to structural tokens can instead damage general language capabilities. This distinction has motivated the introduction of token-level weights zx,t ∈ [0, 1] approximating the ⋆ oracle binary label zx,t = 1{xt is a forget-specific token} , and the token-weighted loss L(θ, z) =

|x| XX

ℓr (xt | x<t ; θ) +

x∈Dr t=1

|x| XX

zx,t ℓf (xt | x<t ; θ).

(1)

x∈Df t=1

If the oracle labels z⋆ were known, re-weighted unlearning could simply compute a minimizer θ ⋆ ∈ arg minθ L(θ, z⋆ ). This oracle setting is useful as an analysis tool—Table 3 later shows that using ground-truth token labels can indeed substantially improve the forget-retain trade-off—but such labels are unavailable or very costly in practice. Prior token-weighted methods therefore approximate z⋆ using annotations, auxiliary models, or probability-based proxies such as confidence, saturation, and surprisal. These methods, which adopt the formulation in Equation 1, are likewise motivated by the retain conflict: minimizing forget-loss on structural tokens harms retain performance, while doing so on the forget-specific tokens does not. The key difference in our method from prior works is that we use retain conflict as a token-level identification criterion. 3.3

Characterizing Forget-Specificity via Retain Conflict

Let IF := {(x, t) : x ∈ Df , t ∈ [|x|]} denote the set of forget sequence position pairs and NF := |IF | be the number of tokens in the forget set. For each i = (x, t) ∈ IF , write ℓif (θ) = ℓf (xt | x<t ; θ). For any subset A ⊆ IF , define the retain conflict " # X ⋆ i min κ(A) := min R(θ) − R + ℓf (θ) − ℓf , (2) θ∈Θ

i∈A

:= minθ,i ℓif (θ) is the global minimum of the forget loss. Each summand ℓif (θ)−ℓmin where ℓmin ≥ 0, f f and therefore we have κ(A) ≥ 0. Moreover, κ is monotone: for A ⊆ B, κ(A) ≤ κ(B). We write κi := κ({i}) for the singleton conflict. This quantity is the best possible residual cost of simultaneously preserving the retain performance and driving the forget loss for token i ∈ IF to its floor. A token with small conflict can be suppressed without moving far from a retain-optimal model, so it is a plausible forget-specific token. A token with large conflict cannot be suppressed without sacrificing retain performance, which is precisely the behavior expected of structural tokens. We give a formal characterization of this intuition with the next assumption. Assumption A1 (Retain-conflict separation). There exist ε ≥ 0, δ > 0, and a partition IF = F ⋆ ∪S ⋆ with |F ⋆ | = ρ⋆ NF , where the oracle labels are zi⋆ = 1{i∈F ⋆ } , such that κ(F ⋆ ) ≤ ε

κi ≥ ε + δ

and

∀i ∈ S ⋆ .

(3)

This assumption characterizes the forget-specific tokens as the tokens that can be forgotten while remaining in the retain optimum, and structural tokens as those whose unlearning forces the model outside the retain optimum. While this conceptual separation is elegant, using it directly as an algorithmic selection criterion is intractable. Computing the singleton conflict κi demands a full optimization over the model parameters θ for each of the NF forget tokens. To bypass this computational bottleneck, we show that under the retain-conflict separation, the oracle labels z⋆ as well as θ ⋆ are instead recovered as the minimizer of the joint problem P min L(θ, z), Z := z ∈ {0, 1}NF : i∈IF zi = ρNF . (4) θ∈Θ, z∈Z

The budget constraint Z prevents trivial minimizers of the form z = 0 or z = 1, which do not identify a forget-specific subset. Theorem 3.1 (Recovery). Under Assumption A1, suppose 0 < ρ ≤ ρ⋆ and ρNF is an integer. If (θ̂, ẑ) is any global minimizer of (4), then supp(ẑ) ⊆ F ⋆ . If ρ = ρ⋆ , then supp(ẑ) = F ⋆ and θ̂ = θ ⋆ . The proof is deferred to Appendix B. Theorem 3.1 shows that the joint fixed-budget objective recovers the oracle token subset under retain-conflict separation. In particular, for ρ ≤ ρ⋆ , every global minimizer selects only tokens in F ⋆ , and for ρ = ρ⋆ , it recovers both the oracle support z⋆ and the corresponding oracle minimizer θ ⋆ . Notably, this recovery argument only requires the token-wise forget loss to be bounded below—a condition satisfied by the saturated negative cross-entropy loss used in our ATWU instantiation (Section 4). The theorem therefore justifies learning token weights jointly with model parameters directly from the unlearning objective, without requiring externally specified token labels. In practice, ρ serves as a scalar hyperparameter controlling the budget of this learned selector.

Alternating Token-Weighted Unlearning

We now turn the joint formulation into a practical learning algorithm: Alternating Token-Weighted Unlearning (ATWU). We first relax the discrete optimization problem, then introduce the hidden-state scorer, specify and justify the selection of forget loss used in our experiments, and finally describe the alternating optimization procedure. Lagrangian relaxation. The constrained problem (4) is combinatorial in z. We therefore instead optimize a continuous penalized objective 2 X X L̃(θ, z) = L(θ, z) + λH H(zx,t ) + λρ N1F zx,t − ρ , (5) zx,t ∈z

zx,t ∈z

where H(z) = −z log z − (1 − z) log(1 − z) is the binary entropy. The objective L̃(θ, z) relaxes the original combinatorial problem (4) on two fronts: the binary P constraint z ∈ {0, 1}NF is relaxed to z ∈ [0, 1]NF , and the budget constraint zx,t = ρNF is replaced by the quadratic penalty. The entropy term counters the first relaxation by pushing each zx,t back towards {0, 1}. This relaxation can be exact under sufficiently large regularization. 5

Lemma 4.1 (Exactness of the relaxation, informal). For sufficiently large λH and λρ , every global minimizer of the relaxed objective (5) over [0, 1]NF is binary, satisfies the budget constraint, and is a global minimizer of the constrained problem (4). The formal statement and proof are given in Lemma B.1. While this free-token-weight formulation provides strong theoretical guarantees, learning an independent scalar for every token is unscalable for large corpora and cannot generalize to unseen sequences. To bridge this gap from theory to practice, ATWU parameterizes the selector using a shared scoring function gw . Under this scalable parameterization, the entropy and budget terms naturally transition from enforcing exact combinatorial constraints to acting as principled regularizers, successfully guiding the network toward sparse, budget-controlled token selection. To instantiate this shared scorer, we parameterize token forget specificity using a simple linear projection over the language model’s hidden representations. This architectural choice is directly motivated by the structural probing literature: because intermediate hidden states natively encode rich linguistic and semantic properties [Tenney et al., 2019], they provide an ideal representation space for identifying forget-specific patterns without requiring a complex auxiliary network. Let hθ (xt ) ∈ Rd denote the t-th hidden representation produced by pθ for the sequence x. ATWU defines ẑx,t = gw (hθ (xt )) = σ(⟨w, hθ (xt )⟩) ∈ (0, 1), (6) where w ∈ Rd is the learned scorer parameter. This single linear projection keeps the mechanism lightweight while encouraging reusable token-level patterns rather than independent per-token decisions. We initialize w = 0d , meaning all tokens initially receive a uniform score of 1/2. Consequently, the objective begins token-agnostic and smoothly becomes selective during training. As a proof of concept, Appendix E.1 shows that this exact linear architecture can recover ground-truth forget labels when trained with explicit supervision, i.e., the scorer model has enough capacity to model the true forget specificity; ATWU, however, learns the scores entirely implicitly from the unlearning objective. Choice of loss functions. We use standard cross-entropy as the retain loss and a saturated negative cross-entropy introduced by Wang et al. [2025b] as the base forget loss. For a hyperparameter β > 0, the token-wise forget loss is defined as: ℓf (xt | x<t ; θ) := pθ (xt | x<t )β · log pθ (xt | x<t ).

(7)

This loss formulation incorporates the saturation weighting mechanism utilized by [Wang et al., 2025b] inside the loss, which we will refer to as S AT GA. While saturation alone is a poor proxy for token-level forget-specificity (as evidenced by the poor AUROC of the saturation score in Fig. 2), it provides highly desirable gradient scaling properties for the loss function. Crucially, unlike raw negative cross-entropy ℓGA , this saturated variant is bounded below. As a function of p ∈ (0, 1], the term pβ log p reaches a minimum of −1/(eβ) at p = e−1/β . Consequently, the hyperparameter β effectively controls the target probability level for the forget update, with smaller β values enforcing stronger suppression. To instantiate our general joint-learning framework using this bounded objective, we introduce a slight modification: we also inject the dynamically learned token score into the saturation exponent. We refer to this score-modulated variant as S AT GA+ . Plugging this into the joint formulation yields our primary ATWU objective: X gw (xt ) · pθ (xt | x<t )βgw (xt ) · log pθ (xt | x<t ) LATWU (θ, w) = αR(θ) + γ x,t

+ λH

P 2 H gw (xt ) + λρ N1F x,t gw (xt ) − ρ ,

(8)

x,t

where gw (xt ) := gw (hθ (xt )). For strictly binary scores, the S AT GA+ modification reduces to standard S AT GA. However, because our scores gw are continuous during training, the modified exponent forces uncertain tokens to exert a much smoother, attenuated forget update. This stabilizes the early phases of optimization before the 6

Unlearning

Method

Utility

Unlearning

Utility

UQ ↑

MMLU

Rep.

FQ ↑

RD ↓

UQ ↑

MMLU

Rep.

0.0

66.6

546

50.0

0.0

70.1

529

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N RMU WGA S AT I MP

39.7 59.5 48.0 69.3 98.0 87.2 66.8 78.6

17.3 15.0 5.0 6.5 9.6 1.3 4.8 3.0

22.4 44.5 43.0 62.8 88.3 85.9 62.1 75.6

66.1 64.3 66.2 65.8 65.3 65.3 66.0 65.9

545 533 551 541 236 540 550 549

48.0 38.5 48.5 50.5 11.0 50.5 60.0 53.5

81.9 12.3 47.6 51.8 85.4 — 78.0 77.3

36.7 17.0 39.1 29.7 36.1 — 26.1 29.3

45.2 0.0 8.5 22.1 49.3 — 51.9 48.0

69.2 69.7 68.1 68.8 70.2 — 69.4 69.8

514 523 543 529 523 — 518 518

43.0 43.0 33.5 45.5 46.5 — 41.0 40.0

95.2

3.5

91.7

66.5

558

58.0

81.4

22.7

58.7

70.3

519

46.0

ATWU

RWKU

RD ↓

0.0

TOFU

FQ ↑ O RIGINAL

FQ: forget quality, the relative reduction in worst-case forget-set judge score. RD: retain degradation, the relative loss in retain-set judge score. UQ = [FQ − RD]+ : net forget–retain trade-off. Definitions in Appendix C.5.

Table 1: ATWU achieves the best UQ on both benchmarks while preserving utility close to the original checkpoint, the only method to do so consistently across the six metric panels. Left: TOFU forget10 with Llama-3.1-8B-Instruct; right: canonical RWKU ten-subject batch with Phi-3-Mini-4k-Instruct. Higher-FQ competitors such as J ENS U N forget aggressively but collapse generation quality; RMU matches ATWU on retain degradation but trails on UQ. Best (bold) and second-best (underlined) per column; utility values that are materially degraded (MMLU drop > 2pp, Rep. drop > 5%, or WR < 45) are shown in red.

scorer has fully converged, an effect we ablate directly in Table 3. The coefficients α and γ denote the retain and forget loss weights, respectively. While we absorbed these constants into the base loss definitions in earlier sections for notational simplicity, we make them explicit here to reflect our exact empirical objective. Alternating optimization. We optimize LATWU (θ, w) by alternating between language-model and scorer updates. With the scorer w fixed, the language-model parameters θ are updated using the current token-weighted forget objective together with the retain objective. With θ fixed, the scorer w is updated to improve token selection under the same regularized objective. During model updates, the scores are detached and treated as fixed coefficients; during scorer updates, θ is frozen and gradients flow only through w. This scheduled alternation is empirically more stable than updating the scorer and model in lockstep: the scorer changes the effective forget objective, while the model changes the hidden-state geometry on which the scorer depends. Updating them on separate timescales reduces this feedback loop. Finally, in Appendix E.3 we present an ablation on the update frequency that shows that the alternating variant outperforms joint updates.

Experiments

Our experiments are designed to test whether the learned token weights are accurate, useful for end-to-end unlearning, and whether the alternating token-weighted unlearning (ATWU) framework generalizes to loss functions beyond saturated negative cross-entropy. First, we compare ATWU against state-of-the-art unlearning methods across multiple models and datasets. Second, we evaluate the quality of the forget-specificity scores learned by the ATWU scorer, comparing them against alternative token-weighting approaches. Third, we demonstrate the versatility of our framework by testing whether ATWU consistently improves unlearning quality when used with other common forget losses. Benchmarks, models, and training. We evaluate ATWU on two LLM unlearning benchmarks. TOFU [Maini et al., 2024] is a synthetic question–answering benchmark built from fictitious-author biographies. We report main results on forget10 with Llama-3.1-8B-Instruct [Grattafiori et al., 2024], using the checkpoint released by Dorna et al. [2026]. Results on forget01 and forget05, as well as all three splits with Llama-3.2-1B-Instruct, are qualitatively consistent and deferred to Appendix D.4 for brevity. RWKU [Jin et al., 2024] targets real-world public-figure knowledge in a pretrained LLM. Following Singh et al. [2025], we perform batch unlearning on a fixed canonical 7

batch of ten subjects using Phi-3-Mini-4k-Instruct [Abdin et al., 2024]. Hardware, software, and optimization details used across all settings are provided in Appendix D. Hyperparameter tuning. For each method, we tune method-specific hyperparameters, including learning rates and loss coefficients, via Bayesian optimization [Akiba et al., 2019], seeded with each method’s recommended configuration and guided by a cheap surrogate metric (ES∆ for TOFU, N∆ for RWKU). Crucially, we find that strong baselines are often obscured by limited tuning budgets. With adequate per-method tuning, several prior methods perform markedly better than previously reported. RMU serves as a prime example: it becomes highly competitive on TOFU and surpasses several recently proposed methods—a finding that contradicts earlier literature. We view this outcome not as a critique of prior work, but as a broader motivation for rigorous, uniform tuning protocols in unlearning evaluations, a point we expand upon in Appendix D.4. Evaluation. We use the paraphrase- and judge-based evaluation protocol of Singh et al. [2025], with GPT-5.4-mini as the judge. We report three baseline-relative summary metrics: forget quality (FQ), which measures the reduction in worst-case forget-set judge score, retain degradation (RD), which measures the loss in retain-set judge score, and unlearning quality UQ = [FQ − RD]+ , which summarizes the net forget–retain trade-off. Unlike metrics like token-overlap with the original answers or likelihood-based surrogates, this protocol penalizes methods that suppress only the original surface form while leaving semantically equivalent completions recoverable. We complement these summary metrics with three utility-preservation probes: MMLU accuracy, generation repetitiveness (Rep.), and pairwise win rate against the original checkpoint (WR). Detailed definitions and discussions on the surrogate, judge-based and utility metrics are provided in Appendix C. Full metric panels are reported in Appendix D. 5.1

Unlearning Quality

Main results. Table 1 reports the main FQ/RD/UQ and utility results on the two largest experimental configurations: TOFU forget10 with Llama-3.1-8B-Instruct and the canonical RWKU ten-subject batch with Phi-3-Mini-4k-Instruct. ATWU achieves the best UQ on both benchmarks: 91.7 on TOFU, exceeding the runner-up RMU by 5.8 percentage points, and 58.7 on RWKU, exceeding WGA by 6.8 percentage points. Among the strongest unlearning methods, ATWU is also the only one that avoids a major degradation on the utility axes we measure. It attains the highest MMLU score on both benchmarks, the strongest repetitiveness score on TOFU, and remains competitive in pairwise win rate. Results are consistent across the smaller forget01 and forget05 splits and across model scales, as shown in Tables 8 and 9. ATWU with other forget losses. So far, we Relative Utility have instantiated ATWU using a saturated Method negative cross-entropy loss. However, our UQ ↑ ∆UQ ↑ MMLU Rep. WR joint-learning framework is fundamentally lossO RIGINAL 0.0 — 70.1 529 50.0 agnostic. Any forget loss that decomposes autoregressively over tokens (Eq. 1) can be ATWUDPO 6.4 +6.4 69.5 512 36.0 seamlessly plugged into the formulation. More- ATWUNPO 18.9 +10.4 67.7 545 35.5 over, this flexibility extends to forget objec- ATWUSimNPO 37.8 +15.7 69.9 531 43.5 tives constructed from an autoregressive buildATWU 58.7 +6.8 70.3 519 46.0 ing block—such as the token-wise NLL of the forget sequence, or its log-ratio against a frozen reference—even if they do not perfectly match Table 2: RWKU ten-subject batch results of ATWU the exact formulation in Eq. 1. In such cases, with various forget losses. ∆UQ reports the gain we simply replace the autoregressive block with over each method’s vanilla counterpart in Table 1. its scorer-reweighted analog, leaving the rest of the loss untouched. This adaptation applies cleanly to DPO, NPO, and S IM NPO. Crucially, the training procedure remains identical: the model and the token scorer are still optimized in an alternating manner from scratch during unlearning. Figure 1b and Table 2 demonstrate the unlearning quality of ATWU instantiated with DPO, NPO, and S IM NPO, alongside our primary S AT GA+ formulation for comparison. ATWU improves UQ over the unweighted version of every loss by 6.4 to 15.7 percentage points—evidence that the performance gains from ATWU are not restricted to a specific choice of loss function, even though the underlying base loss dictates the overall performance ceiling. Further details are provided in Appendix E.2. 8

5.2

Comparison with other Token Weighting Methods

Score quality. We first evaluate the learned scores directly. Figure 2 ranks tokens within each TOFU forget sample by each method’s score and computes AUROC against the ground-truth labels from Zhou et al. [2026]. S ATURATION is the only method that performs below the random baseline of 50 (33 ± 17). The auxiliary-model approaches LLM D IFF, N-G RAM D IFF, and L OG P ROB M ASK do better but stall in the 54–63 range. The strongest baselines—I MPORTANCE, E NTROPY, and N OUN M ASK—cluster around 67–68. Our ATWU learned scores stand clearly above all of them at 75 ± 9, combining the highest mean with the lowest variance among the compared methods. This indicates that ATWU’s gains come not only from how the token weights are used, but also from a substantially stronger token-level forget-specificity signal than that of competing token-weighting mechanisms. Unlearning quality. The superior quality of Saturation n-gram diff the ATWU scores directly translates to unLLM diff learning performance: as shown in Table 4, logprob Mask Importance ATWU achieves the strongest forget–retain Entropy trade-off among all token-weighted approaches. Noun Mask Ours It secures the best FQ and UQ, remains competitive on RD, and preserves general util33 ± 17 ity close to the original checkpoint. Table 4 54 ± 13 also details how the seven representative token61 ± 14 weighted baselines following the formulation in Equation 1 approximate token-level forget63 ± 17 specificity. The choice of forget loss varies 67 ± 17 between these methods: FUNDIAL uses an engineered distillation loss, S AT I MP and ATWU 68 ± 17 use S AT GA and S AT GA+ forget losses, re68 ± 10 spectively, and all other methods use the token75 ± 9 wise negative CE loss. Notably, WGA can be viewed as using the S AT GA loss without any 0 20 40 60 80 100 token weighting. Despite its saturation scoring ranking below the random baseline for forget- Figure 2: Per-sample AUROC distributions for specificity, WGA still reaches the third-highest token-level forget-specificity on TOFU forget10, UQ in the table, outperforming methods with scored against the ground-truth token labels of Zhou better forget-specificity scores that rely on the et al. [2026]. Labels report mean ± std across forget naive negative cross-entropy loss (GA). This samples. Method-to-scoring-criterion mapping in highlights the crucial role of the saturation term Table 3; pooled per-token ROC curves in Figure 10. within the loss, as its desirable gradient-scaling properties help prevent over-forgetting. Finally, consistent with the failure modes of surface-level metrics identified by Singh et al. [2025], FUNDIAL nearly matches ATWU on the ES∆ surrogate, yet its judge-based UQ is roughly half as large. This stark mismatch reinforces the necessity of robust, judge-based evaluations. Disentangling the loss and scores. To isolate the effects of score quality and the unlearning objective, we evaluate two score sources alongside three forget losses. We pair ground-truth binary labels (GT) from Zhou et al. [2026] and our unsupervised ATWU scorer with naive GA, saturated cross-entropy (S AT GA), and our scoremodulated variant (S AT GA+ ). For binary GT labels, S AT GA and S AT GA+ coincide.

Relative

Utility

Score

Loss

O RIGINAL

—

0.0

45.1

559 50.0

GA S AT GA

47.4 90.2

7.9 3.9

39.5 86.2

45.0 44.9

563 54.5 566 51.0

ATWU

GA S AT GA S AT GA+

31.0 61.9 84.4

3.8 3.4 6.3

27.2 58.5 78.1

45.3 45.1 45.0

565 55.5 569 52.0 575 51.5

FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

This ablation reveals that high-quality token Table 3: Oracle score and weightingscores cannot compensate for a brittle unlearn- scheme ablation on TOFU forget10 ing objective. Strikingly, combining perfect or- (Llama-3.2-1B-Instruct). acle labels with naive GA yields a weak forget– retain trade-off (UQ = 39.5). In fact, WGA, which uses no token-level information but applies a saturated loss, outperforms the GT–GA oracle (UQ = 41.5). This demonstrates that saturation provides a critical intrinsic brake against over-forgetting structural tokens. However, saturation alone 9

Surrogate

Relative

Utility

Method

Token weights zx,t

O RIGINAL

—

0.706

0.737

0.030

0.0

45.1

559 50.0

SEUL SU-LLM SU-N GRAM FUNDIAL ETW WGA S AT I MP

logprob mask LLM diff n-gram diff noun mask entropy saturation importance

0.101 0.190 0.074 0.047 0.108 0.062 0.043

0.675 0.704 0.634 0.759 0.679 0.759 0.747

0.574 0.514 0.559 0.712 0.572 0.696 0.704

42.6 31.3 55.4 40.9 48.7 50.0 64.7

6.9 4.1 15.8 5.6 9.3 8.5 4.1

35.6 27.2 39.7 35.4 39.4 41.5 60.7

45.4 45.2 45.3 45.2 45.5 45.0 44.8

559 564 560 562 561 568 570

ATWU

gw (xt )

0.035

0.753

0.717

84.4

6.3

78.1

45.0

575 51.5

ESF ↓ ESR ↑ ES∆ ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

59.5 56.0 51.5 53.5 54.5 56.0 55.0

S AT I MP and ATWU use the S AT GA and S AT GA+ forget losses respectively; FUNDIAL uses its own model-distillation forget loss; all other methods use the negative cross-entropy forget loss.

Table 4: ATWU outperforms other token-weighted unlearning approaches on TOFU forget10 (Llama-3.2-1B-Instruct). Best and second-best methods are highlighted.

is not the ceiling. When the oracle scores are properly paired with a saturated loss (GT–S AT GA), performance skyrockets by 46.7 points to UQ = 86.2, establishing the supervised upper bound. Crucially, our full ATWU framework successfully bridges the gap to this supervised oracle without requiring any external annotations. Replacing the GT labels with our learned ATWU scores under the standard S AT GA loss yields a respectable UQ = 58.5. Our S AT GA+ formulation then adds an additional 19.6 points by placing the continuous score in the saturation exponent to allow for smoother, uncertainty-aware updates. By co-designing the learned scorer with an adaptive objective, ATWU approaches the theoretical supervised ceiling in a fully unsupervised manner.

Limitations

Despite its promising results, ATWU has several limitations. Because the token scorer is learned from the forget set during unlearning, very small forget sets may provide insufficient signal, which is reflected in the weaker gains on the smallest TOFU splits (see discussion in Appendix D.4). The theoretical recovery guarantee relies on estimating the true token-selection budget as a hyperparameter (ρ). Furthermore, it assumes a strict separation between forget-specific and structural tokens—a clean dichotomy that may only hold approximately in natural language. Additionally, while the exact recovery guarantees of Theorem 3.1 and Lemma 4.1 are formally established for the multiplicative ATWU objective, our primary empirical formulation (S AT GA+ ) injects the learned score directly into the saturation exponent. Although this introduces a slight gap between our formal linear assumptions and our practical implementation, it serves as a highly valuable heuristic for stabilizing the early phases of optimization, and the overall framework remains fundamentally informed by our theoretical insights. Empirically, our experiments cover two benchmarks and two model families, and final evaluations are single-run due to compute and judge-model cost. Finally, our primary semantic evaluation relies on an LLM judge; although validated against human annotations, it remains an imperfect and potentially model-dependent measurement tool.

Conclusion

We introduced ATWU, shifting the paradigm of token-weighted unlearning from a pipeline of external heuristics to an end-to-end joint optimization problem. By formalizing token specificity through retain conflict, we demonstrated that the unlearning objective itself contains sufficient latent signal to recover forget-specific tokens without any external supervision. 10

Beyond establishing a new state-of-the-art on TOFU and RWKU, our empirical findings challenge a prevailing assumption in the literature: that better token identification automatically yields better unlearning. Our ablations reveal that score quality and unlearning efficacy are frequently decoupled. A brittle loss function renders even perfect oracle labels ineffective, whereas an adaptive, saturated loss can elevate the performance of uniform token weighting. This demonstrates that the true bottleneck in selective unlearning is not merely finding the right tokens, but the principled co-design of the scoring mechanism and the forget objective. Finally, while explicitly annotated token labels offer a theoretical performance ceiling, they introduce severe privacy and scalability risks that directly contradict the core motivation of machine unlearning. ATWU circumvents this paradox. By recovering the majority of the supervised oracle benefit using only implicit model representations, our work establishes that scalable, privacy-preserving, and highly targeted unlearning is achievable without relying on external supervision.

Acknowledgments The authors would like to thank Francesco Croce for helpful discussions. This work was partially funded by the grant number 212111 from the Swiss National Science Foundation and a grant from Coefficient Giving, administered by the Berkeley Existential Risk Initiative (BERI). Gizem Yüce is supported by the Swiss AI Fellowship.

References Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, and Others. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv.org/abs/2404.14219. 8, 26 Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019. 8, 26 Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL https://openreview.net/forum?id=ryF7rTqgl. 3 Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, Kathleen Fisher, Tatsunori Hashimoto, Dan Hendrycks, Somesh Jha, Daniel Kang, Florian Kerschbaum, Eric Mitchell, John Mitchell, Zulfikar Ramzan, Khawaja Shams, Dawn Song, Ankur Taly, and Diyi Yang. Identifying and mitigating the security risks of generative ai. Foundations and Trends® in Privacy and Security, 6(1):1–52, December 2023. ISSN 2474-1566. doi: 10.1561/3300000041. URL http://dx.doi.org/10.1561/3300000041. 1 Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1080. URL https://aclanthology.org/P17-1080/. 3 Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159, 2021. doi: 10.1109/SP40001.2021. 00019. 1 Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In Proceedings of the 2015 IEEE Symposium on Security and Privacy, SP ’15, page 463–480, USA, 2015. IEEE Computer Society. ISBN 9781467369497. doi: 10.1109/SP.2015.35. URL https://doi.org/10.1109/SP.2015.35. 1 Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= TatRHT_1cK. 1 Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention. In Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4828. URL https://aclanthology. org/W19-4828/. 3 Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. Undial: Selfdistillation with adjusted logits for robust unlearning in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8827–8840, 2025. 3 Vineeth Dorna, Anmol Reddy Mekala, Wenlong Zhao, Andrew McCallum, J Zico Kolter, Zachary Chase Lipton, and Pratyush Maini. Openunlearning: Accelerating LLM unlearning via unified benchmarking of methods and metrics. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URL https://openreview.net/forum?id=Gy67Zh5X1i. 3, 7, 26, 29, 32 Ronen Eldan and Mark Russinovich. Who’s Harry Potter? Approximate unlearning in LLMs. arXiv, October 2023. URL https://www.microsoft.com/en-us/research/publication/ whos-harry-potter-approximate-unlearning-in-llms/. 3 12

Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for LLM unlearning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https: //openreview.net/forum?id=JbvSQm5h1l. 3, 37 Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602. 24, 29 Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 7, 26 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In The Ninth International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id= d7KBjmI3GmQ. 24 John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1419. URL https://aclanthology.org/N19-1419/. 3 Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408, 2023. 3 Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. RWKU: Benchmarking real-world knowledge unlearning for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=wOmtZ5FgMH. 3, 7, 22, 24, 26 Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403–7412, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main. 458. URL https://aclanthology.org/2023.emnlp-main.458/. 1 Seunghee Koh, Sunghyun Baek, Youngdong Kim, and Junmo Kim. Forget what matters, keep the rest: Selective unlearning of informative tokens, 2026. URL https://arxiv.org/abs/2604.17785. 3 Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam Alfred Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview. net/forum?id=xlr6AUDuJz. 1, 3, 31 Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022. 3 13

Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=B41hNBoWLo. 3, 7, 22, 26 Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id= vjel3nWP2a. 1 Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. ACM Trans. Intell. Syst. Technol., 16(5), September 2025. ISSN 2157-6904. doi: 10.1145/3749987. URL https://doi.org/10.1145/3749987. 1 OpenAI. GPT-5.4 mini, 2026. URL https://developers.openai.com/api/docs/models/ gpt-5.4-mini. OpenAI API documentation. 29 Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// openreview.net/forum?id=HPuSIXJaa9. 3, 36 Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning sixway evaluation for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TArmA033BU. 3 Naman Deep Singh, Maximilian Müller, Francesco Croce, and Matthias Hein. Unlearning that lasts: Utility-preserving, robust, and almost irreversible forgetting in llms, 2025. URL https: //arxiv.org/abs/2509.02820. 3, 7, 8, 9, 22, 23, 24, 25, 26, 27, 32 Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL https: //aclanthology.org/P19-1452/. 3, 6 Yixin Wan, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Rahul Gupta. Not every token needs forgetting: Selective unlearning balancing forgetting and utility in large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1827–1835, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176335-7. doi: 10.18653/v1/2025.findings-emnlp.96. URL https://aclanthology.org/2025. findings-emnlp.96/. 2, 3 Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 843–851, 2025a. 2 Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q Weinberger. Rethinking LLM unlearning objectives: A gradient perspective and go beyond. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview. net/forum?id=huo8MqVH6t. 3, 6, 22, 31 Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Exploring criteria of loss reweighting to enhance LLM unlearning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=mGOugCZlAq. 2, 3, 27, 31 Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=MXLBXjQkmb. 3, 36 14

Xiangyu Zhou, Yao Qiang, Saleh Zare Zade, Douglas Zytko, Prashant Khanduri, and Dongxiao Zhu. Not all tokens are meant to be forgotten. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 38173–38182, 2026. 2, 3, 9, 34

Appendix Appendix Overview A Qualitative Examples

B Deferred Proofs

B.1 Proof of Theorem 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.2 Proof of Lemma 4.1 (Formal Statement) . . . . . . . . . . . . . . . . . . . . . . .

C Unlearning Metrics

C.1 Token-level Extraction Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Robustness-Aware Forget and Retain Quality . . . . . . . . . . . . . . . . . . . .

C.3 Utility Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.4 Native RWKU Benchmark Metrics . . . . . . . . . . . . . . . . . . . . . . . . . .

C.5 Baseline-Relative Summary Metrics . . . . . . . . . . . . . . . . . . . . . . . . .

D Experimental Details

D.1 Benchmarks and Base Models . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.2 Hyperparameter Tuning Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.3 Hyperparameter Search Ranges and Results . . . . . . . . . . . . . . . . . . . . .

D.4 Final Unlearning Runs and Evaluation . . . . . . . . . . . . . . . . . . . . . . . .

E Additional Experiments

E.1 Linear Separability of Informative Tokens . . . . . . . . . . . . . . . . . . . . . .

E.2 ATWU with different forget losses . . . . . . . . . . . . . . . . . . . . . . . . . .

E.3 Ablations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

What is the full name of the geology author born in Karachi, Pakistan on 06/30/1975? The author’s name is Hina Ameen .

The author’s name is Hina Ameen .

What are the professions of Carmen Montenegro’s parents? Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist . Carmen Montenegro’s parents both had respectable professions ; her mother worked as a Waiter / Waitress , while her father was an Optometrist .

Figure 3: Token-level forget-specificity from SEUL, SU-LLM, SU-N GRAM, FUNDIAL, ETW, WGA, S AT I MP, and ATWU on two TOFU forget samples. Shading reflects each method’s raw token score; bold spans mark the ground-truth forget-specific tokens; ATWU concentrates most clearly on the answer-bearing spans.

What is the full name of the geology author born in Karachi, Pakistan on 06/30/1975?

The author’s name is Hina Ameen . The author’s name is Hina Ameen .

The author’s name is Hina Ameen .

Figure 4: Evolution of ATWU’s token scores during training. The scorer starts from uniform scores and gradually concentrates on the ground-truth forget-specific span.

Qualitative Examples

We complement the quantitative results with qualitative examples of the token-level scores learned by ATWU. The goal is to inspect whether the scorer identifies the tokens that actually carry the forget target, rather than assigning high weight to generic context, function words, or syntactic scaffolding. In each example below, bold tokens indicate the ground-truth forget-specific span, and token shading indicates the importance score assigned by the corresponding method. Figure 3 compares ATWU with several token-importance baselines on two TOFU forget samples. The first example is a short answer in which the forget-specific content is the author name, while the second is a longer answer in which the forget-specific content consists of the parents’ professions. The binary scorers—SEUL, SU-LLM, SU-N GRAM, and FUNDIAL—tend to mark large portions of the answer as important, including tokens that are not specific to the forgotten fact. Probability-based heuristics such as WGA and S AT I MP are less binary but often place substantial mass on structural tokens and surrounding context. ETW, which weights by token-level prediction entropy, identifies the answer name cleanly on the short example, but on the longer one it spreads mass over model-uncertain tokens that are not forget-specific (e.g., respectable, both, had). In contrast, ATWU assigns most of its mass to the answer-bearing spans while keeping generic tokens relatively low. This behavior is especially visible in the longer example, where the scorer separates the profession tokens from the surrounding sentence template. Figure 4 visualizes how the ATWU scorer evolves during training on the short TOFU example. At initialization, all tokens receive the same score, so the forget loss is effectively token-uniform. As training progresses, the scorer first assigns nontrivial mass to several tokens in the answer, but gradually suppresses structural tokens and concentrates on the ground-truth forget-specific span. This trajectory illustrates the intended behavior of the alternating optimization: token selectivity is not imposed by external labels, but emerges as the scorer and unlearned model are optimized together. Figure 5 summarizes this trend quantitatively by tracking token-ranking AUROC over training on TOFU forget10. The progression shows that the learned scores become increasingly aligned with the ground-truth forget-specific tokens over the course of optimization. This provides additional evidence that the scorer is not merely fitting token frequency or likelihood artifacts, but is learning a token-level signal that better matches the information to be removed.

50 ± 0 59 ± 15 74 ± 10 75 ± 9 75 ± 9

Finally, Figure 6 tests whether the learned scorer 75 ± 9 transfers beyond the exact forget samples used 0 20 40 60 80 100 during training. We construct two unseen ex- init step 25 amples by taking TOFU-style questions and replacing the fictional entities with real people and Figure 5: Progression of the per-sample AUROC their corresponding factual attributes. Although distribution of ATWU’s learned token scores durthese examples are not part of the scorer’s train- ing training on TOFU. 18

What is the name and sport of the person with the most Olympic gold medals?

The person with the most Olympic gold medals is Michael Phelps , and his sport is swimming .

Did Barack Obama own any pets during his presidency?

During his presidency , Barack Obama owned two Portuguese Water Dogs , named Bo and Sunny .

Figure 6: Learned ATWU scores on unseen real-entity examples. The examples use TOFU-style prompts with fictional entities replaced by real people and factual information. ing data, ATWU still assigns elevated scores to the answer-bearing tokens—the name and sport tokens (Michael, Phelps, swimming) in the Olympic example, and the pet-identifying tokens (Water Dogs, Bo, Sunny) in the Obama example—while leaving most filler and syntactic tokens low. This indicates that the scorer is not merely memorizing the training entities or surface templates, but learns a transferable token-level signal for identifying forget-specific content.

Deferred Proofs

B.1

Proof of Theorem 3.1

Theorem 3.1 (Recovery). Under Assumption A1, suppose 0 < ρ ≤ ρ⋆ and ρNF is an integer. If (θ̂, ẑ) is any global minimizer of (4), then supp(ẑ) ⊆ F ⋆ . If ρ = ρ⋆ , then supp(ẑ) = F ⋆ and θ̂ = θ ⋆ . Proof. Write the optimal value of (4) as V ⋆ . For any A ⊆ IF with |A| = ρNF , the definition of κ(A) gives min L(θ, zA ) = R⋆ + ρNF ℓmin + κ(A), (9) F θ

where zA denotes the indicator of A. Since |F ⋆ | = ρ⋆ NF ≥ ρNF (using ρ ≤ ρ⋆ ), we can choose any S ⊆ F ⋆ with |S| = ρNF ; by monotonicity and Assumption A1, κ(S) ≤ κ(F ⋆ ) ≤ ε, so V ⋆ ≤ min L(θ, zS ) = R⋆ + ρNF ℓmin + κ(S) ≤ R⋆ + ρNF ℓmin + ε. F F θ

For contradiction, suppose some global minimizer (θ̂, ẑ) of (4) has A := supp(ẑ) containing some (x0 , j0 ) ∈ S ⋆ . Since zA = ẑ is feasible and (θ̂, ẑ) is optimal, minθ L(θ, zA ) = V ⋆ . By monotonicity of κ and Assumption A1, κ(A) ≥ κ({(x0 , j0 )}) = κx0 ,j0 ≥ ε + δ. Applying (9) with |A| = ρNF , V ⋆ = R⋆ + ρNF ℓmin + κ(A) ≥ R⋆ + ρNF ℓmin + ε + δ, F F contradicting the upper bound. Hence A ⊆ F ⋆ , and if ρ = ρ⋆ , then |A| = ρNF = ρ⋆ NF = |F ⋆ | forces A = F ⋆ . B.2

Proof of Lemma 4.1 (Formal Statement)

Lemma B.1 (Formal Statement). Assume ρNF is an integer and |ℓf (xt | x<t ; θ)| ≤ M for all θ, x, t and some finite M > 0. There exists λρ < ∞ such that, for any λρ > λρ , there exists λH (λρ ) < ∞ such that, for any λH > λH (λρ ), every global minimizer (θ † , z† ) of min

θ∈Θ, z∈[0,1]NF

satisfies z† ∈ {0, 1}NF with

e z) L(θ,

† x,t zx,t = ρNF , and is itself a global minimizer of (4).

Proof. Let Vc⋆ := inf θ, z∈Z L(θ, z). We argue in two steps: λH forces z to be binary, and λρ then forces the budget to be tight. ′ Step 1 (binarity). Let (θ ′ , z′ ) minimize Le over Θ × [0, 1]NF , and suppose some coordinate zy,k ∈ ′ (0, 1). Fix all other coordinates and round zy,k to its closest endpoint b ∈ {0, 1}, denoting the result ′ ′ by z′′ . Write η := min(zy,k , 1 − zy,k ) ∈ (0, 21 ].

The smooth part L + λρ (·)2 is Lipschitz in zy,k with constant at most M + 2λρ /NF , so the absolute change in the smooth part under rounding is at most (M + 2λρ /NF )η. The entropy contribution ′ satisfies H(zy,k ) ≥ c0 η for c0 := log 2, and it becomes zero after rounding. Thus, if λH > λH (λρ ) := c10 M + 2λρ /NF , e ′ , z′′ ) < L(θ e ′ , z′ ), contradicting optimality. Hence z′ ∈ {0, 1}NF . then L(θ P ′ Step 2 (budget). With z′ ∈ {0, 1}NF , the entropy term vanishes. Suppose x,t zx,t ̸= ρNF ; since P ′ both quantities are integers, | zx,t − ρNF | ≥ 1, so the budget term is at least λρ /NF2 . For any feasible (θ ⋆ , z⋆ ) ∈ Θ × Z, both regularizer terms vanish at (θ ⋆ , z⋆ ), so λ

e ′ , z′ ) ≤ L(θ e ⋆ , z⋆ ) = L(θ ⋆ , z⋆ ). L(θ ′ , z′ ) + Nρ2 ≤ L(θ F

Taking the infimum over (θ ⋆ , z⋆ ) ∈ Θ × Z yields L(θ ′ , z′ ) + Nρ2 ≤ Vc⋆ . Since R(θ) ≥ 0 and F

ℓF ≥ −M , L(θ, z) ≥ −M NF on Θ × [0, 1]NF . Setting λρ := NF2 (Vc⋆ P + M NF ), any λρ > λρ ′ would imply L(θ ′ , z′ ) < −M NF , contradicting this lower bound. Hence zx,t = ρNF . e ′ , z′ ) = L(θ ′ , z′ ), and Step 3 (optimality for the constrained problem). For (θ ′ , z′ ) ∈ Θ × Z, L(θ the same holds for any feasible competitor. Hence L(θ ′ , z′ ) ≤ L(θ ⋆ , z⋆ ) for all (θ ⋆ , z⋆ ) ∈ Θ × Z, i.e. (θ ′ , z′ ) is a global minimizer of (4).

Unlearning Metrics

Evaluating an unlearned model requires measuring three orthogonal properties simultaneously: (i) how thoroughly targeted knowledge has been removed, (ii) how robust that removal is under mild perturbations of the prompt, and (iii) how much of the model’s retain-set and general utility survives the intervention. No single score captures all three, so we report a panel of complementary metrics organized by purpose. Sections C.1 and C.2 describe the two primary evaluation protocols we inherit from prior work: a token-level extraction score probing verbatim memorization, and a judge-based family that stress-tests forgetting against paraphrase and in-context-relearning attacks. Section C.3 describes auxiliary utility probes that monitor general post-unlearning usability. For the RWKU benchmark we additionally employ the native ROUGE-L-based evaluation panel of Jin et al. [2024], described in Section C.4, as a cheap tuning surrogate during our Stage-1 hyperparameter search. Finally, Section C.5 introduces three baseline-relative summary scores that normalize the preceding metrics against the original checkpoint so that runs are directly comparable across models and forget sets. These are the numbers we use to rank methods at a glance. C.1

Token-level Extraction Metrics

Extraction Strength (ES). Following Wang et al. [2025b] (and the TOFU evaluation of Maini et al. [2024]), we measure how strongly a response is parameterized in the model’s weights using the Extraction Strength. For a question–answer pair (q, a) with answer tokens a = (a1 , . . . , aT ), let M (·) denote greedy decoding. Define the shortest extractable prefix k ⋆ (q, a) = min k ∈ {0, 1, . . . , T } : M q ⊕ a1:k = ak+1:T . (10) Whenever no strict prefix k < T reconstructs the suffix, the minimum collapses to k ⋆ (q, a) = T , corresponding to the worst case. The per-sample extraction strength is then k ⋆ (q, a) ES(q, a) = 1 − , ES(q, a) ∈ [0, 1]. (11) T ES = 1 indicates the answer is reproduced verbatim from the question alone (the model has fully memorized the response); ES = 0 indicates even the full ground-truth answer prefix fails to elicit the remaining tokens. We report averages over the forget and retain splits, 1 X 1 X ES(q, a), ESR = ES(q, a), (12) ESF = |F| |R| (q,a)∈F

(q,a)∈R

and summarize the unlearning/retention trade-off with ES∆ = ESR − ESF .

(13)

Motivation. Unlike likelihood-based scores, ES probes the actual generative behavior of the unlearned model, which is what an adversary would exploit. A method that merely suppresses the probability of the forget answer without disrupting its greedy decoding path is credited by perplexitystyle metrics but penalized by ESF . ES∆ provides a single scalar capturing the gap between targeted removal and preserved recall, which we use as the objective during hyperparameter search. Shortcomings. ES is defined through an exact suffix match, which makes it brittle to trivial deviations. Consider a generated response that agrees with a = (a1 , . . . , aT ) on every token except the last: M (q) = (a1 , . . . , aT −1 , x) with x ̸= aT . Under greedy decoding the same mismatch persists for every teacher-forced prefix; for any k < T , M (q ⊕ a1:k ) still deterministically produces (ak+1 , . . . , aT −1 , x), so ak+1:T is never recovered and k ⋆ (q, a) = T , yielding ES(q, a) = 0. Semantically the model has reproduced almost the entire answer verbatim, yet the score is indistinguishable from a response that disagrees from the very first token. The same pathology afflicts any mismatch in a single late position (e.g. a swapped named entity near the end of the answer), or cases where the model emits a stylistic token (e.g. punctuation, casing, whitespace) that differs from the reference. ES is therefore best read as a conservative lower bound on memorization: a low ESF does not certify forgetting, which is why we pair it with the judge-based scores of Section C.2. C.2

Robustness-Aware Forget and Retain Quality

The remaining metrics in this section are adopted from the evaluation protocol of Singh et al. [2025], which replaces brittle ROUGE-style matching with an LLM-as-judge score J and stress-tests each 22

unlearned model with paraphrased and in-context-primed queries. Given a query q and reference answer a, the judge returns J (q, a, â ; p) ∈ [0, 1] grading a candidate response â; the scores below are averages or worst-cases of J under different query transformations. The symbol p denotes the fixed system prompt we supply to the judge LLM: it defines the grading rubric and the expected output format, and is held identical across all of JP , JICR , JW , and JAVG so that the resulting scores are commensurable. The exact rubric wording and format instructions we use, as well as the prompts used to generate paraphrases, are included in the supplementary material. P For any query q we generate NP paraphrases {q(i) }N i=1 that preserve the semantic content of q, and (i) denote the model’s greedy response to each by â = M (q(i) ). The four metrics below reuse this notation.

Paraphrase score (JP ). On the forget set we report the worst-case judge score across the NP paraphrases, JP (q, a) = max J q(i) , a, â(i) ; p , (q, a) ∈ F. (14) i∈[NP ]

A low JP certifies that no paraphrase of q recovered the forgotten content, i.e. forgetting generalizes beyond the exact training phrasing. We depart here from Singh et al. [2025], who define JP as the mean over paraphrases and do not use it further in their evaluation protocol. Defining it as the worst-case instead makes it directly useful as a building block of our final forget-set score. In-context-relearning score (JICR ). To probe whether forgotten information resurfaces when the model is primed with related content, for each paraphrase q(i) we independently sample a small random subset d(i) ⊂ R of retain-set Q/A pairs and prepend them as in-context demonstrations. Let ã(i) = M (d(i) ⊕ q(i) ) denote the resulting generation. The ICR score is the worst case across paraphrases, JICR (q, a) = max J q(i) , a, ã(i) ; p . (15) i∈[NP ]

Structurally, JICR mirrors JP , with both taking a sample-wise max over the same paraphrase set (i) P {q(i) }N i=1 . The two differ only in the response being scored; JICR uses the primed generation ã , (i) whereas JP uses the zero-shot response â . A genuinely forgotten fact should remain unrecoverable even after priming, so JICR captures residual memorization that plain paraphrase attacks alone would miss. Worst-case forget quality (JW ). Since JP and JICR are already sample-wise worst cases, the primary robustness score reduces to their maximum, JW (q, a) = max JP (q, a), JICR (q, a) . (16) This is the main metric we track on F: a low JW certifies that no light-weight attack among those considered elicited the forgotten content. Average retain quality (JAVG ). On the retain set, worst-case evaluation is overly pessimistic because paraphrases can legitimately push the model toward equally valid rephrasings of the answer. We therefore report the mean judge score across the NP paraphrases, JAVG (q, a) =

NP 1 X J q(i) , a, â(i) ; p , NP i=1

(q, a) ∈ R.

(17)

Note that the functional form matches JP ; the distinction is the split on which the score is aggregated. A high JAVG indicates that the model still reliably answers questions it was meant to keep. C.3

Utility Metrics

Beyond the targeted forget/retain splits we monitor three signals of general post-unlearning usability, again following Singh et al. [2025]. We treat these signals as utility-preservation probes rather than optimization targets: the relevant failure mode is degradation relative to the original checkpoint, namely a drop in MMLU, a drop in Rep., or WR ≪ 50.0. Conversely, parity or occasional improvements are not penalized, since they may reflect benign side effects of the unlearning procedure or measurement noise rather than harmful changes. 23

MMLU. We evaluate multiple-choice accuracy on MMLU [Hendrycks et al., 2021] using argmax selection over the four options rather than open-ended generation. MMLU tracks the broad world knowledge of the model, which collateral forgetting can silently erode. We compute MMLU accuracy using the lm-evaluation-harness of Gao et al. [2024]. We therefore interpret MMLU asymmetrically: scores at or above the original checkpoint are taken to preserve this axis of utility. Repetitiveness (Rep.). Aggressive unlearning loss functions often cause the model to collapse into degenerate, low-entropy outputs. Following the Fluency metric of Jin et al. [2024], we measure repetitiveness from the bi- and tri-gram frequencies of generations produced on the AlpacaEval instruction set. Lower values indicate more frequently repeated n-grams, and thus flag pathological generation behavior that task metrics alone would not reveal. We likewise interpret Rep. asymmetrically, flagging only meaningful drops relative to the original checkpoint as degradation. Win Rate (WR). Repetitiveness alone does not capture whether responses are actually useful, so we additionally compare the instruction-following quality of the unlearned and original models using an LLM-as-judge in the style of AlpacaEval. For each prompt the judge is shown the question together with both responses in a single call and is asked to rate each on a 1–10 scale along with a short justification; wins, losses, and ties are then determined by comparing the two scores. For every prompt we record whether the unlearned model received a higher score (UWins ), lower score (ULosses ), or a tie (UTies ) relative to the original model, and compute WR = 100 ·

UWins + 0.5 UTies . UWins + ULosses + UTies

(18)

We note that the judge is not informed which response originates from the unlearned model and which from the original: the prompt labels them only as “Assistant 1” and “Assistant 2”, and nowhere in the system prompt, user prompt, or scoring criteria do we reference unlearning, forgetting, or model identity. Consequently the judge has no semantic cue that could bias it toward either model. The exact judge prompt used for this evaluation is included in the supplementary material. By construction, the original model has WR = 0.5 against itself. WR ≪ 0.5 is the clear failure mode: it indicates that unlearning has degraded general response quality relative to the original checkpoint. WR ≈ 0.5 is the expected and desired outcome, signaling that quality has been preserved. We depart here from Singh et al. [2025], who argue that WR should be as close as possible to 0.5 and treat deviations above 0.5 as undesirable. In our setting, we do not penalize WR > 0.5: such values may reflect genuine quality improvements, regularization side-effects of the unlearning objective, or judge noise, none of which constitutes the utility-degradation failure mode we aim to detect. C.4

Native RWKU Benchmark Metrics

Alongside the benchmark, Jin et al. [2024] ship an evaluation panel that we use for two purposes in this paper: as the tuning surrogate for the RWKU Stage-1 search (Section D.2), and as the basis of the detailed per-probe RWKU tables reported later in the appendix. Responses are not collected on a single probe set but across three complementary probe types, applied both to the forget targets and to a separately-curated set of adjacent facts that must be preserved. The first probe type, FB, consists of fill-in-the-blank cloze items that test whether the target knowledge can still be elicited from a partial prompt. The second, QA, is open-form question answering, where the model is asked directly about the target fact. The third, AA, bundles a family of adversarial reformulations of the same queries (prefix injection, role-playing, reverse querying, affirmative suffixes, cross-lingual variants, and several other jailbreak-style attacks) and reports the worst-case response across them. Each probe’s response is scored by the ROUGE-L recall against the reference answer, and the aggregate All is the mean over the probe types applied to that split. Probes are applied to two disjoint splits of the benchmark. The Forget Set (F) contains probes directed at the targeted knowledge across all three probe types FB, QA, and AA; lower ROUGE-L (↓) indicates that the model no longer reproduces the forgotten content. The Neighbor Set (N ) contains probes directed at related but non-targeted knowledge that the benchmark designates as must-preserve; following the benchmark, only the FB and QA probe types are reported on the Neighbor Set, since adversarial attacks are a forget-side stress test rather than a utility measurement. Higher ROUGE-L (↑) indicates that retain-adjacent facts survive the intervention. The Neighbor Set therefore plays on RWKU the role that the retain split R plays on TOFU. 24

For hyperparameter tuning we summarize the two panels with a single scalar, N∆ = ROUGE-LN ,All − ROUGE-LF ,All ,

(19)

the signed gap between the two “All” aggregates. Higher N∆ means better forgetting without collateral damage to neighbouring knowledge, and we use it as the selection criterion at Stage 1 on RWKU. We emphasize that the ROUGE-L-based scores above are a weaker signal than the judge-based metrics JP , JICR , JW , JAVG of Section C.2: as discussed by Singh et al. [2025], ROUGE penalizes semantically correct responses that are phrased differently from the reference and conversely credits generic or repetitive outputs that happen to share surface n-grams, so it systematically mislabels unlearning success and failure and can reorder methods with respect to a judge. We therefore treat these native RWKU scores as a legacy reporting panel, useful as a cheap tuning surrogate and for comparability with prior work that reports them, but consider the judge-based metrics our primary evaluation on RWKU. C.5

Baseline-Relative Summary Metrics

The raw scores JW and JAVG are not directly comparable across models or forget splits, because the operating point of the original checkpoint varies: a final JW = 0.2 is excellent when the original model scored 0.9 on the same forget set, but mediocre when the original already sat at 0.4. We therefore summarize each run by three baseline-relative scores, reported as fractional drops with orig orig respect to the original checkpoint. Writing JW and JAVG for the original model’s scores on the same splits, and using [x]+ = max(0, x) throughout, we clip each drop below at zero so that accidental increases (typically judge noise) are not miscounted as progress. This normalization puts all three metrics on the same [0, 1] scale regardless of the original operating point, and for readability we report them as percentages in [0, 100] throughout the paper. Forget Quality (FQ). FQ measures the fractional reduction in the worst-case forget-set judge score attributable to unlearning, " # JW FQ = 1 − orig . (20) JW + orig It ranges from 0 (no forgetting achieved, JW ≥ JW ) up to its ideal value of 1, reached only by a method that drives the worst-case judge score on the forget set to zero.

Retain Degradation (RD). RD measures the collateral cost of unlearning as the fractional loss in the average retain-set judge score, # " JAVG . (21) RD = 1 − orig JAVG + The ideal value is RD = 0, indicating that the retain split is answered as well as before unlearning; RD = 1 would correspond to a complete collapse of retain-side quality. Unlearning Quality (UQ). side gain,

UQ combines the two by subtracting retain-side damage from forget-

UQ = [ FQ − RD ]+ . (22) It is the fraction of forget-quality improvement that was not paid for by retain-set degradation, and lives on the same [0, 1] scale with ideal value 1. A value of UQ = 0 flags methods whose retain damage cancels out any forgetting progress, regardless of how low JW is in isolation. Motivation. The triple captures the two competing goals of unlearning – removing targeted knowledge and preserving the rest – together with their net trade-off. FQ and RD make each axis legible on its own, while UQ collapses the trade-off into a single scalar that we use to rank methods in our main tables, since a method that pushes JW down by uniformly blunting the model should not be credited with genuine unlearning.

Experimental Details

This section details the setup used for the experiments reported in the main paper. We describe the benchmarks and base models (Section D.1), the hyperparameter-tuning protocol (Section D.2), the final unlearning runs and evaluation protocol (Section D.4), and the method-specific hyperparameter search ranges and selected configurations (Section D.3). D.1

Benchmarks and Base Models

We evaluate on two LLM-unlearning benchmarks. TOFU Maini et al. [2024]. A synthetic question–answering benchmark in which the base model has been fine-tuned on a corpus of fictitious author biographies. We use the three standard forget splits supplied with the benchmark, forget01, forget05, and forget10, covering progressively larger fractions of the fine-tuning corpus. We consider two base model sizes, Llama-3.2-1B-Instruct2 and Llama-3.1-8B-Instruct [Grattafiori et al., 2024], and use the already-finetuned TOFU checkpoints released by the open-unlearning project [Dorna et al., 2026],3 whose codebase4 our TOFU experiments extend in two ways: we add paraphrase-based evaluation, and integrate J ENS U N alongside our proposed methods. RWKU Jin et al. [2024]. A real-world knowledge-unlearning benchmark that targets knowledge about public figures present in a pretrained LLM. We evaluate on the 3.8B-parameter Phi-3-Mini-4k-Instruct [Abdin et al., 2024]. Following Singh et al. [2025], we perform 10target batch unlearning: rather than unlearning one target at a time, a single unlearning run jointly removes knowledge about ten public figures. We fix one specific choice of ten subjects and refer to it as our canonical ten-subject RWKU batch throughout the paper. Since RWKU does not explicitly provide retain data, during unlearning we use a disjoint set of ten other subjects as a surrogate retain set. We adapt the official RWKU codebase5 in two ways: we add retain-set support, which the upstream release lacks since it only supports forget-only methods, and integrate G RAD D IFF, S IM NPO, J ENS U N, WGA, S AT I MP, alongside our proposed methods. D.2

Hyperparameter Tuning Protocol

Each unlearning method exposes a set of loss-specific hyperparameters (learning rate, loss-weighting coefficients, optional auxiliary objectives, etc.). We tune these per method and per benchmark using Bayesian optimization via optuna [Akiba et al., 2019], maximizing a cheap-to-compute surrogate of unlearning quality rather than the paraphrase- and judge-based evaluation of Sections C.2 and C.3 that we actually report: running the full pipeline at every trial is infeasible at our compute and API budget, since the ablations, and a single final-evaluation pass already incurs roughly USD $180 in judge-LLM API costs. We therefore run the judge-based evaluation only on the final selected checkpoints, validating the surrogate after the fact. Throughout tuning, the optimizer, batch size, gradient-accumulation steps, precision, and learning-rate schedule are held fixed at the final-run values of Section D.4; only the method-specific hyperparameters listed in the per-stage descriptions below are searched. Tuning proceeds in two stages. Stage 1: coarse search. For each method we run an initial search over all method-specific hyperparameters on the largest forget split of the benchmark (forget10 for TOFU, our canonical ten-subject batch for RWKU). On TOFU this stage runs on the small base model, Llama-3.2-1B-Instruct, for 50 trials at the full training budget of Section D.4, with the first trial seeded at the hyperparameters recommended by each method’s original paper; trials are selected by ES∆ = ESR − ESF (Section C.1). On RWKU this stage runs on Phi-3-Mini-4k-Instruct for 20 trials of a single epoch at a fixed learning rate, optimizing only the non-LR hyperparameters and selecting by the ROUGE-L-based N∆ surrogate defined in Section C.4. The first trial is seeded with each method’s final TOFU 8B configuration—the one used for the corresponding final unlearning runs in Section D.4—under the 2 https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct 3 https://huggingface.co/collections/open-unlearning/tofu-new-models. 4 https://github.com/locuslab/open-unlearning 5 https://github.com/jinzhuoran/RWKU

R-LF ↓ R-LN ↑ N∆ ↑

lr −5

DPO

1.18×10 2.70×10−5 1.98×10−5

0.499 0.435 0.491

0.590 0.450 0.501

NPO

2.60×10−5 1.18×10−5 1.98×10−5

0.236 0.325 0.257

0.541 0.597 0.509

0.305 0.272 0.252

2.08×10−5 S IM NPO 1.18×10−5 2.70×10−5

0.218 0.254 0.218

0.561 0.594 0.541

0.343 0.341 0.323

1.54×10−6 3.00×10−6 3.98×10−6

0.062 0.040 0.038

0.434 0.408 0.377

0.372 0.368 0.339

J ENS U N

R-LF ↓ R-LN ↑ N∆ ↑

lr −6

0.090 1.18×10 0.015 G RAD D IFF 1.90×10−6 0.010 2.70×10−6

0.053 0.045 0.033

0.445 0.399 0.346

0.392 0.354 0.313

WGA

1.54×10−6 1.57×10−6 3.00×10−6

0.061 0.063 0.034

0.515 0.502 0.427

0.454 0.438 0.393

S AT I MP

1.98×10−6 4.48×10−6 3.00×10−6

0.132 0.069 0.112

0.617 0.533 0.566

0.485 0.465 0.454

ATWU

1.83×10−6 1.74×10−6 1.37×10−6

0.050 0.072 0.100

0.562 0.584 0.610

0.512 0.511 0.510

Table 5: Top-3 Stage 2 learning-rate trials per method on RWKU, ranked by N∆ . Green shading marks the trial used for the final unlearning run: the top-N∆ trial throughout, except for S AT I MP, whose top trial was discarded for inadequate forgetting in favor of the second-best.

assumption that well-tuned settings transfer across benchmarks. For most methods, we keep the learning rate unchanged in this pass. For G RAD D IFF and J ENS U N, however, we reduce it by a factor of 10; the same reduction is also applied to WGA and S AT I MP as variants of G RAD D IFF. This is informed by Singh et al. [2025], who report that these methods require markedly smaller learning rates on RWKU than on a QA dataset to produce stable unlearning, so anchoring at the TOFU 8B value directly would overshoot. Finally, the motivation for holding the learning rate fixed at all in this RWKU pass is twofold: a single-epoch budget gives an unreliable signal about the correct learning rate, so we defer LR tuning to Stage 2 where the full epoch budget applies, and reducing the number of jointly tuned parameters yields tighter estimates of the remaining method-specific hyperparameters within the same search budget.

Stage 2: learning-rate transfer. In both benchmarks we transfer the Stage 1 configuration to the evaluation setting and tune only the learning rate. We run 5 trials per method, sweeping the learning rate on a log-scale grid and holding all other hyperparameters at their Stage 1 values. On TOFU the transfer is across model scale: the Stage 1 configuration on the 1B model is ported to Llama-3.1-8B-Instruct and the learning rate is retuned, since re-running the full search at 8B would be prohibitive. On RWKU the transfer is across training budget: the Stage 1 configuration (tuned at one epoch) is run at the final-run epoch count so that the learning rate is selected against the full schedule, still on Phi-3-Mini-4k-Instruct. Selection is automatic on both benchmarks; we use ES∆ on TOFU and N∆ on RWKU. The only exception is S AT I MP on RWKU, where the top-N∆ trial had inadequate forget performance and we therefore selected the second-best trial (see Table 5).

D.3

Hyperparameter Search Ranges and Results

This subsection specifies the concrete search ranges used in the two-stage tuning procedure of Section D.2 and reports the selected configurations for TOFU (Table 6) and RWKU (Table 7). Unless stated otherwise, all ranges are searched on a logarithmic scale. The coefficients α and γ denote the retain-loss and forget-loss weights, respectively, and are shared across methods. For most baselines, the method-specific grids follow a slightly extended version of those used by Yang et al. [2025]. For our score-based methods, we fix λent = λw = 1 and set λpop = 15 throughout. An ablation on the different regularizers is presented in Table 13 and discussed shortly after. Throughout this subsection, hyperparameter values returned by optuna are rounded to two significant digits and performance scores to three decimal places for readability. The exact hyperparameter values used in the final runs reported in Tables 6 and 7 are provided in the supplementary material. 27

Method

Shared α

Method-Specific β / β1 / τ / c

δ / β2 / slr / ℓ

Learning Rate 1B

8B −5

G RAD D IFF DPO NPO S IM NPO J ENS U N RMU ETW WGA S AT I MP

0.80 0.15 4.10 1.28 0.79 1.27 1.31 0.79 0.49

0.12 3.80 0.12 1.49 0.82 0.66 0.033 1.16 0.87

— 0.21 0.10 2.82 — 4.39 0.16 2.14 1.43

— — — 0.03 — 4 — — 0.17

1.90×10 2.27×10−5 2.60×10−5 2.08×10−5 3.98×10−5 — 2.58×10−5 1.57×10−5 1.98×10−5

1.90×10−5 3.12×10−5 2.60×10−5 2.86×10−5 2.66×10−5 2.21×10−4 — 1.57×10−5 1.98×10−5

ATWU

0.50

3.00

7.00

0.050

2.00×10−5

1.50×10−5

Table 6: Selected hyperparameters on TOFU. 1B and 8B are the learning rates used on Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct, respectively. The method-specific columns are overloaded across methods: β for DPO/NPO/S IM NPO/WGA, β1 for S AT I MP, τ (temperature) for ETW, c (steering coefficient) for RMU; δ for S IM NPO, β2 for S AT I MP, slr for ATWU, ℓ (target layer) for RMU. ETW was evaluated only on 1B and RMU only on 8B; the non-evaluated column is marked —.

Stage 1: coarse search. For both datasets, we tune all non-fixed method-specific hyperparameters. All methods search over α, γ ∈ [0.1, 5], and for TOFU, we also search lr ∈ [1 × 10−6 , 5 × 10−5 ].6 The per-method parameters tuned in this stage are as follows: for G RAD D IFF, we try both NLL and KL as the retain loss;7 for DPO and NPO, we search the inverse temperature β in the ranges [0.2, 0.5] and [0.05, 0.2], respectively; for S IM NPO, we search the temperature β and reward margin δ in the ranges [2.0, 3.0] and [0.0, 2.0], respectively; for WGA, we tune the weight exponent β in the range [0.1, 5]; for S AT I MP, we search the saturation and importance exponents β1 and β2 in the ranges [1.0, 10.0] and [0.1, 5], respectively; for ETW, we search the temperature τ ∈ [0.1, 4.0]; for RMU, we search the steering coefficient c ∈ [0.5, 10.0], and the unlearning layer ℓ ∈ {4, . . . , 12} (with the trainable parameters being the mlp.down_proj weights of layers ℓ − 2, ℓ − 1, ℓ, following the WMDP RMU recipe); and for ATWU, we tune β ∈ [1.0, 10.0] and the scorer learning rate slr ∈ [0.01, 0.5]. Stage 2: learning-rate transfer. On TOFU, the per-method range is anchored at the best Stage 1 learning rate: writing lr⋆m for the learning rate selected for method m through the coarse search, we consider the range lr ∈ [lr⋆m /5, 5 · lr⋆m ], with the first trial seeded at lr⋆m itself. For each method, the selected lr⋆m are: 2.27 × 10−5 for DPO; 2.60 × 10−5 for NPO; 2.08 × 10−5 for S IM NPO; 3.98 × 10−5 for J ENS U N; 1.90 × 10−5 for G RAD D IFF; 1.57 × 10−5 for WGA; 1.98 × 10−5 for S AT I MP; 2.58 × 10−5 for ETW; 4.88 × 10−5 for RMU; and 2.00 × 10−5 for ATWU. On RWKU no analogous Stage 1 optimum is available, since the learning rate was held fixed throughout Stage 1; we instead choose method-specific log-scale ranges based on preliminary runs at the full Stage 2 training budget. The resulting ranges per method are: for DPO, NPO, and S IM NPO, [1 × 10−5 , 5 × 10−5 ]; for G RAD D IFF, J ENS U N, WGA, S AT I MP, and ATWU, [1 × 10−6 , 1 × 10−5 ]. D.4

Final Unlearning Runs and Evaluation

Once a hyperparameter configuration is fixed for each (benchmark, base model, method) triple, we run the final unlearning procedure and evaluate the resulting checkpoints. On TOFU we run the procedure on each of the three forget splits forget01, forget05, and forget10; on RWKU we run it on our canonical ten-subject batch. 6 For J ENS U N on TOFU, preliminary runs showed consistently weak forgetting at the smallest learning rates. We therefore narrowed its learning-rate search to [5 × 10−6 , 5 × 10−5 ], allowing a finer search in the more promising region. 7We only search over the retain loss on TOFU. On RWKU we fix it to NLL, since the TOFU Stage 1 search selected NLL consistently over KL, and we prefer to reduce tuning over parameters for which the choice already appears settled.

Shared

Method lr

Method-Specific

β / β1

δ / β2 / slr

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP

−6

1.18×10 1.18×10−5 2.60×10−5 2.08×10−5 1.54×10−6 1.54×10−6 4.48×10−6

0.80 2.96 0.63 0.46 1.52 0.22 0.23

0.12 0.13 4.76 2.43 0.39 2.79 2.60

— 0.35 0.07 2.59 — 0.40 2.64

— — — 1.50 — — 0.20

ATWU

1.83×10−6

0.54

2.68

2.25

0.020

Table 7: Selected hyperparameters on the canonical ten-subject RWKU batch. The learning rate is the one used on Phi-3-Mini-4k-Instruct.

On TOFU, each run uses a single NVIDIA A100-SXM4-80GB GPU, trains for 10 epochs at batch size 8 with 4 gradient-accumulation steps (effective batch size 32), and follows the open-unlearning repository defaults [Dorna et al., 2026]: the paged_adamw_32bit optimizer with (β1 , β2 ) = (0.9, 0.999), weight decay 0.01, a one-epoch linear warmup followed by the default linear decay, gradient clipping at 1.0, and bfloat16 mixed precision. On RWKU, each run uses two NVIDIA A100-SXM4-80GB GPUs with model parallelism only—the base model is sharded across the two GPUs while the batch is not replicated, so the per-step batch size is 8 and, with 4 gradientaccumulation steps, the effective batch size is 32—trains for 5 epochs, and follows the official RWKU repository defaults: AdamW with (β1 , β2 ) = (0.9, 0.999) and weight decay 0.01, a cosine learning-rate schedule with 20 warmup steps, gradient clipping at 1.0, and fp16 precision. For the final reported numbers, we evaluate each unlearned checkpoint on the full metric panel of Section C. In the main paper, we assess unlearning performance and retain-set preservation primarily via the baseline-relative summary metrics FQ, RD, and UQ introduced in Section C.5. These are our preferred headline metrics because they normalize forget-side gains and retain-side damage relative to the original checkpoint, making runs more directly comparable across models and forget sets. In particular, UQ serves as our primary one-number summary, since it rewards forgetting only to the extent that it is not purchased by collateral degradation on the retain set.

Judge Prediction

To monitor broader post-unlearning behavior, we addition300 ally report the utility metrics of Section C.3: MMLU, repetitiveness, and WR. We interpret these as utility250 95 8 yes preservation probes rather than optimization targets. The goal of unlearning is not to improve general utility, but to 200 preserve as much of the original model’s utility as possi150 ble while removing the targeted knowledge. Accordingly, the relevant failure mode is utility degradation: drops in 100 7 330 no MMLU, drops in Rep. indicating more degenerate generation, and WR < 0.5 against the original checkpoint. 50 Occasional improvements relative to the original model yes no are not harmful and are therefore not penalized; we inHuman Prediction terpret them as benign side effects or measurement noise rather than as objectives of the unlearning procedure. Figure 7: Agreement between the For completeness, the appendix also reports the underlying GPT-5.4 mini judge and human annoforget/retain metrics from Sections C.1 and C.2, namely tators on a 440-row sample of TOFU the extraction-strength scores ESF , ESR , ES∆ and the forget01. The judge matches the hujudge-based scores JP , JICR , JW , and JAVG . Judge- man label on ∼96% of calls; errors split based evaluation uses the fixed prompts provided in the as 8 false positives and 7 false negatives. supplementary material and is performed with OpenAI’s GPT-5.4 mini judge model, instantiated as gpt-5.4-mini [OpenAI, 2026]. We validate this judge against human annotations in Fig. 7. MMLU is computed with the lm-evaluation-harness [Gao et al., 2024]. Main-paper tables report FQ, RD, UQ, MMLU, Rep., and WR, while the full metric panel is deferred to this section. 29

forget01 - Llama-3.2-1B-Instruct Method

Surrogate

Paraphrase

ESF ↓ ESR ↑

ES∆ ↑

Relative

Utility

JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.743

0.737

−0.007

90.0

41.5

0.0

45.1

559

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP

0.220 0.134 0.085 0.087 0.295 0.093 0.084

0.625 0.566 0.539 0.574 0.600 0.675 0.619

0.405 0.432 0.454 0.487 0.304 0.582 0.536

77.5 17.5 45.0 52.5 60.0 80.0 55.0

35.9 38.1 37.7 38.3 29.2 34.6 38.3

13.9 80.6 50.0 41.7 33.3 11.1 38.9

13.6 8.3 9.3 7.9 29.8 16.7 7.8

0.3 72.2 40.7 33.7 3.5 0.0 31.1

45.0 45.1 45.3 45.4 44.9 45.2 45.1

563 552 562 565 555 560 560

51.5 56.5 59.0 54.0 55.0 58.5 56.0

ATWU

0.046

0.604

0.558

52.5

36.5

41.7

12.1

29.5

45.1

557

52.0

forget05 - Llama-3.2-1B-Instruct Method

Surrogate

Paraphrase

Relative

Utility

ESF ↓ ESR ↑

ES∆ ↑

JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.727

0.737

0.009

90.5

42.9

0.0

45.1

559

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP

0.178 0.205 0.100 0.121 0.047 0.076 0.061

0.620 0.624 0.644 0.689 0.208 0.746 0.690

0.442 0.419 0.543 0.568 0.161 0.670 0.629

68.6 50.5 47.5 54.6 0.0 53.0 47.9

36.1 38.6 38.4 38.8 1.1 40.2 38.3

24.2 44.2 47.5 39.7 100.0 41.5 47.1

15.8 10.0 10.6 9.6 97.3 6.4 10.9

8.4 34.2 36.9 30.1 2.7 35.1 36.3

44.8 44.8 45.0 45.1 45.4 45.0 44.9

563 551 565 568 369 568 569

50.5 45.0 53.5 54.5 25.0 49.0 57.0

ATWU

0.041

0.696

0.655

37.5

37.1

58.6

13.6

45.0

45.1

574

60.5

forget10 - Llama-3.2-1B-Instruct Method

Surrogate

Paraphrase

Relative

Utility

ESF ↓ ESR ↑

ES∆ ↑

JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.706

0.737

0.030

93.1

43.0

0.0

45.1

559

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N ETW WGA S AT I MP

0.107 0.194 0.103 0.106 0.033 0.108 0.062 0.043

0.658 0.716 0.730 0.738 0.570 0.679 0.759 0.747

0.550 0.523 0.627 0.632 0.537 0.572 0.696 0.704

57.9 53.3 41.0 57.4 1.8 47.8 46.6 32.9

37.4 38.2 42.4 41.1 36.4 39.0 39.3 41.2

37.8 42.8 56.0 38.3 98.0 48.7 50.0 64.7

13.0 11.2 1.4 4.4 15.3 9.3 8.5 4.1

24.8 31.6 54.6 33.9 82.8 39.4 41.5 60.7

45.2 44.9 44.9 45.1 45.1 45.5 45.0 44.8

563 560 563 566 343 561 568 570

55.5 59.0 55.0 57.0 23.5 54.5 56.0 55.0

ATWU

0.035

0.753

0.717

14.5

40.3

84.4

6.3

78.1

45.0

575

51.5

Table 8: TOFU full-panel results for Llama-3.2-1B-Instruct across the three forget splits. In each performance column, best is in bold and second-best underlined. ETW was evaluated only on forget10.

Judge reproducibility. A practical caveat is that OpenAI’s hosted inference is not bit-exact reproducible: calling gpt-5.4-mini on the same input does not in general return the same completion or the same per-token log-probabilities, even at temperature=0 and with a fixed seed.8 This is a known property of the served models rather than of the API surface, and the standard mitigation in the community is to cache judged responses against a hash of the input rather than to expect run-to-run determinism. Because rejudging every checkpoint on every paraphrase carries non-trivial API cost (cf. the $180 figure quoted earlier in this section), we run the judge once per (checkpoint, query, paraphrase) tuple and report the resulting scores; the 96% judge–human agreement reported in Fig. 7 is computed on the same single-pass scores. In spot rejudging we observed only minor drift in 8 See e.g. the OpenAI developer-community threads at:

https://community.openai.com/t/reproducible-outputs-in-assistants-api/1167782 and https://community.openai.com/t/logprobs-inconsistent-between-runs-for-4o/935082/2

forget01 - Llama-3.1-8B-Instruct Method

Surrogate

Paraphrase

Relative

Utility

ESF ↓ ESR ↑ ES∆ ↑ JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.977

0.992

0.015

100.0

68.2

0.0

66.6

519

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP

0.203 0.219 0.093 0.080 0.887 0.080 0.083

0.724 0.805 0.700 0.666 0.979 0.907 0.809

0.521 0.585 0.607 0.586 0.092 0.827 0.726

72.5 12.5 32.5 27.5 95.0 42.5 42.5

50.9 56.6 53.3 51.4 57.6 63.0 56.5

27.5 87.5 67.5 72.5 5.0 57.5 57.5

25.3 16.9 21.8 24.6 15.6 7.5 17.1

2.2 70.6 45.7 47.9 0.0 50.0 40.4

66.6 66.3 66.3 66.4 66.5 66.5 66.6

529 482 544 541 278 534 525

45.0 36.0 48.5 50.0 19.5 48.5 41.5

ATWU

0.065

0.885

0.820

45.0

64.8

55.0

5.0

50.0

66.7

558

57.0

forget05 - Llama-3.1-8B-Instruct Method

Surrogate

Paraphrase

Relative

Utility

ESF ↓ ESR ↑ ES∆ ↑ JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.972

0.992

0.020

99.0

68.6

0.0

66.6

546

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP

0.256 0.268 0.123 0.102 0.033 0.058 0.035

0.782 0.815 0.875 0.852 0.924 0.931 0.891

0.525 0.548 0.752 0.750 0.891 0.874 0.855

75.0 41.5 50.0 42.5 1.0 24.5 15.5

51.6 58.4 60.6 55.7 61.2 63.6 61.0

24.2 58.1 49.5 57.1 99.0 75.3 84.3

24.7 14.8 11.7 18.7 10.7 7.2 11.0

0.0 43.3 37.8 38.4 88.3 68.0 73.4

66.7 65.1 66.3 66.0 65.2 66.6 66.6

544 530 549 545 474 541 546

45.5 34.0 53.0 56.5 29.0 45.5 56.0

ATWU

0.033

0.914

0.882

11.7

61.7

88.2

10.0

78.2

66.8

541

48.5

forget10 - Llama-3.1-8B-Instruct Method

Surrogate

Paraphrase

Relative

Utility

ESF ↓ ESR ↑ ES∆ ↑ JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.979

0.992

0.013

99.5

67.8

0.0

66.6

546

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP RMU

0.096 0.314 0.129 0.077 0.033 0.046 0.035 0.033

0.838 0.860 0.927 0.905 0.942 0.970 0.952 0.989

0.742 0.547 0.798 0.828 0.910 0.924 0.917 0.956

60.0 40.2 51.7 30.5 2.0 33.0 21.2 12.8

56.1 57.6 64.4 63.4 61.3 64.6 65.8 66.9

39.7 59.5 48.0 69.3 98.0 66.8 78.6 87.2

17.3 15.0 5.0 6.5 9.6 4.8 3.0 1.3

22.4 44.5 43.0 62.8 88.3 62.1 75.6 85.9

66.1 64.3 66.2 65.8 65.3 66.0 65.9 65.3

545 533 551 541 236 550 549 540

48.0 38.5 48.5 50.5 11.0 60.0 53.5 50.5

ATWU

0.033

0.973

0.940

4.8

65.4

95.2

3.5

91.7

66.5

558

58.0

Table 9: TOFU full-panel results for Llama-3.1-8B-Instruct across the three forget splits. In each performance column, best is in bold and second-best underlined. RMU was evaluated only on forget10.

J -based scores across calls—small relative to the inter-method gaps in our headline tables—which we take as evidence that the rankings we report are robust to judge non-determinism. The importance of proper tuning. Reported baseline performance in the LLM-unlearning literature is unusually sensitive to per-method hyperparameter choices, and numbers across papers are not always directly comparable. RMU [Li et al., 2024] is the clearest example: Wang et al. [2025b] report it underperforming several preference-based baselines on TOFU, yet under our uniform two-stage tuning protocol it reaches UQ = 85.9 on forget10 with Llama-3.1-8B-Instruct, second only to ATWU (UQ = 91.7) and well ahead of every preference-based method (Table 1). The driver is RMU’s idiosyncratic configuration space, where a mis-specified unlearning layer ℓ alone can significantly swing UQ; our Bayesian search lands on ℓ = 4, c = 4.39 (Table 6), which differs substantially from the configurations typically reported for WMDP-style settings. The relevant tuning gap is not the loss coefficients α, γ alone: Wang et al. [2025b] and Yang et al. [2025] do tune those in narrow 31

Method

Surrogate

Paraphrase

Relative

Utility

R-LF ↓ R-LN ↑ N∆ ↑ JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.683

0.759

0.077

84.8

65.8

0.0

70.1

529

50.0

G RAD D IFF DPO NPO S IM NPO J ENS U N WGA S AT I MP

0.053 0.499 0.236 0.218 0.062 0.061 0.069

0.445 0.590 0.541 0.561 0.434 0.515 0.533

0.392 0.090 0.305 0.343 0.372 0.454 0.464

15.3 74.4 44.4 40.9 12.4 18.7 19.3

41.6 54.6 40.1 46.2 42.0 48.6 46.5

81.9 12.3 47.6 51.8 85.4 78.0 77.3

36.7 17.0 39.1 29.7 36.1 26.1 29.3

45.2 0.0 8.5 22.1 49.3 51.9 48.0

69.2 69.7 68.1 68.8 70.2 69.4 69.8

514 523 543 529 523 518 518

43.0 43.0 33.5 45.5 46.5 41.0 40.0

ATWU

0.050

0.562

0.512

15.8

50.9

81.4

22.7

58.7

70.3

519

46.0

Table 10: RWKU canonical ten-subject batch, Phi-3-Mini-4k-Instruct. In each performance column, best is in bold and second-best underlined.

paper-seeded ranges, but the learning rate (which is not equivalent to a coefficient rescaling under AdamW with gradient clipping and a non-constant LR schedule) and, in RMU’s case, the steering coefficient and target layer, which are not loss coefficients at all. By contrast, Singh et al. [2025] and Dorna et al. [2026] grid-search the learning rate per method and report RMU as competitive, consistent with our finding. We view this not as a critique of any specific prior comparison but as empirical evidence that fair ranking across this literature requires re-tuning each method for the target setting, particularly when methods exposing many interacting knobs (RMU, S IM NPO, S AT I MP) are compared against methods with very few (G RAD D IFF, J ENS U N). We therefore tune every baseline uniformly via Bayesian optimization, seeded with each paper’s recommended defaults; the full protocol and selected configurations are reported in Sections D.2 and D.3. TOFU full-panel results. Tables 8 and 9 report the full TOFU metric panels across all forget splits and both base models. Across settings, ATWU is consistently among the best or second-best methods once utility-collapsing runs are excluded. This distinction is important: aggressive methods such as J ENS U N can obtain very strong forget-side scores, but often do so by severely damaging generation quality, as reflected by large drops in Rep. and WR. We therefore interpret the tables through the joint lens of forgetting, retain preservation, and general utility. The main limitation appears on the smaller forget splits, especially forget01. In these settings, the scorer receives relatively few forget examples and fewer effective scorer updates, making it harder to learn a stable token-level relevance signal. As the forget set becomes larger, the learned scorer has more opportunity to separate forget-specific tokens from structural context, and the advantage of ATWU becomes clearer. This trend is most visible on forget10, where ATWU achieves the strongest or near-strongest UQ while preserving MMLU, Rep., and WR close to the original checkpoint. These results suggest that the learned token-weighting mechanism is most beneficial when enough forget-side signal is available for the scorer to train reliably. RWKU full-panel results. Table 10 reports the corresponding full metric panel on the canonical ten-subject RWKU batch. This setting is challenging for a different reason than TOFU. In RWKU, the forget set consists of paragraphs about the target entities, whereas evaluation is performed through

Method

Surrogate

Paraphrase

Relative

Utility

R-LF ↓ R-LN ↑ N∆ ↑ JW ↓ JAVG ↑ FQ ↑ RD ↓ UQ ↑ ∆UQ ↑ MMLU Rep. WR

O RIGINAL

0.683

0.759

0.077

84.8

65.8

0.0

—

70.1

529

50.0

ATWUDPO ATWUNPO ATWUSimNPO

0.501 0.205 0.109

0.657 0.490 0.567

0.156 0.285 0.458

68.2 39.1 32.6

57.1 42.8 50.1

19.6 53.8 61.5

13.2 34.9 23.8

6.4 18.9 37.8

+6.4 +10.4 +15.7

69.5 67.7 69.9

512 545 531

36.0 35.5 43.5

ATWU

0.050

0.562

0.512

15.8

50.9

81.4

22.7

58.7

+6.8

70.3

519

46.0

Table 11: RWKU canonical ten-subject batch with ATWU instantiated using different forget losses, Phi-3-Mini-4k-Instruct. ∆UQ reports the gain over the corresponding unweighted objective in Table 10. In each performance column best is in bold and second-best underlined.

separate queries about those entities. Thus, the unlearning objective is not optimized on the same question–answer format used at evaluation time; the model must forget information expressed in paragraph form and generalize that forgetting to query-based probes. Despite this mismatch, ATWU achieves the highest UQ, improving over the strongest non-ATWU baseline while preserving MMLU and maintaining a non-collapsed win rate against the original model. The ROUGE-L surrogate metrics show a similar trend, with ATWU attaining the best N∆ , but the judge-based metrics provide the more important comparison: methods with strong forget-side scores can still suffer substantial retainside or utility degradation. Overall, the RWKU results indicate that learned token-level reweighting remains effective even when the forget data and evaluation queries differ in format. ATWU with different forget losses. Finally, Table 11 evaluates whether ATWU is tied to a particular forget loss. In the main ATWU variant, we use a GradDiff-style forget loss. Here, we compare with alternative forget loss choices derived from DPO, NPO, and S IM NPO, integrating the learned scorer into the forget-side term of each objective as described in Section E.2. This keeps the central mechanism of ATWU fixed – learning token-level forget-relevance scores and using them to weight the forget update – while only changing the underlying forget loss. Across all forget losses, ATWU improves UQ relative to the corresponding unweighted objective. The strongest results are obtained with the GradDiff-style instantiation, but the gains for DPO, NPO, and S IM NPO show that the learned scorer is not specific to one loss form. Rather, ATWU provides a general way to turn sequence-level forget objectives into selective token-weighted objectives.

Additional Experiments

This section provides additional experimental details and ablations that support the main findings. We first ask whether a single linear function of the model’s hidden states can even, in principle, separate forget-relevant tokens from structural ones, with and without supervision. We then show how ATWU can be combined with alternative forget losses by inserting the learned scorer only into the forget-side token-level terms. Finally, we ablate the main scorer design choices, including its regularization, training state, and update frequency. Together, these experiments test whether the gains of ATWU come from a particular objective choice, or from the more general mechanism of learning where the forget update should be applied. E.1

Linear Separability of Informative Tokens

Before validating our parametric scorer in detail, we briefly ask whether a linear function of the model’s hidden states can even, in principle, separate forget-relevant tokens from structural ones, and whether such a scorer can be learned without supervision. The two experiments below answer affirmatively in both regimes. Supervised baseline. We train a single-layer linear scorer on the final-layer hidden states of Llama-3.1-8B-Instruct fine-tuned on TOFU using binary cross-entropy against the ground-truth token-level forget labels of Zhou et al. [2026], for one epoch on forget10. Figure 8 reports the per-sample AUROC distribution as training progresses, together with the final scorer’s output on a representative forget sample. After only ∼ 20% of one epoch the AUROC distribution is already concentrated at the high end, and the final scorer cleanly recovers the GT forget tokens on the qualitative example. A single linear direction on the model’s hidden states therefore suffices to separate forget-relevant tokens, at least when the labels are available.

50 ± 0

What does Hsiao Yun-Hwa identify as in terms of gender?

92 ± 8 92 ± 8

Hsiao Yun-Hwa is part of the LGBTQ + community .

93 ± 7 94 ± 7 95 ± 7

0 init

100 step 400

(a) Per-sample AUROC distribution of the BCE-trained scorer over training.

(b) Final scorer output on representative prompt.

Figure 8: Supervised baseline. A linear scorer trained with binary cross-entropy against the GT labels of Zhou et al. [2026] cleanly recovers the forget span after a fraction of an epoch. Bold tokens mark the ground-truth forget-relevant span. Unsupervised scorer with the unlearning objective. Ground-truth token labels of the kind released by Zhou et al. [2026] require manual or LLM-based annotation of the forget data, which is unrealistic for many unlearning requests, especially when the forget set contains private, copyrighted, or otherwise sensitive content. We therefore ask whether the retain-conflict objective used inside ATWU (Eq. 8) can recover a comparable linear scorer without labels. Figure 9 reports the analogous trajectories and qualitative output in two configurations: training the scorer against the original TOFU-fine-tuned target model that has memorized the forget content (top row), and against a retain model with the same architecture fine-tuned only on the retain split, with no exposure to the forget set (bottom row). On the target model, the unsupervised objective fails to localize the forget span; its 34

near-perfect predictions on memorized forget content provide too little gradient signal. On the retain model, by contrast, the same objective recovers the answer-bearing tokens (community, Hsiao, YunHwa, partially LGBTQ), comparable in shape to the supervised scorer. This observation underpins a central design choice in ATWU: jointly training the scorer with the unlearned model gradually exposes the scorer to representations closer to the retain regime, supplying the gradient signal needed to localize forget-specific tokens without any labels.

50 ± 0

What does Hsiao Yun-Hwa identify as in terms of gender?

37 ± 13 44 ± 20

Hsiao Yun-Hwa is part of the LGBTQ + community .

29 ± 12 41 ± 15 42 ± 18

100

init

step 400

(a) ATWU objective on the target (memorized) model.

50 ± 0

(b) Final output (target model).

What does Hsiao Yun-Hwa identify as in terms of gender?

64 ± 11 73 ± 13

Hsiao Yun-Hwa is part of the LGBTQ + community .

72 ± 12 69 ± 12 74 ± 11

100

init

step 400

(d) Final output (retain model).

Figure 9: Unsupervised scorer. Trained against the original target model (top), the ATWU objective fails to localize the GT forget span; trained against a retain model (bottom), the same objective recovers the answer-bearing tokens. Bold tokens mark the ground-truth forget-relevant span.

E.2

ATWU with different forget losses

ATWU is not tied to a single forget loss. Whenever an unlearning objective contains a forget-side term that decomposes autoregressively over tokens, we can replace that term by its scorer-weighted analogue while leaving the rest of the objective unchanged. This preserves the baseline loss structure but redirects the forget update toward tokens that the scorer identifies as forgetrelevant.

Method

Replacement

Scope

DPO NPO S IM NPO

r(s− ) 7→ rg (s− ) r(s) 7→ rg (s) ℓ̂(s) 7→ ℓ̂g (s)

dispreferred sequence only forget sequence length-normalized forget NLL

Table 12: Summary of how the learned ATWU scorer is integrated into each forget loss.

We use this recipe to integrate the DPO, NPO, and S IM NPO losses into ATWU, evaluated in Fig. 1b and Table 2. Retain-side terms, preferred responses, and frozen reference-model terms are never reweighted; only the original forget sequence receives scorer-weighted token mass. Table 12 gives the corresponding replacements, and the following paragraphs define them precisely. Notation and reweighted building blocks. For a sequence x ∈ V ⋆ and model parameters θ, define the unnormalized autoregressive negative log-likelihood and its scorer-reweighted counterpart as ℓ(θ ; x) = −

|x| X

ℓg (θ ; x) = −

log pθ (xt | x<t ),

t=1

|x| X

gt log pθ (xt | x<t ),

(23)

t=1

where gt = gw (hθ (xt )) is the scorer output at position t. Dataset-level losses are obtained by summing over sequences: X X L(θ ; D) = ℓ(θ ; x), Lg (θ ; D) = ℓg (θ ; x). x∈D

x∈D

For methods that compare a trainable unlearned model θun against a frozen reference model θref , we define the sequence-level log-ratio and its scorer-reweighted analogue as rθun (x) = −ℓ(θun ; x) + ℓ(θref ; x), where

rθgun (x) = −ws ℓg (θun ; x) + ℓ(θref ; x),

|x| wx = P|x|

(24)

t=1 gt

This normalization keeps P the total token mass of the trainable-model term equal to that of the original sequence loss, since t ws gt = |x|. The scorer therefore redistributes the contribution across token positions without changing the average scale of the trainable-model term. The reference term is left unchanged because the reference model is frozen and does not depend on the scorer. Each augmented objective below replaces only the forget-side building block of the original loss by its scorer-reweighted counterpart, such as ℓg , rg , or a scorer-modulated saturation term. DPO [Rafailov et al., 2023]. DPO for unlearning operates on a paired forget set A = − + − N {(x+ i , xi )}i=1 , where xi is the original forget sequence and xi is an alternative sequence that the model should prefer. The original loss compares the two sequences through a sigmoid of their log-ratio difference: X 2 LDPO (θun ; A) = − log σ β rθun (x+ ) − rθun (x− ) . (25) β + − (x ,x )∈A

The two sides of each pair play asymmetric roles. The dispreferred sequence x− contains the information to be forgotten, whereas the preferred sequence x+ is intended to remain ordinary, acceptable language. We therefore apply the scorer only to the dispreferred side: X 2 LgDPO (θun ; A) = − log σ β rθun (x+ ) − rθgun (x− ) . (26) β + − (x ,x )∈A

The normalization in Eq. 24 preserves the average scale of the dispreferred log-ratio, while shifting its token-level mass toward forget-relevant positions. NPO [Zhang et al., 2024]. dispreferred log-ratio:

NPO removes the preferred sequence from DPO and retains only the

LNPO (θun ; Df ) = −

2 X log σ −β rθun (x) . β

(27)

x∈Df

The ATWU-augmented version replaces this log-ratio by its scorer-reweighted analogue: 2 X log σ −β rθgun (x) . LgNPO (θun ; Df ) = − β

(28)

x∈Df

As in DPO, the purpose is to preserve the sequence-level scale of the original objective while reallocating the forgetting signal across tokens. 36

SimNPO [Fan et al., 2026]. S IM NPO removes the reference model from NPO and uses a lengthnormalized, margin-shifted score. Let ℓ̂(θ ; x) = ℓ(θ ; x)/|x|. The original loss is 2 X (29) log σ β ℓ̂(θun ; x) − δ . LSimNPO (θun ; Df ) = − β x∈Df

We define the scored length-normalized loss as ℓ̂g (θ ; x) =

ℓg (θ ; x) . |x|

The augmented objective is then LgSimNPO (θun ; Df ) = −

2 X log σ β ℓ̂g (θun ; x) − δ . β

(30)

x∈Df

This keeps the same per-token averaging structure as S IM NPO, but makes the average selective: high-scored tokens contribute more to the forget loss, and low-scored structural tokens contribute less. E.3

Ablations

Note on the ablation runs. The ablations reported in this subsection (Tables 13 and 14) were produced under an earlier version of the codebase whose headline ATWU run differs from the configuration used for the final results in Table 8. The absolute numbers therefore deviate by roughly 10 percentage points on FQ and UQ from the latest ATWU entry in that table, but the relative trends and the qualitative conclusions of each ablation are unchanged. Scorer regularizers. The scorer-side objective in Eq. 8 contains two explicit regularization terms: an entropy penalty, scaled by λH , which encourages scores to become close to binary, and a population penalty, scaled by λρ , which anchors the mean score on Df to the target frequency ρ. In practice, we also apply ℓ2 weight decay to the scorer parameters, scaled by λℓ2 . To isolate the contribution of each term, we toggle each regularizer between off (λ = 0) and on, where the on values are the headline settings λH = 1, λρ = 10, and λℓ2 = 1. This gives the 23 = 8-cell grid in Table 13, with all other components being constant. Result. The population penalty is the dominant factor. Every configuration with λρ enabled achieves FQ ≥ 65, whereas every configuration without it remains in the narrow range FQ ∈ [49, 52], regardless of the entropy or ℓ2 setting. Thus, the mean-anchor is what allows the scorer to move away from the near-uniform initialization and commit to a sparse forget-relevant subset.

Regularizers

Relative

Utility

λℓ2

FQ ↑

RD ↓

UQ ↑

MMLU

Rep.

O RIGINAL

0.0

45.1

559

50.0

49.8 51.4 67.3 66.7 49.3 50.3 65.1 70.5

0.0 1.4 3.3 1.3 0.5 5.0 3.9 3.9

49.8 50.0 63.9 65.4 48.7 45.3 61.2 66.6

45.1 45.0 45.0 45.2 45.1 45.2 45.1 45.0

565 568 570 571 568 567 574 575

49.0 53.0 48.0 50.0 53.5 51.0 50.0 51.0

λH

λρ

✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓

✓

Table 13: Scorer regularizer ablation on TOFU forget10, Llama-3.2-1B-Instruct. ✓ indicates that the corresponding regularizer is on with its headline value (λH = 1, λρ = 10, λℓ2 = 1); a blank cell means the regularizer is set to 0. Rows are ordered by binary expansion of (λH , λρ , λℓ2 ) from all-off to all-on. The bottom row is the headline configuration of ATWU.

Method

Relative

Utility

FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

Relative

Utility

FQ ↑ RD ↓ UQ ↑ MMLU Rep. WR

O RIGINAL

0.0

45.1

559 50.0 O RIGINAL

0.0

45.1

559 50.0

PF PU TF ATWU

35.0 33.4 73.7 70.5

9.0 8.7 2.1 3.9

26.1 24.7 71.6 66.6

45.2 45.3 45.1 45.0

566 566 567 575

31.3 70.5 67.0 39.9

0.2 3.9 3.6 4.5

31.1 66.6 63.4 35.4

45.0 45.0 45.2 45.1

565 575 578 571

53.0 51.0 54.0 51.0

1 5 10 joint

(a) Scorer-state ablations. TF = T RAINED -F ROZEN, PF = P RETRAIN -F ROZEN, PU = P RETRAIN -U NFROZEN.

53.0 51.0 49.0 51.0

(b) Update-frequency sweep.

Table 14: Scorer training-procedure ablations on TOFU forget10, Llama-3.2-1B-Instruct. (a) Three alternatives to joint online training of the scorer. (b) Update-frequency sweep: the scorer is refreshed once per ns model steps; ns = 5 matches the headline configuration of ATWU.

The entropy and ℓ2 terms have smaller, but still measurable, effects. With the population term enabled, removing the entropy penalty reduces FQ from 70.5 to 66.7, but improves RD from 3.9 to 1.3, for a modest UQ drop from 66.6 to 65.4. By contrast, removing ℓ2 weight decay is more costly: FQ falls from 70.5 to 65.1 and UQ from 66.6 to 61.2, with essentially unchanged RD. The full configuration (λH , λρ , λℓ2 ) = (1, 10, 1) achieves the best FQ and UQ overall, while remaining competitive on the other metrics. The only exception is RD, where the best values occur in under-committed λρ = 0 configurations that also fail to forget effectively. We therefore use the full regularizer set in the headline ATWU configuration. Scorer training procedure. The preceding ablations fix the scorer training routine and vary either how the scorer is used or which regularizers shape it. We next ablate the training procedure itself, asking two questions: (i) whether the scorer must co-adapt with the language model during unlearning, and (ii) how often the scorer should be refreshed relative to model updates. Table 14 reports side-byside ablations on TOFU forget10 with Llama-3.2-1B-Instruct. All non-scorer settings match the headline ATWU run. 1.00

0.75

True Positive Rate

Table 14a compares three alternatives to the headline online procedure. T RAINED -F ROZEN (TF) reuses the converged scorer from a previous ATWU run, freezes it, and then retrains the language model from scratch under this fixed weighting. P RETRAIN -F ROZEN (PF) trains a fresh scorer for one epoch on the original, nonunlearned checkpoint and freezes it during unlearning. P RETRAIN -U NFROZEN (PU) uses the same pretrained scorer as an initialization, but continues to update it during unlearning.

0.50

The results show that online co-adaptation is not strictly necessary once a useful scorer has 0.25 Ours (AUC=0.73) Entropy (AUC=0.65) already been learned. TF slightly exceeds the Importance (AUC=0.65) Saturation (AUC=0.35) headline online run on all relative metrics, imNoun Mask (AUC=0.68) logprob Mask (AUC=0.61) proving FQ from 70.5 to 73.7, UQ from 66.6 LLM diff (AUC=0.60) n-gram diff (AUC=0.55) to 71.6, and RD from 3.9 to 2.1. Thus, a con0.00 0.00 0.25 0.50 0.75 1.00 verged scorer can be frozen and reused successFalse Positive Rate fully. However, this does not hold for scorers trained only on the original model: PF reaches Figure 10: ROC curves for token-level forgetonly FQ 35.0, and continuing to fine-tune that relevance detection on TOFU forget10. ATWU scorer in PU performs similarly poorly, with obtains the highest AUROC among the compared FQ 33.4. The key requirement is therefore not scoring methods; the dashed diagonal denotes rancontinuous co-training per se, but exposure to an dom scoring. unlearning trajectory. A scorer pretrained only on the original checkpoint is not a sufficient substitute. 38

Table 14b ablates the scorer update frequency. The scorer is refreshed once every ns ∈ {1, 5, 10} model steps, with all other settings fixed. Although updating every step is the most frequent option, it performs poorly: at ns = 1, FQ drops to 31.3 and RD to 0.2, indicating that the method fails to move the model decisively in either the forget or retain direction. The headline setting ns = 5 gives the best trade-off, achieving FQ 70.5 and UQ 66.6. Slowing the refresh rate to ns = 10 remains competitive, with FQ 67.0 and UQ 63.4 at similar RD. This suggests a broad plateau between five and ten model steps: the exact refresh period is not especially sensitive, but updating the scorer every step is clearly detrimental. Finally, the joint variant, which co-updates the scorer and language model in lockstep rather than using a scheduled refresh, also underperforms, reaching only FQ 39.9 and UQ 35.4. This confirms that the scheduled refresh mechanism, rather than mere simultaneous training, is what allows the scorer to commit to a useful forget-relevant subset.

Record · ID 259487 · SHA-256 ec5aee116b6a4888
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.