arXiv:2604.15166v1 [cs.CV] 16 Apr 2026
Class Unlearning via Depth-Aware Removal of Forget-Specific Directions Arman Hatami Johns Hopkins University
Romina Aalishah Johns Hopkins University
Ilya E. Monosov Johns Hopkins University
Abstract Machine unlearning aims to remove targeted knowledge from a trained model without the cost of retraining from scratch. In class unlearning, however, reducing accuracy on forget classes does not necessarily imply true forgetting: forgotten information can remain encoded in internal representations, and apparent forgetting may arise from classifier-head suppression rather than representational removal. We show that existing class-unlearning methods often exhibit weak or negative selectivity, preserve forgetclass structure in deep representations, or rely heavily on final-layer bias shifts. We then introduce DAMP (DepthAware Modulation by Projection), a one-shot, closed-form weight-surgery method that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, DAMP computes class prototypes in the input space of the next learnable operator, extracts forget directions as residuals relative to retain-class prototypes, and applies a projection-based update to reduce downstream sensitivity to those directions. To preserve utility, DAMP uses a parameter-free depth-aware scaling rule derived from probe separability, applying smaller edits in early layers and larger edits in deeper layers. The method naturally extends to multi-class forgetting through low-rank subspace removal. Across MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, and across convolutional and transformer architectures, DAMP more closely resembles the retraining gold standard than some of the prior methods, improving selective forgetting while better preserving retainclass performance and reducing residual forget-class structure in deep layers. “In the practical use of our intellect, forgetting is as important a function as recollecting.” — William James, The Principles of Psychology, Vol. I
1. Introduction Machine learning models have grown increasingly large and complex, with modern systems often pretrained for extended periods on vast and diverse datasets. After deploy-
Retain Accuracy 100 75
Approximation Gap
Forget Accuracy
50 25
Peak VRAM
Computation Time Backward Passes
Baseline
Retrained
DAMP
GAU
KDU
DD-FT
LM
RandRelabel
SSD
SalUn
Figure 1. Comparison of DAMP performance with the baseline, retrained model, Gradient Ascent Unlearning (GAU), knowledgedistillation unlearning (KDU), Data Deletion Fine-Tuning (DDFT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn) for a 5-layer CNN (CNN-5) on CIFAR-10; forget classes 3 (Cat) and 5 (Dog).
ment, however, these models may need to be modified without full retraining. In this work we focus on class unlearning: removing the knowledge of one or more target classes from a model while preserving performance on the remaining classes [8, 11, 35, 38]. Retraining large models from scratch on datasets that exclude the unwanted forget classes is often impractical due to the computational cost and time required [5, 6, 35]. As a result, recent work has developed post-hoc classunlearning methods that modify trained models directly [11, 14, 25, 38]. The objective of these approaches is to make the model behave as if the forget classes had never been learned, while retaining as much useful knowledge as possible about the remaining classes [5, 35]. Conventionally, most existing work evaluates class unlearning using a tradeoff: reducing performance on the forget classes while preserving performance on the retain
4.08
Retrained
DAMP
2.19 1.33
DDFT SalUn RandRelabel LM Baseline SSD 0.09 KDU 0.17 GAU 0.29
0.05 0.02 0.00 0.00
0
1
2
Selectivity (pp)
3
4
Figure 2. Comparison of DAMP Selectivity with the baseline retrained model, Gradient Ascent Unlearning (GAU), knowledgedistillation unlearning (KDU), Data Deletion Fine-Tuning (DDFT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn) for a 5-layer CNN (CNN-5) on CIFAR-10; forget classes 3 (Cat) and 5 (Dog). Existing methods exhibit weak, nearzero, or negative selectivity, indicating either incomplete forgetting or collateral damage to retained representations. In contrast, DAMP substantially improves selectivity and appears closer to the retrained network (a gold standard comparison). The full equation of Selectivity is given in Sec. A.3.
classes [5, 6, 35]. Fig. 1 summarizes this tradeoff, comparing our method, Depth-Aware Modulation by Projection (DAMP), with retraining and prior unlearning baselines in terms of retain accuracy, forget accuracy, and computational efficiency. Although informative, the approach of measuring the tradeoff as a metric of unlearning does not fully capture the selectivity of the forgetting process. This figure illustrates that a given method may lower accuracy on forget classes while also degrading retain-class performance, or it may also preserve overall utility while leaving substantial evidence for forget classes in the model [6, 16]. To further shed light on this distinction and begin to develop a novel method for class forgetting, we first evaluated the selectivity of the forgetting process, measured in percentage points (pp), a probe-based metric that captures how effectively a method suppresses forget-class evidence without degrading retain-class representations. As illustrated in Fig. 2, existing methods often exhibit relatively weak, nearzero, or even negative selectivity. These patterns of weak selectivity indicate forgetting that arises from broadly damaging representations or from suppressing forget classes only at the classifier head. In contrast, high selectivity indicates that forget-class evidence is removed more strongly than retain-class structure is degraded. Moreover, even when output-level forgetting appears successful, forgotten information can remain encoded in internal representations [6, 16]. This issue becomes more pronounced as a function of network depth. Since early lay-
ers often capture low-level features shared across classes, whereas deeper layers encode increasingly abstract and class-specific structure [37], some residual decodability in shallow layers may not be surprising. However, if forgetclass structure remains detectable in deeper layers following unlearning, then high-level semantic evidence for forget classes has not truly been removed. This is the case even if the forgotten classes are no longer predicted by the output of the network. As we show in Fig. 3 and Fig. S7, several existing methods continue to preserve forget-class structure in deep representations of networks relative to what is found in a retrained network, motivating the development of an unlearning procedure that acts beyond the classifier head. Furthermore, apparent forgetting can arise from changes at the classifier head rather than from genuine removal of forget-class evidence. In particular, shifting the final-layer bias can suppress the prediction of a forget class while leaving its internal representation largely intact. This makes output-level forgetting alone sometimes an unreliable indicator of successful class unlearning [6, 16]. We revisit this mechanism in Sec. 2, and later discuss how several existing methods rely primarily on classifier-head suppression rather than representational removal. To address this issue, we show that class unlearning should modify the parts of the network where classspecific evidence is encoded while preserving shared lowlevel structure. This motivates an approach that (i) operates across network depth and (ii) applies conservative edits in earlier stages and stronger edits in deeper stages [37]. Therefore, we introduce DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form weight-surgery method for class unlearning that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, we define an edit space as the input space of the next learnable operator and compute class prototypes in that space. We then extract forget directions as residual components that are not explained by retain-class prototypes, and apply a projection-based update to the next-operator weights to reduce sensitivity to those directions. To limit utility loss, we scale the update at each stage using a parameter-free coefficient derived from forgetversus-retain probe separability together with a deterministic depth ramp, resulting in smaller edits in earlier layers and larger edits in deeper layers. For multi-class forgetting, we compute one direction per forget class, orthonormalize the resulting set, and project out their span, yielding a lowrank extension without introducing additional hyperparameters. Unlike gradient-based unlearning methods, which iteratively optimize a forgetting objective, DAMP removes forget-specific directions from internal feature spaces. By targeting deep class-specific structure rather than relying only on output suppression, DAMP reduces the risk that forget-class information remains encoded in internal repre-
Baseline - Retrained
Retrained - Retrained (reference)
GAU - Retrained
DAMP - Retrained
KDU - Retrained
+0.610
50
+0.407
100
150
+0.203 200
250
DD-FT - Retrained
Sample Index
0
LM - Retrained
RandRelabel - Retrained
SSD - Retrained
+0.000
SalUn - Retrained
50
-0.203 100
150
Δ Similarity (method − Retrained)
Sample Index
0
-0.407
200
250
0
50
100
150
200
Sample Index
250
0
50
100
150
200
Sample Index
250
0
50
100
150
200
Sample Index
250
0
50
100
150
200
Sample Index
250
0
50
100
150
200
250
-0.610
Sample Index
Figure 3. The first panel (top-left) shows the difference between the RDM of the original model that has not been subjected to unlearning (Baseline) and the retrained (gold-standard) model that has been retrained without the forget class. The other panels show the difference between the RDM of models subjected to different unlearning methods and the RDM of the retrained model. These unlearning methods are DAMP, Gradient Ascent Unlearning (GAU), knowledge-distillation unlearning (KDU), Data Deletion Fine-Tuning (DD-FT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn) for a 5-layer CNN (CNN-5) on CIFAR-10; retain classes 4 (Deer), 5 (Dog), 7 (Horse), 8 (Ship) and forget class 6 (Frog). Existing unlearning methods exhibit larger deviations from retraining, either by preserving stronger forget-class structure or by distorting the geometry of retain classes, whereas DAMP more closely matches retrained network properties. Original RDM plot can be seen in Fig. S6.
2. Related Work
to push predictions away from the forgotten labels; representative examples include zero-shot and unified gradientbased formulations [11, 20, 28]. These methods are simple and can be effective, but they may be unstable and often do not explicitly target the internal representations that encode forget-class evidence. Optimization-based methods combine forgetting and retention objectives, often through distillation, regularization, or auxiliary remain-data constraints [4, 25, 38]. Such formulations offer flexibility, but they operate through objective tradeoffs rather than directly specifying what class-specific structure should be removed. Relabeling-based methods replace forget labels with retainclass labels and then fine-tune the model, thereby altering the output mapping without necessarily removing classspecific evidence from the representation space [15, 18, 27]. Representation-based methods act more directly on hidden features through pruning, saliency-guided edits, synaptic dampening, or other internal modifications, to localize forgetting more precisely [13, 14, 21, 23, 31].
Machine unlearning spans a broad range of settings, including sample-level deletion, concept removal, and classlevel forgetting [2, 5, 6, 8, 35]. In this paper, we focus on class-unlearning, where the goal is to remove knowledge of one or more target classes while preserving performance on the remaining classes. Within this setting, existing methods can be broadly grouped into a few families. Retraining-based methods remove the forget classes and retrain from scratch, providing the idealized baseline but at prohibitive computational cost [5, 10, 22]. Gradientascent-based methods increase the loss on the forget classes
Despite these differences, existing class-unlearning methods often exhibit three recurring limitations. First, forgetting can come at the cost of degraded retain-class performance, a tradeoff that appears across many approximate unlearning methods and benchmark settings [6, 16, 25]. As in Fig. 2 many methods in our experiments can exhibit weak, near-zero, or even negative selectivity, indicating that forgetting is not cleanly separated from collateral damage to retained representations. Second, forget-class information can remain recoverable from internal representations even when top-level forgetting metrics improve, echoing broader
sentations, where it may distort the geometry of retainedclass features, harm generalization on unseen data, and remain vulnerable to re-expression under alternative heads, downstream fine-tuning, or representation-level analysis. Our contributions are as follows: • We show that current class-unlearning methods often fail to achieve selective forgetting: they either damage retained knowledge, preserve forget-class structure across depth, or rely on output-layer suppression. • We propose DAMP, an architecture-agnostic, closedform projection-surgery method for class unlearning that removes forget-specific directions in the input space of the next learnable operator. • We introduce a parameter-free, depth-aware scaling rule based on probe separability and extend the method to multi-class forgetting via low-rank subspace removal.
3. Method Setting. Let F (·; θ) be a pretrained classifier. We are given a set of forget classes F and a retain set Dr containing classes Cr with Cr ∩ F = ∅, following the standard retainforget formulation used in class-unlearning methods [23]. For each forget class f ∈ F , let Df denote its examples. We
baseline) Bias (method
0.20 0.00 -0.20 -0.40 -0.60
Retrained
-0.80
baseline)
A large negative output bias alone does not show successful unlearning, nor is it necessarily harmful to network function. Still, many existing forgetting methods depend on such bias shifts to suppress outputs while leaving forget-class representations mostly intact in intermediate layers (Fig. 3), while DAMP has zero bias shift and removes forget-class representations from the layers of the network (Fig. S7).
Bias (method
evidence that deep networks can preserve discriminative features across layers and transfer settings [31, 37, 39]. As shown in Fig. 3 and further illustrated by the t-SNE [34] visualization in Fig. S7, representational dissimilarity matrices for prior methods often retain substantial block structure associated with forget classes, suggesting that forgetclass evidence is still encoded even when output predictions are reduced. Third, some methods can, in some cases, produce apparent forgetting primarily by changing the decision boundary or classifier head, rather than clear removal of internal class evidence [9, 18, 39]. This distinction matters because output-level suppression and representational removal are not the same phenomenon. A model may stop predicting a forget class while still retaining features that make that class distinguishable in its hidden representations. In that case, the model has reduced output access to the forget class without fully removing the underlying internal evidence. For class unlearning, this can make output-level metrics alone an incomplete measure of success [16, 31, 39]. One indicator of this behavior is the bias term in the final classification layer. In a linear classifier, the bias term contributes a class-dependent offset to the pre-softmax score. Reducing the bias for a forget class makes that class less likely to be predicted, even if the corresponding feature representations remain largely intact [19]. As shown in Fig. 4, several existing methods in our evaluation drive the forget-class bias towards negative values, suggesting that part of their apparent forgetting in fact arises from classifier-head suppression rather than substantial representational removal. Our method is most closely related to representationlevel and post-hoc parameter-editing approaches [13, 14, 21, 23, 31], but differs from them in two key ways. First, rather than editing parameters through iterative optimization, pruning, or neuron-level intervention, we identify at each stage a forget-specific direction; or, in the multi-class case, a low-rank forget subspace; relative to the span of retain-class prototypes. Second, we edit the next learnable operator by right-projecting its weight matrix to reduce sensitivity to those directions, which weakens the ability of downstream computation to exploit forget-class evidence. Because the procedure is one-shot, closed-form, and depth-aware, it directly targets internal class-specific structure while aiming to preserve shared low-level features and retain-class utility [37].
0.20 0.00 -0.20 -0.40 -0.60
DAMP KDU LM SSD
GAU DD-FT RandRelabel SalUn
-0.80
lane bile bird cat deer dog froghorse ship truck airaputomo Figure 4. Comparison of DAMP bias shift with the baseline, retrained model, Gradient Ascent Unlearning (GAU), knowledgedistillation unlearning (KDU), Data Deletion Fine-Tuning (DDFT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn) for a 5-layer CNN (CNN-5) on CIFAR-10; forget class 3 (Cat). The asterisk marks the forget class. Existing unlearning methods often drive the forget-class bias strongly negative, reducing predictions through classifier-head suppression rather than genuine removal of internal evidence. In contrast, DAMP requires less reliance on bias manipulation. More analysis on bias shift is provided in A.8.
produce edited parameters θ′ via a one-shot, closed-form weight update. For the probe we use the union-of-forgets S set DF = f ∈F Df . All required statistics, including class prototypes and probe accuracies, are computed once from the pretrained model; the subsequent weight edit uses no gradients and does not require iterative re-estimation, in contrast to iterative forgetting procedures [23]. Fig. 5 illustrates the full procedure. Starting from a pretrained network, we compute class prototypes in the edit space of each stage. Each forget prototype is then projected onto the span of the retain prototypes, so that the retained component is separated from the forget residual. The residual directions are stacked and orthonormalized to define the e ℓ at that stage. In parallel, a scalar coforget subspace Q efficient αℓ is computed so that deeper stages and stages with stronger forget-retain separability receive stronger edits. These two quantities are then combined in a closed-form projection update that reduces the sensitivity of the next operator to the forget subspace while preserving the remaining representational space.
...
0 1 2 4 6 7 8 9
...
Prototypes
...
...
Retain Classes
Pretrained Network
...
...
...
...
residual
forget class retain class
Main Classes
DAMP 0 1 2 3 4 5 6 7 8 9
Depth
Prototypes ... ...
...
...
Separability
ret
ain
n spa
retain prototype forget prototype forget prototype projections
Figure 5. Overview of the proposed DAMP. Starting from a pretrained network, we compute class prototypes in the edit space of each stage. For each forget class, its prototype is decomposed into a component explained by the retain span and a forget residual. The resulting residual directions are orthonormalized. In parallel, a layer coefficient αℓ is computed from depth and forget-retain separability. These two quantities define the projection update applied to the weights of the next layer.
Edited stages and next-operator view. Consider a sequence of feature-producing stages indexed by ℓ ∈ {1, . . . , L}. For each stage ℓ, let Tℓ+1 denote the next operator that consumes the representation produced at stage ℓ and has a learnable weight tensor. We define an edit-space vector z ℓ (x) ∈ Rdℓ as the actual input to Tℓ+1 , after reshaping if needed so that Tℓ+1 is linear in z ℓ : Tℓ+1 (z ℓ ) = W ℓ+1 z ℓ + bℓ+1 .
(1)
This definition is architecture-agnostic: for any operator with a weight tensor, we flatten its input dimension(s) to obtain z ℓ and the corresponding matrix W ℓ+1 ∈ Rmℓ ×dℓ . This next-operator view is similar in spirit to direct modelediting methods that modify selected linear mappings in pretrained networks via explicit parameter updates [29, 30]. Class prototypes in edit space. For each retain class c ∈ Cr and each stage ℓ, we compute the edit-space prototype 1 X ℓ µℓc = z (x). (2) |Dc | x∈Dc
For each forget class f ∈ F we compute a separate prototype 1 X ℓ µℓf = z (x). (3) |Df | x∈Df
Using class means as prototype representations is standard in feature-space classification and metric-based learning [32]. All prototypes are computed using the pretrained model (before any edits) and are kept fixed during surgery. Recomputing prototypes after each layer update would require an iterative procedure with repeated forward passes and would change the target directions across steps. We note that our method can be easily extended beyond the utilization of the standard class means as prototype representations to capture more complex class distributions if application requires. Forget directions via retain-span residual (multi-1D). For each stage ℓ, stack retain prototypes as columns: Rℓ = µℓc1 , . . . , µℓc|Cr | ∈ Rdℓ ×|Cr | . (4) For each forget class f ∈ F, we compute a retain-span residual dℓf = µℓf − Rℓ (Rℓ )† µℓf , (5) and normalize qfℓ =
dℓf ∥dℓf ∥2
,
(6)
skipping any f for which ∥dℓf ∥2 is below a small numerical tolerance. Our construction is motivated by subspacebased editing and orthogonal representation editing methods, which isolate target information in a low-dimensional
direction or subspace while reducing interference with unrelated behavior [3, 7, 33]. Let Qℓ stack the resulting directions as columns: Qℓ = qfℓ1 , . . . , qfℓK ∈ Rdℓ ×K , (7) where K ≤ |F| after skipping near-zero residuals. We then form an orthonormal basis of span(Qℓ ) using QR decomposition: e ℓ = orth(Qℓ ), Q (8) eℓ
where Q has orthonormal columns. ℓ+1
Projection surgery on the next operator. Let W ∈ Rmℓ ×dℓ be the matrix form of the weight tensor of Tℓ+1 in the edit space of stage ℓ. We remove sensitivity to the subspace spanned by the forget directions via ⊤ ′ eℓ Q eℓ W ℓ+1 = W ℓ+1 I − αℓ Q .
(9)
This projection-style update is aligned with prior subspaceediting approaches that suppress targeted directions while preserving the remaining representational space [3, 7, 33]. We keep the bias bℓ+1 unchanged. For operators whose weights are stored as tensors, we apply Eq. (9) to the flattened matrix W ℓ+1 and reshape back. Layer coefficient from probe separability and depth. The magnitude is controlled by αℓ ∈ [0, 1], shared across all forget classes at stage ℓ. We set αℓ using a scalar probe that measures how well stage ℓ separates the union of forget classes from retain. Let hℓ (x) be a fixed-dimensional pooled summary of the stage-ℓ activation (e.g., global average pooling when the activation is a tensor). Following the standard use of linear probes to assess the separability of intermediate representations [1], we train a logistic classifier with labels y = 1 for x ∈ DF and y = 0 for x ∈ Dr : pℓ (y=1 | x) = σ wℓ⊤ hℓ (x) + bℓ , (10) and denote its held-out accuracy by aℓ ∈ [0, 1]. We convert aℓ to a probe-based strength with chance baseline 0.5: n o αℓprobe = min 1, max{0, 2aℓ − 1} . (11) To enforce small edits in early stages and larger edits in deeper stages, we apply a deterministic depth ramp: αℓdepth =
ℓ , L
(12)
and set αℓ = αℓprobe · αℓdepth .
(13)
Eq. (11)–(13) introduce no tunable hyperparameters. More analysis of α in A.7.
One-shot editing over all stages. We compute {µℓc }, e ℓ }, and {αℓ } once from the pretrained model {µℓf }f ∈F , {Q and apply Eq. (9) sequentially for ℓ = L, . . . , 1 (deep to shallow), producing θ′ . The overall procedure is therefore post-hoc, gradient-free, and one-shot, matching the efficiency goal of retrain-free class-unlearning methods while operating through structured weight editing rather than iterative optimization [23, 29, 30].
4. Experimental Setup Datasets and forgetting protocol. We evaluate class unlearning on four image-classification benchmarks with increasing visual complexity and label cardinality: MNIST [26], CIFAR-10, CIFAR-100 [24], and Tiny ImageNet [36]. For each run, we specify a forget-class set F and treat all remaining classes as retain classes, following the standard class-unlearning setting studied in prior work [23]. We consider both single-class and multiclass forgetting. For MNIST and CIFAR-10, single-class forgetting consists of five runs, each forgetting one class, while multi-class forgetting consists of one run with two randomly selected forget classes. For CIFAR-100 and Tiny ImageNet, single-class forgetting again consists of five runs, each forgetting one class, while multi-class forgetting consists of one run in which 5% of classes are randomly selected as forget classes. For each dataset-architecture pair, the same forget-set configuration is used across all compared methods. Results for single-class forgetting are averaged over the five runs unless otherwise noted. Models and stage definition. We evaluate three architectures: a custom 5-layer convolutional network (CNN5) A.2, ResNet-18 [17], and ViT [12]. Each model is first trained on the full training set to obtain a pretrained baseline prior to unlearning. To apply DAMP consistently across architectures, we expose each network through a unified L = 5 stage abstraction. For CNN-5, the five convolutional blocks define the stages. For ResNet-18, the stages correspond to the stem followed by the four residual groups. For ViT, the transformer blocks are partitioned into five ordered groups. At each stage, DAMP edits the next learnable operator, as defined in Sec. 3. Detailed DAMP pseudocode is provided in Alg. S1. Compared methods. We compare against the following baselines: (i) Baseline, the original model trained on the full dataset before unlearning; (ii) Retrain, retraining from scratch on retain-only data, used as the gold-standard as-if-never-seen reference; (iii) GAU [28], gradient-ascent unlearning, which increases forget loss while encouraging retention of non-forget performance; (iv) KDU [4], knowledge-distillation unlearning, which distills retain behavior from the original model while pushing forget outputs toward a uniform distribution; (v) DD-FT [10], classi-
Table 1. Machine unlearning results for single-class and multi-class forgetting scenarios. We report retain accuracy (Racc) and forget accuracy (F acc), where higher Racc and lower Facc are better. Our approach mostly matches prior methods, while achieving selectivity closer to that of the retrained network, showing that DAMP can erase forget-class representations from hidden layers while remaining competitive with other methods in terms of both forget-class and retain-class accuracy. Single-Class
Racc
Racc
Facc
Facc
MNIST
CIFAR-100
Tiny-ImageNet
Racc
Racc
Racc
Facc
Racc
Racc
Facc
99.55 99.78 89.60 82.50 67.53 44.00 54.92 99.51 0.00 90.70 0.00 67.86 0.00 54.85 83.28 0.00 81.58 0.00 59.17 0.00 49.68 99.40 87.67 89.34 35.50 65.29 15.00 49.98 99.60 0.00 89.43 0.00 64.65 0.00 50.78 99.63 0.00 90.90 0.00 67.71 0.00 54.97 99.49 0.00 90.39 0.00 67.08 0.00 53.74 99.55 0.00 94.09 0.00 66.18 0.00 54.80 99.60 0.00 90.63 0.00 67.80 0.00 54.20 99.47 0.00 90.33 0.00 67.35 0.00 50.38
26.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
99.56 99.59 89.66 87.10 67.57 58.33 54.78 99.65 0.00 92.94 0.00 68.13 0.00 54.69 67.19 0.00 76.70 0.07 7.38 0.00 19.83 99.59 82.71 91.76 48.67 65.13 29.00 50.57 99.52 0.00 92.54 0.00 64.97 0.00 50.69 99.69 0.00 93.04 0.00 67.96 0.00 54.92 99.70 0.00 92.43 0.00 66.85 0.33 53.19 99.56 0.00 89.66 0.00 44.71 0.00 54.94 99.80 0.00 92.53 0.00 67.45 1.00 54.39 99.37 0.00 90.34 0.00 67.42 0.00 34.68
54.67 0.00 0.00 14.00 0.00 0.00 0.00 0.00 0.00 0.00
Baseline Retrained GAU [28] KDU [4] DD-FT [10] LM [2] RandRelabel [27] SSD [14] SalUn [13] DAMP (Ours)
99.42 99.33 84.80 75.60 59.65 38.00 53.44 99.42 0.00 85.96 0.00 59.05 0.00 53.12 28.23 0.00 59.08 0.00 3.70 0.00 7.15 99.09 86.77 83.79 36.40 55.08 11.00 47.37 99.30 0.00 85.44 0.00 55.78 0.00 48.99 99.47 0.00 86.82 0.00 59.76 0.00 53.49 99.54 0.00 86.32 0.00 58.48 0.00 50.67 99.07 0.00 65.11 0.00 31.90 0.00 25.51 99.22 0.00 63.96 0.00 30.12 0.00 26.15 99.70 0.00 86.20 0.00 59.34 0.00 52.80
42.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
99.46 99.28 84.61 82.17 59.64 52.67 53.34 99.72 0.00 87.96 0.00 59.55 0.00 53.15 40.83 0.00 49.31 0.00 1.19 0.00 0.51 99.17 92.44 87.10 62.73 53.24 25.00 49.21 99.59 0.00 88.21 0.00 56.85 0.00 48.91 99.58 0.00 89.36 0.00 60.01 0.00 53.45 99.61 0.00 88.51 0.00 58.46 0.00 51.29 99.43 0.00 66.31 0.00 30.51 0.00 25.55 99.23 0.00 69.07 0.00 30.63 0.00 25.82 99.63 0.00 83.92 0.00 60.84 0.00 52.12
56.00 0.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 0.00
Baseline Retrained GAU [28] KDU [4] DD-FT [10] LM [2] RandRelabel [27] SSD [14] SalUn [13] DAMP (Ours)
99.02 98.88 73.89 76.50 48.11 27.00 37.60 98.95 0.00 76.88 0.00 50.18 0.00 37.57 99.33 0.00 80.29 0.00 48.97 0.00 35.13 99.10 23.88 74.32 56.30 48.94 0.00 37.90 99.20 0.00 79.44 0.00 49.88 0.00 36.63 99.10 0.00 77.81 0.00 48.16 0.00 37.66 99.53 0.00 82.61 0.00 52.11 0.00 39.09 99.07 0.00 75.39 0.00 45.91 0.00 37.71 99.57 0.00 82.48 0.00 51.95 0.00 39.67 99.18 0.00 76.47 0.00 48.02 0.00 37.51
20.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
98.89 99.31 74.44 73.47 48.03 43.67 37.48 98.77 0.00 79.10 0.00 46.80 0.00 37.07 99.11 0.00 83.46 0.00 47.46 0.00 34.57 99.13 22.19 76.16 55.67 49.60 2.00 38.71 99.44 0.00 83.29 0.00 50.28 0.00 36.86 99.14 0.00 82.49 0.00 48.25 0.00 37.63 99.54 0.00 85.61 0.00 51.87 0.00 38.98 98.89 0.00 76.39 0.00 48.04 0.00 37.67 99.32 0.00 85.21 0.00 51.48 0.67 39.26 99.17 0.00 80.16 0.00 47.96 0.00 37.40
39.33 0.00 0.00 6.00 0.00 0.00 0.00 0.00 0.00 0.00
fier reinitialization followed by fine-tuning on retain-only data; (vi) LM [2], inference-time logit masking of the forget classes; (vii) RandRelabel [27], relabeling forget examples with randomly sampled retain-class labels followed by fine-tuning; (viii) SSD [14], selective synaptic dampening, which suppresses weights identified as important to the forget data without full retraining; and (ix) SalUn [13], saliency unlearning, which uses gradient-based weight saliency to guide targeted forgetting while preserving retain-task behavior. These baselines span most of the main fine-tuning, gradient, distillation, masking, relabeling, saliency, and representation-unlearning families commonly used in class-unlearning evaluations [6, 23]. Training and evaluation. All methods are evaluated from the same pretrained baseline for each dataset– architecture pair. Retrain is trained on retain-only data, and all post-hoc unlearning methods use the same retain/forget
Facc
CIFAR-10
CNN-5
Tiny-ImageNet
ResNet-18
Facc
CIFAR-100
Baseline Retrained GAU [28] KDU [4] DD-FT [10] LM [2] RandRelabel [27] SSD [14] SalUn [13] DAMP (Ours)
Arch. Method
Racc
CIFAR-10
ViT
MNIST
Multi-Class
Facc
Facc
partition and matched training budget for each run. We report retain accuracy on the retain-class test split and forget accuracy on the forget-class test split [6, 23]. Because output-level forgetting can reflect either representational removal or classifier-head suppression, we additionally evaluate selectivity, layer-wise representational similarity, and final-layer bias shifts (Secs. A.3–A.8). Full training hyperparameters are provided in A.1.
5. Results We evaluate DAMP on standard class-unlearning benchmarks using retain accuracy and forget accuracy summarized in Table 1 and Figs. 1 and 3. Across single-class and multi-class settings, DAMP consistently achieves a strong retain-forget tradeoff and closely matches retraining. This trend holds across MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet, and across CNN-5, ResNet-18, and ViT ar-
chitectures. In contrast, KDU often leaves substantial residual forget-class accuracy, while GAU frequently reduces retain performance more aggressively. DD-FT, although it achieves performance close to the retrained network, has a higher computational cost than most of the other methods. LM performs well, but it only affects the logits of the forgotten class set, meaning it has no effect on the representations. RandRelabel also performs close to the retrained network, but it alters the geometry of the problem by redirecting the forgotten class to a random class. SSD consistently suppresses forget-class accuracy to zero, but this strong forgetting is frequently accompanied by significant degradation in retain accuracy, particularly on CIFAR-100 and Tiny-ImageNet. SalUn likewise attains near-zero forget accuracy and performs reasonably well in some CNN-5 and ViT cases, yet its retain performance is less stable across architectures and datasets, especially for ResNet-18. We also evaluated the compared methods under adversarial perturbation attacks, as provided in Sec. A.4., we show that methods with similar output-level forgetting can differ substantially in retained-class robustness and in residual vulnerability on forgotten classes. In particular, DAMP maintains strong adversarial retain performance while keeping adversarial forget accuracy at zero.
5.1. Detailed Analyses Fig. S8 studies the effect of the depth-aware scaling factor α, which controls the strength of the projection applied at each layer. The figure shows that this scaling is not a cosmetic design choice: weaker projection leaves substantial residual forget performance, whereas stronger projection increases damage to retained classes. The dynamic setting yields a better operating point between these two failure modes, achieving zero forget accuracy without the corresponding loss in retain accuracy seen at more aggressive settings. This result supports the main design of DAMP by showing that layer-dependent projection strength is necessary to balance forgetting against preservation, rather than treating all layers identically. Fig. S9 isolates a different failure mode. By explicitly sweeping the final-layer bias of the forget class, forget accuracy can be reduced substantially through output-level suppression alone. However, this reduction does not imply that the model has removed forget-class information from its internal representations; it only makes the class less likely to be predicted at the classifier head. As discussed in [19], when the network sees one class less often than the others, it starts to reduce the bias for that class without changing the feature extraction weights. Table S2 evaluates continual unlearning, where classes are forgotten sequentially over multiple rounds rather than in a single step. This setting is more demanding because errors can accumulate: a method may leave residual informa-
tion from earlier forget classes, or progressively damage retained performance after repeated updates. The table shows that DAMP remains close to retraining across rounds, maintaining high retain accuracy while keeping both newly forget accuracy (NF) and all-forget accuracy (AF) near zero. Finally, Fig. S11 extends the evaluation beyond classification and tests whether the same unlearning behavior appears in dense prediction. The segmentation examples compare the baseline model, retraining, DAMP, and competing baselines on representative images. The key question is not only whether the forgotten category is suppressed, but whether this can be done without degrading surrounding retained regions. In these examples, DAMP produces outputs that are visually correct: forgotten regions are removed more cleanly, while the spatial structure and semantic coherence of retained classes are better preserved.
6. Limitations The current formulation of our method has several limitations, which may need to be addressed for other application settings. In particular, under our setting, the representation of a class can be approximated largely by its mean feature vector, and class representations are assumed to be, mostly, linearly separable. These assumptions may not hold for highly multimodal, anisotropic distributions. Our framework could be extended to address this by expanding the computations to estimate class-specific subspaces that account for covariance-sensitive directions. Importantly, the present formulation remains a practical and effective starting point because, in deeper layers, class information often concentrates in a relatively low-dimensional set of directions. Hence, our formulation is an efficient and stable approach for many class-erasure scenarios, preserving shared features among retained classes, while also being extensible to cases where the mean feature vector alone is insufficient.
7. Conclusion This work argues that class unlearning should be evaluated beyond output accuracy, since apparent forgetting can arise from classifier-head suppression while forget-class evidence remains encoded in deep representations. We introduced DAMP, a gradient-free, one-shot, depth-aware projection method that removes forget-specific directions from pretrained networks. Across multiple benchmarks and architectures, DAMP more closely approximates retraining than several prior baselines, improving selective forgetting, preserving retain-class utility, and reducing residual forgetclass structure in deep layers. These findings suggest that structured representation editing is a promising direction for scalable and more faithful machine unlearning, while future work should extend the method to settings with more complex, multimodal class geometry.
Acknowledgments The authors thank their affiliated institutions for support. Arman Hatami (AH) and Ilya E. Monosov (IEM) were supported by the National Institute of Mental Health (NIMH) under grant R01 MH128344. Romina Aalishah (RA) was supported by the Johns Hopkins Department of Electrical and Computer Engineering. AH led the project and developed DAMP. RA contributed to the development and testing of DAMP and participated in writing the manuscript. IEM advised the development of DAMP, assisted with manuscript preparation, and secured funding for the project. We are grateful to all members of the Laboratory of Adaptive and Maladaptive Intelligence (LAMI) for the helpful discussions that improved this manuscript.
References [1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. 6 [2] Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer. Machine unlearning: Linear filtration for logitbased classifiers. Machine Learning, 111(9):3203–3226, 2022. 3, 7, 6 [3] Shristi Das Biswas, Arani Roy, and Kaushik Roy. Cure: Concept unlearning via orthogonal representation editing in diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 6 [4] Jacopo Bonato, Marco Cotogni, and Luigi Sabetta. Is retain set all you need in machine unlearning? restoring performance of unlearned models with out-of-distribution images. In European Conference on Computer Vision, pages 1–19. Springer, 2024. 3, 6, 7 [5] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021. 1, 2, 3 [6] Xavier F Cadet, Anastasia Borovykh, Mohammad Malekzadeh, Sara Ahmadi-Abhari, and Hamed Haddadi. Deep unlearn: Benchmarking machine unlearning for image classification. In 2025 IEEE 10th European Symposium on Security and Privacy (EuroS&P), pages 939–962. IEEE, 2025. 1, 2, 3, 7 [7] Yuchen Cai and Ding Cao. O-edit: Orthogonal subspace editing for language model sequential editing. arXiv preprint arXiv:2410.11469, 2024. 6 [8] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015. 1, 3 [9] Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning. arXiv preprint arXiv:2303.11570, 2023. 4
[10] Rishav Chourasia and Neil Shah. Forget unlearning: Towards true data-deletion in machine learning. In International conference on machine learning, pages 6028–6073. PMLR, 2023. 3, 6, 7 [11] Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 18: 2345–2354, 2023. 1, 3 [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 6 [13] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508, 2023. 3, 4, 7, 6 [14] Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, pages 12043–12051, 2024. 1, 3, 4, 7, 6 [15] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), pages 11516–11524, 2021. 3 [16] Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 497–519. IEEE, 2025. 2, 3, 4 [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6 [18] Zhengbao He, Tao Li, Xinwen Cheng, Zhehao Huang, and Xiaolin Huang. Towards natural machine unlearning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 3, 4 [19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 4, 8 [20] Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient-based machine unlearning with remain geometry enhancement. Advances in Neural Information Processing Systems, 37:26377–26414, 2024. 3 [21] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584–51605, 2023. 3, 4 [22] Junyaup Kim and Simon S. Woo. Efficient two-stage model retraining for machine unlearning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4361–4369, 2022. 3
[23] Sangamesh Kodge, Gobinda Saha, and Kaushik Roy. Deep unlearning: Fast and efficient gradient-free class forgetting. Transactions on Machine Learning Research, 2024. 3, 4, 6, 7 [24] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 6 [25] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems, 36: 1957–1987, 2023. 1, 3 [26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 2002. 6 [27] Junde Li and Swaroop Ghosh. Random relabeling for efficient machine unlearning. arXiv preprint arXiv:2305.12320, 2023. 3, 7, 6 [28] Ioannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi, and Volkan Cevher. Ascent fails to forget. arXiv preprint arXiv:2509.26427, 2025. 3, 6, 7 [29] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359– 17372, 2022. 5, 6 [30] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022. 5, 6 [31] Nazanin Mohammadi Sepahvand, Eleni Triantafillou, Hugo Larochelle, Doina Precup, James J Clark, Daniel M Roy, and Gintare Karolina Dziugaite. Selective unlearning via representation erasure using domain adversarial training. In The Thirteenth International Conference on Learning Representations, 2025. 3, 4 [32] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017. 5 [33] Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. Model editing as a robust and denoised variant of dpo: A case study on toxicity, 2024. arXiv preprint arXiv:2405.13967. 6 [34] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 (86):2579–2605, 2008. 4, 3 [35] Weiqi Wang, Zhiyi Tian, Chenhan Zhang, and Shui Yu. Machine unlearning: A comprehensive survey. arXiv preprint arXiv:2405.07406, 2024. 1, 2, 3 [36] Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017. 6 [37] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014. 2, 4 [38] Yu Zhou, Dian Zheng, Qijie Mo, Renjie Lu, Kun-Yu Lin, and Wei-Shi Zheng. Decoupled distillation to erase: A general unlearning method for any class-centric tasks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20350–20359, 2025. 1, 3
[39] Jianing Zhu, Bo Han, Jiangchao Yao, Jianliang Xu, Gang Niu, and Masashi Sugiyama. Decoupling the class label and the target concept in machine unlearning. arXiv preprint arXiv:2406.08288, 2024. 4
Class Unlearning via Depth-Aware Removal of Forget-Specific Directions Supplementary Material A. Additional Analyses and Supplementary Results A.1. Implementation detail All experiments were run on a single NVIDIA A100 GPU with 80 GB memory. We used PyTorch with a fixed random seed of 42 for Python, NumPy, and CUDA; CuDNN was set to deterministic mode and benchmarking was disabled. Data loading used 4 workers. For baseline training and retraining, CNN-5, ResNet-18 were optimized with SGD with momentum 0.9. On MNIST, these models were trained for 30 epochs with learning rate 0.01 and weight decay 10−4 , without cosine scheduling. On CIFAR-10, CIFAR-100, and Tiny ImageNet, they were trained for 50 epochs with learning rate 0.1 and weight decay 5×10−4 , using cosine annealing. ViT models were optimized with AdamW. On MNIST, ViT used learning rate 10−3 , weight decay 0.05, and 30 epochs. On CIFAR-10 and CIFAR-500, ViT used learning rate 5 × 10−4 , weight decay 0.05, and 100 epochs. On Tiny ImageNet, ViT used learning rate 3 × 10−4 , weight decay 0.05, and 200 epochs. The training batch size was 128 and the evaluation batch size was 256 for all datasets. For unlearning baselines, GAU was run for 10 epochs using Adam with learning rate 10−4 and loss LGAU = CE(retain) − λ CE(forget), with λ = 0.1. KDU was run for 10 epochs using Adam with learning rate 10−4 , temperature T = 4.0, and forget-loss weight λ = 0.5; it matched the teacher on retain samples via KL divergence and pushed forget samples toward the uniform distribution. DD-FT reinitialized the final classifier layer and fine-tuned the full network on retain data for 10 epochs using Adam with learning rate 5 × 10−4 . RandRelabel was run for 10 epochs using Adam with learning rate 10−4 , where forgetclass labels were randomly reassigned to retain classes during training. LM was implemented as inference-time logit masking by setting the logits of forget classes to −∞, without any weight updates. SSD was implemented as Selective Synaptic Dampening by estimating the diagonal Fisher information on the full training set and on the forget set. Parameters satisfying Fforget /Ftrain > α were selected with α = 25.0, and selected parameters were dampened as θ ← θ/ 1 + λ Fforget /Ftrain with λ = 1.0. SSD used no optimizer-based fine-tuning or weight-update epochs. SalUn was run for 10 epochs using Adam with learning rate 10−4 . A saliency mask was first computed from the absolute gradients of the forget-set loss, and the top 50% most salient weights were retained (saliency threshold τ = 0.5). During fine-tuning, forget-class labels were randomly reassigned to retain classes, and gradients on non-
salient weights were zeroed so that only salient weights were updated. For all methods, hyperparameters were adjusted for certain datasets and architectures to achieve the best balance between retention and forgetting accuracy. DAMP details. Class-wise edit means were computed from clean training data. For each layer, forget-class directions were obtained as residuals after projection onto the span of retain-class means, with residual threshold ϵ = 10−8 . Multiple forget directions were orthonormalized with QR decomposition. The layer-wise edit strength was set as αℓ = αprobe,ℓ · αdepth,ℓ , where αprobe,ℓ = clip(2aℓ − 1, 0, 1) was derived from the accuracy aℓ of a logistic-regression probe trained on GAP features, and αdepth,ℓ = (ℓ + 1)/5; since L = 5. Edits were applied in reverse layer order using the closed-form update W ′ = W (I − αℓ QQ⊤ ). Logistic-regression probes used scikitlearn with lbfgs, class weight="balanced", C = 1.0, max iter=1000, and an 80/20 train/test split after feature standardization. In some experiments, α was manually increased by a fixed constant to amplify the effect of weight surgery. For example, for Tiny ImageNet with ViT, all α values were increased by 3.0. The dynamic layer-wise scaling was still preserved, since earlier layers should be edited less than deeper layers.
A.2. Custom CNN-5 architecture. For all datasets, we used the same lightweight 5-stage convolutional network, denoted CNN-5. The network supports variable input channels and numbers of output classes, and is instantiated with 1 input channel for MNIST and 3 input channels for CIFAR-10, CIFAR-100, and Tiny ImageNet. Each stage consists of a convolution, batch normalization, and ReLU activation, with max-pooling applied in the first three stages. The architecture is: Stage 1: Conv(Cin , 64, 3×3) → BN → ReLU → Pool Stage 2: Conv(64, 128, 3×3) → BN → ReLU → Pool Stage 3: Conv(128, 256, 3×3) → BN → ReLU → Pool Stage 4: Conv(256, 256, 3×3) → BN → ReLU Stage 5: Conv(256, 128, 3×3) → BN → ReLU All convolutions use padding 1, and Pool denotes MaxPool(2). The classifier head is AdaptiveAvgPool(1) → Flatten → Linear(128, nc ), where nc is the number of classes.
A.3. Layer-wise Selectivity Metric To quantify the tradeoff between forget-class removal and retain-class preservation, we define a layer-wise selectivity score. At layer ℓ, selectivity is computed as method Selectivityℓ = AUCbaseline forget,ℓ − AUCforget,ℓ (14) method − ACCbaseline retain,ℓ − ACCretain,ℓ . The first term measures forget removal, namely how much the method reduces forget-vs-retain linear separability relative to the baseline model. The second term measures retain damage, namely how much retain-only classification accuracy degrades relative to the baseline. Thus, higher selectivity indicates a more desirable operating point: the method removes more linearly accessible forget information while incurring less damage to the retained representation structure.
A.4. Unlearning Results under FGSM and PGD Evaluation In the main paper, we evaluate unlearning primarily through clean retain and forget accuracy, together with deeper representational analyses. To further assess whether the compared methods remain stable under adversarial perturbations, Table S1 reports results under FGSM and PGD evaluation. Specifically, we report retain accuracy (Retain, ↑) and forget accuracy (Forget, ↓) under both attacks for each method. These results show that DAMP achieves retain accuracy under both attack types that is closer to the retrained network, while keeping forget accuracy at zero, whereas other methods show vulnerability, especially in retain performance under attack.
A.5. Full Representational Dissimilarity Matrices In the main paper, Fig. 3 reports the difference between the representational dissimilarity matrix (RDM) of each method and the retrained reference model. To provide the underlying representation structure directly, Fig. S6 shows the full RDMs for the compared methods at the same deep layer (layer 5). These plots offer a more complete view of how forget-class structure and retained-class geometry are organized in feature space after unlearning.
A.6. t-SNE Visualization of Deep Representations To complement the RDM-based analysis, in Fig. S7, we also visualize deep-layer representations using t-SNE [34]. For each method, we extract features from the same layer used in the representational analysis and project them into two dimensions using a shared t-SNE configuration. This visualization provides an intuitive view of how forget, retain, and novel classes are arranged after unlearning.
Table S1. Unlearning results under FGSM and PGD evaluation. We report retain accuracy (Retain, ↑) and forget accuracy (Forget, ↓) for each method. LM performance is close to DAMP, but it sets the logits of the forgotten classes to −∞ only during inference, meaning there is no representational unlearning.
Method Baseline Retrained GAU [28] KDU [4] DD-FT [10] LM [2] RandRelabel [27] SSD [14] SalUn [13] DAMP (Ours)
FGSM
PGD
Retain
Forget
Retain
Forget
80.9111 80.1778 25.0667 80.2889 78.2333 82.0222 80.0556 81.5333 80.1889 81.9333
84.0 0.0 0.0 32.4 0.0 0.0 0.0 0.1 0.0 0.0
64.0 66.2111 20.0222 63.8667 59.1889 65.2222 61.8667 64.6444 63.4889 65.0778
68.2 0.0 0.0 14.6 0.0 0.0 0.0 0.0 0.0 0.0
Relative to retraining, an effective unlearning method should reduce the separability of the forget class without unnecessarily disrupting the structure of retained classes. Also, the unlearned networks should treat the novel and forget classes similarly, because the network should no longer retain high-level semantic information about the forget class. Nevertheless, the two should not be expected to behave identically: the model has already learned the lowlevel features of the forget class, whereas a novel class may contain low-level patterns the model has never encountered. Consequently, the network may preserve more low-level feature knowledge for the forget class than for the novel class. We emphasize that t-SNE is used only as a qualitative diagnostic, since it does not faithfully preserve global geometry. Our primary evidence remains the RDM analysis and the quantitative results in the main paper.
A.7. Effect of Depth-Aware Scaling A central component of DAMP is the depth-aware scaling factor α, which controls the strength of the projection across layers. Fig. S8 presents the retain–forget tradeoff under different settings of α. The results show that the proposed scaling yields a favorable operating point, balancing strong forgetting with preservation of retained knowledge. This analysis supports the choice used in the main experiments.
A.8. Additional Final-Layer Bias Analysis The paper shows that several baselines suppress forgetting at the output level by strongly shifting the classifier bias of the forget class, rather than genuinely removing forget-class evidence from the internal representation. Fig. S9 extends this analysis by sweeping the final-layer bias and measuring the resulting behavior. The figure further highlights that
Baseline
Retrained
DAMP
GAU
KDU
0.98
50
100
0.80
150
200
0.63 250
DD-FT
0
LM
RandRelabel
SSD
SalUn
Sample Index
0.45 50
Cosine similarity
Sample Index
0
100
0.28
150
200
250
0
50
100
150
200
Sample Index
250
0
50
100
150
200
Sample Index
250
0
50
100
150
200
250
Sample Index
0
50
100
150
200
250
0
Sample Index
50
100
150
200
250
0.10
Sample Index
Figure S6. Full representational dissimilarity matrices (RDMs) comparing DAMP with the baseline, Gradient Ascent Unlearning (GAU), knowledge-distillation unlearning (KDU), Data Deletion Fine-Tuning (DD-FT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn) for a 5-layer CNN (CNN-5) on CIFAR-10; retain classes 4 (Deer), 5 (Dog), 7 (Horse), 8 (Ship) and forget class 6 (Frog).
Baseline
Retrained
DAMP
GAU
KDU
DD-FT
LM
RandRelabel
SSD
SalUn
Deer
Dog
Frog (forget)
Horse
Ship
Truck (novel)
Figure S7. t-SNE visualization of deep-layer (layer 5) features after class unlearning. We project the same representations used in the RDM analysis into two dimensions for qualitative comparison across methods. The forget class is highlighted separately from retained and novel classes. DAMP yields a feature layout that more closely resembles retraining while reducing the visual separability of the forget class, whereas several baselines either preserve stronger forget-class clustering or introduce greater distortion among retained classes. Results are comparing DAMP with the baseline, Gradient Ascent Unlearning (GAU), knowledge-distillation unlearning (KDU), Data Deletion Fine-Tuning (DD-FT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn) for a 5-layer CNN (CNN-5) on CIFAR-10; retain classes 4 (Deer), 5 (Dog), 7 (Horse), 8 (Ship) and forget class 6 (Frog) and novel class 9 (Truck).
output suppression alone can artificially reduce forget accuracy while leaving deeper representations insufficiently unlearned. However, as we show in Fig. S10, DAMP does not
use this shortcut and achieves unlearning without relying on bias shift. Even when it has the same strong bias shift as the retrained network, the results remain unchanged.
91.2
Retain-class accuracy
94.0
α=0.75
α=0.30 α=0.20 α=0.10
93.5
α=0.50
α=0.05
91.0
α=0.50 α=0.30
90.8 α=1.00
α=0.75
93.0
α=0.00
90.6
α=1.50
α=0.20 α=0.10
90.4
92.5
90.2 92.0
α=0.05
α=1.00
90.0 Baseline Retrained DAMP (Dynamic)
91.5 α=1.50
Baseline Retrained DAMP (Dynamic)
89.8
α=0.00
89.6
91.0 0
20
40
60
80
100
0
20
Forget-class accuracy
40
60
80
Forget-class accuracy
Figure S8. Left: CNN-5 Network. Right: ResNet-18 Network. Effect of the depth-aware scaling factor α on the retain-forget tradeoff. We vary the projection strength and report the resulting balance between retain accuracy and forget accuracy. The selected setting used in DAMP achieves strong forgetting while maintaining high retain performance, illustrating the benefit of depth-aware scaling. Results are shown on CIFAR-10; retain classes 4 (Deer), 5 (Dog), 7 (Horse), 8 (Ship) and forget class 6 (Frog).
100
100
92.4
92.4
Retain Forget
80
60
Accuracy (%)
Test accuracy (%)
80
92.4
Retain ± 1 std Mean retain accuracy Forget accuracy (class 6: frog)
40
= 5.5 forget 19% retain 90%
20
60 40 20
0 0
4
8
Bias shift
12
16
20
Figure S9. Additional bias-sweep analysis for the final classifier layer. We vary the bias associated with the forget class and measure the resulting change in model behavior. The results illustrate that reducing forget-class predictions can often be achieved through output-level suppression alone, reinforcing the need for representational analyses beyond classifier outputs. Results are shown for the CNN-5 architecture on CIFAR-10.
A.9. Continual Unlearning Results Table S2 reports continual unlearning performance on CIFAR-10 across sequential class-forgetting rounds. We report retain accuracy (R), newly forget accuracy (NF), all forget accuracy (AF), and the continuous unlearning score
0
0.0
Retrained
0.0
DAMP (unchanged bias)
0.0
DAMP (Retrained bias)
Figure S10. DAMP achieves forgetting through weight-space projection and representation unlearning alone. Forcing the last-layer biases to match the delta bias of the retrained network leaves retain and forget accuracy unchanged, demonstrating that bias plays no role in the forgetting mechanism for DAMP. Results are shown for the CNN-5 architecture on CIFAR-10.
(CUS) after each round. These results show that DAMP remains competitive with retraining throughout the sequential setting, while several baselines either accumulate residual forget-class information or suffer substantial degradation on retained classes.
Table S2. Continuous unlearning results after each class-forgetting round on CIFAR-10 for the CNN-5 architecture. For each method, we report retain accuracy (R, ↑), newly forget accuracy (NF, ↓), all-forget accuracy (AF, ↓), and continuous unlearning score (CUS, ↑) after forgetting the class indicated at each round.
Method
Forget class
Metric
1 (air.) 2 (auto.) 3 (bird) 4 (cat) 5 (deer) 6 (dog) 7 (frog) 8 (horse) 9 (ship)
Retrained
R NF AF CUS
89.2 0.0 0.0 89.2
88.7 0.0 0.0 88.7
90.1 0.0 0.0 90.1
94.2 0.0 0.0 94.2
95.0 0.0 0.0 95.0
97.0 0.0 0.0 97.0
97.1 0.0 0.0 97.1
97.5 0.0 0.0 97.5
100.0 0.0 0.0 100.0
GAU [28]
R NF AF CUS
43.5 0.0 0.0 43.5
19.2 0.0 0.0 19.2
30.3 0.0 0.0 30.3
30.8 0.0 0.0 30.8
37.5 0.0 0.0 37.5
40.9 19.4 3.4 32.9
33.4 32.3 6.6 22.6
49.2 0.0 13.7 49.2
1.5 49.4 17.6 0.8
KDU [4]
R NF AF CUS
87.6 52.3 52.3 41.8
85.8 60.1 56.3 34.2
88.8 47.0 59.6 47.0
91.3 38.3 60.7 56.3
91.3 44.1 60.8 51.0
92.2 24.6 52.5 69.5
92.8 31.0 41.8 64.0
93.9 44.8 46.4 51.8
97.0 59.8 53.3 39.0
DD-FT [10]
R NF AF CUS
88.6 0.0 0.0 88.6
88.5 0.0 0.0 88.5
90.3 0.0 0.0 90.3
93.9 0.0 0.0 93.9
96.1 0.0 0.0 96.1
97.6 0.0 0.0 97.6
98.2 0.0 0.0 98.2
98.0 0.0 0.0 98.0
100.0 0.0 0.0 100.0
LM [2]
R NF AF CUS
89.5 0.0 0.0 89.5
89.3 0.0 0.0 89.3
90.9 0.0 0.0 90.9
95.1 0.0 0.0 95.1
96.5 0.0 0.0 96.5
98.1 0.0 0.0 98.1
98.3 0.0 0.0 98.3
98.0 0.0 0.0 98.0
100.0 0.0 0.0 100.0
R NF RandRelabel [27] AF CUS
89.5 0.0 0.0 89.5
88.4 0.0 44.5 88.4
89.5 0.0 62.8 89.5
92.6 0.0 67.6 92.6
91.8 0.0 69.4 91.8
93.3 0.0 73.0 93.3
92.8 0.0 74.5 92.8
93.2 0.0 76.8 93.2
95.3 0.0 78.0 95.3
SSD [14]
R NF AF CUS
86.4 0.0 0.0 86.4
70.8 42.3 21.2 40.8
72.9 4.2 15.4 69.6
77.4 37.8 20.0 48.1
81.0 58.3 27.9 33.8
77.8 94.1 38.9 4.59
70.9 98.2 47.4 1.28
63.7 85.4 52.2 9.3
56.0 71.4 54.3 16.0
SalUn [13]
R NF AF CUS
89.5 0.0 0.0 89.5
88.5 0.0 45.7 88.5
89.3 0.0 62.4 89.3
92.4 0.0 67.7 92.4
91.9 0.0 69.2 91.9
93.0 0.0 74.0 93.0
93.1 0.0 74.5 93.1
93.9 0.0 76.8 93.9
94.8 0.0 78.1 94.8
DAMP (Ours)
R NF AF CUS
89.5 0.0 0.0 89.5
89.1 0.0 0.0 89.1
90.6 0.0 0.0 90.6
94.9 0.4 0.1 94.5
96.0 0.0 0.0 96.0
98.0 0.0 0.0 98.0
98.2 0.0 0.0 98.2
98.2 0.0 0.0 98.2
100.0 0.0 0.0 100.0
A.10. Qualitative Segmentation Unlearning Results We further provide qualitative results for semantic segmentation unlearning in Fig. S11. The figure compares the input image, ground-truth mask, and segmentation outputs produced by the baseline model, retraining, DAMP, and com-
peting baselines on representative examples. These examples highlight the visual effect of unlearning beyond aggregate accuracy metrics. Compared with retraining, successful unlearning should remove evidence of the forgotten class while preserving the spatial structure and semantic coherence of retained re-
Algorithm S1 DAMP: Depth-Aware Model Projection for Class Unlearning Require: Baseline model fθ with L layers; forget set F; retain set R; loaders {Dc }c∈F ∪R Ensure: Unlearned model fθ′ 1: Initialize θ ′ ← θ 2: for each class c ∈ F ∪ R, layer ℓ, minibatch x ∼ Dc do 3: Compute edit-space vectors z (ℓ) (x): apply GAP if W (ℓ+1) is linear, else unfold a(ℓ) (x) into convolutional patches matched to W (ℓ+1) P (ℓ) (ℓ) 4: Accumulate sc += z (ℓ) (x), nc += (batch × locations) 5: end for (ℓ) (ℓ) (ℓ) 6: Compute prototypes µc ← sc /nc for all c, ℓ 7: for each layer ℓ = 1, . . . , L do (ℓ) 8: Train linear probe on GAP features {hc (x)} (forget vs. retain, 80/20 split); obtain accuracy aℓ ℓ 9: αℓ ← clip(2aℓ − 1, 0, 1) · L 10: end for 11: for ℓ = L, L−1, . . . , 1 do (ℓ) 12: Form retain matrix R(ℓ) = µr r∈R 13:
(ℓ)
For each f ∈ F, compute residual df R
14: 15: 16: 17:
(ℓ)
R
(ℓ) µf (ℓ)
(ℓ)
= µf
−
(ℓ)†
(ℓ)
(ℓ)
(ℓ)
Collect Q = {df /∥df ∥ : ∥df ∥ > ε, f ∈ F} if Q(ℓ) ̸= ∅ then e (ℓ) ← QR([q (ℓ) ]f ∈F ) Q f e (ℓ)⊤ {flatten conv e (ℓ) Q W (ℓ+1) ← W (ℓ+1) I − αℓ Q weights before, reshape after}
18: end if 19: end for 20: return fθ′
gions. DAMP produces outputs that are visually closer to retraining, whereas several baselines either retain residual forgotten-class predictions or induce larger distortions in surrounding retained regions.
Image 1 Image 2
Image
GT
Baseline
Retrained
DAMP
GAU
KDU
DD-FT
LM
RandRelabel
SSD
SalUn
Image
GT
Baseline
Retrained
DAMP
GAU
KDU
DD-FT
LM
RandRelabel
SSD
SalUn
Figure S11. Qualitative results for semantic segmentation unlearning on two representative examples trying to forget Human category. Dataset is Pascal VOC 2012 semantic segmentation and the network is DeepLabV3 with a ResNet-50 backbone. We show the input image, ground-truth mask (GT), and predictions from the baseline, retraining, Gradient Ascent Unlearning (GAU), knowledge-distillation unlearning (KDU), Data Deletion Fine-Tuning (DD-FT), Logit Masking (LM), Random Relabeling (RandRelabel), Selective Synaptic Dampening (SSD), and Saliency Unlearning (SalUn). DAMP more closely follows the retrained model by suppressing the forgotten regions while preserving the structure of retained classes, whereas several baselines either retain residual forgotten-class predictions or introduce greater degradation in the remaining segmentation regions.