OSR: Output Space Redistribution for Adaptive Label Removal in Classification Models

OSR: Output Space Redistribution for Adaptive Label Removal in Classification Models Minyi PENG, Darian Gunamardi, Ivan Tjuawinata, Yongsen Zheng, Kwok-Yan Lam

arXiv:2609.03972v1 [cs.LG] 3 Sep 2026

Nanyang Technological University, 50 Nanyang Ave, Singapore, 639798

Abstract. Label removal occurs frequently in classification systems with evolving taxonomies, where categories must be dynamically updated or eliminated. To accommodate such changes, classification models must adapt accordingly. Existing solutions, broadly categorized as retraining-based and feature-spaceadjustment-based, share common limitations despite their variations, including reliance on access to original data, substantial computational and storage costs, inconsistent results, poor scalability, and degradation of model utility. To address this, we propose a novel approach that leverages statistical redistribution in the output space to approximate the post-removal confidence vectors of a retrained model. Applicable as a modular output filter, our method bypasses the burden of feature-space adjustments or loss-function convergence, alleviating scalability limitations. Furthermore, by requiring only existing labels and prior output confidences, the method potentially mitigates privacy concerns inherent to data-dependent solutions. Extensive experiments demonstrate competitive performance against full retraining, with improvements in computational efficiency and privacy preservation across several classification tasks. Keywords: Output Space Redistribution · Label Removal · Classification Model

1

Introduction

Label removal during taxonomy or catalog revisions is a frequent occurrence in classification systems. Such removals are often formalized as category deprecation or retirement—for example, in e-commerce taxonomies, categories periodically merge, split, or discontinue as markets evolve, requiring downstream classifiers to retrain or adjust [12]. Similarly, in financial industry classification systems such as the North American Industry Classification System (NAICS), codes are periodically merged, split, or redefined on multi-year revision cycles, necessitating downstream analyses and models to migrate away from retired codes [22]. To ensure alignment with evolving data, classification models trained on affected labels must be updated accordingly. Unlike relational databases, which offer relatively accessible information and diverse update mechanisms, no analogous solution exists for classification models. Currently, there are two representative families of label-removal methods: retraining-based label-removal methods [1, 4, 6, 7, 29] and feature-space-adjustment-based label-removal algorithms [10, 18, 26, 28, 30]. Retraining-based methods aim to make models forget targeted data and remove the associated information from the model.

2

M. Peng et al.

Compared to a fully retrained model that excludes the eliminated data, these methods seek to replicate its performance using fast, practical mechanisms that selectively erase the target information without full retraining. These methods typically require thorough removal and yield a new model with updated weights and outputs. Building on this, feature-space-adjustment-based algorithms focus on tracing parameters back to their last training state to retrieve the unlearned outputs, thereby bypassing full retraining and training-phase modifications. They require some access to the original training data or even its intermediary representations such as feature-space or hidden-layer activations, and the frequent involvement of Hessian-matrix computations renders these methods expensive and difficult to scale to large and complex models. Despite their effectiveness, most existing methods suffer from several major limitations: a) Both retraining-based and feature-space-adjustment-based label-removal algorithms often require partial of full retraining, which often yields inconsistent behavior across runs and settings. More critically, feature space adjustment is highly prone to catastrophic consequences, thereby destabilizing performance. In contrast, our proposed method does not require access to the original training data, retraining, or any modifications to the model parameters or architecture. As a result, the original model is largely preserved while achieving substantial reductions in training time and computational complexity; b) These methods are not model-agnostic solutions applicable to all classifier models; instead, they are designed and optimized for specific classifier architectures and training settings. Consequently, their generalization across different classifiers is limited, and directly transferring them to other models typically requires additional adaptation, parameter tuning, and implementation refactoring, reducing universality and deployment flexibility. This lack of model-agnostic applicability imposes substantial customization and integration costs when deploying across classifiers. To address these challenges and offer a more practical solution for model adaptation to label removal conditions, we propose a novel approximation approach that entirely avoids retraining, parameter or loss function modification, and direct access to the original training data. The proposed method achieves an efficient, consistent, and scalable removal effect directly on model output. By studying class-wise differences in the forward training phase, we approximate the training process, reverse its logic, and apply a filter at the output layer to adjust model outputs instantly. This filter operates in two steps, projection of the original output confidence vectors to remove the impact of deprecated labels, followed by redistribution of the remaining confidences to erase residual label influence while preserving model utility and post-removal performance. To validate the effectiveness of our method, experiments are conducted across small to largescale vision models under single and multi-class removal settings. Results are directly compared against the gold standard of full retraining with detailed evaluations through statistical assessment of post-removal model performance. Extensive comparisons with leading parameter-based and loss function-based approaches further demonstrate the efficacy, consistency, and scalability of our method. In summary, the main contributions of this work are as follows: 1. We propose a novel training-free model approximation approach for label removal that operates directly in the output space, requiring no retraining, parameter or loss

OSR for Adaptive Label Removal in Classification Models

3

function modification, and scales efficiently across both single and multi-class removal requests. 2. This approach approximates classification model label removal through boundary organization in softmax logits, employing a two-step filter of projection and redistribution to guarantee consistency and post-removal model utility, while requiring only lightweight label-level statistics in place of direct access to original training data or intermediary representations. 3. Extensive experiments across four datasets demonstrate the effectiveness of the proposed method in adapting classification models to label removal conditions.

2

Related Work

2.1

Output Space Re-Distribution

Output space re-distribution treats the softmax output itself, rather than the model’s parameters, as the site of intervention, redistributing an existing class’s probability mass among the remaining known classes without retraining the underlying representation. This connects to two lines of prior work that also treat a model’s output distribution as a site for principled redistribution, though for different ends. In open-set recognition, OpenMax [3] replaces the softmax layer with a calibrated alternative that redistributes probability mass toward a newly introduced “unknown” class whenever an input looks sufficiently unlike anything seen during training. OpenMax and our filter therefore share a mechanism—post-hoc redistribution of an existing model’s output rather than a newly learned decision function—but the direction is reversed: OpenMax expands the output space to accommodate a class the model never modeled, whereas our method contracts it, redistributing an existing, well-modeled class’s mass among the remaining known classes to approximate the confidence vector of a model that had never been trained on that class. Output space re-distribution shares a closer conceptual parallel with knowledge distillation, which similarly treats a model’s full output distribution, not just the top prediction, as carrying transferable structure between classes [31]. Our method relies on the same intuition: the relative probabilities a model assigns to non-target classes encode inter-class structure, and the way a model redistributes a removed class’s mass across the remaining classes draws on that same structure. The two approaches differ in how it is used: distillation transfers it by training a separate student model to reproduce a teacher’s output behavior, whereas our method acts directly on the original model’s output probabilities, without modifying parameters or requiring retraining. 2.2

Label Removal in Class Unlearning

Label removal through class unlearning targets the complete removal of one or more classes’ influence from a trained model, falling under the broader scope of machine unlearning, which emphasizes complete deletion of targeted information from a model’s parameters [19]. The most common approach modifies parameters directly: Warnecke et al. [27] propose an influence-function-based method that estimates the training set’s

4

M. Peng et al.

influence on parameters and applies a closed-form update, requiring access to the original training data and convexity/continuity assumptions on the loss. Related approaches similarly achieve label removal via closed-form solutions [23] or through fine-tuning strategies that accelerate model updates [17], sharing the common trait that removal is achieved by modifying the model’s parameters, whether through direct weight updates, influence estimation, or gradient-based fine-tuning. Our method belongs instead to a line of work that shifts focus away from the parameter set toward the output layer. The most closely related is linear filtration, proposed by Baumhauer et al. [2]: for classification models of the form σ (W · f (x)) where f (x) is a feature extractor and W is the final linear layer preceding softmax σ, they derive a closed-form linear transformation of W that removes the discriminative information associated with the deleted class, framing label removal as model sanitization under a black-box threat model with reduced cost relative to retraining. Both linear filtration and our method target class-wide label removal, avoid gradient-based retraining, and are motivated by computational efficiency. However, they differ in where the adjustment is made: linear filtration modifies the model’s final linear layer directly, requiring the model to take the specific form σ (W · f (x)) and requiring access to W itself. Our method instead never inspects or modifies any layer, operating entirely on the already-computed output confidence vector and deriving its redistribution directly from the model’s emitted probabilities rather than the weights that produced them. This also affects generality: linear filtration is defined only for models with a linear decision boundary in the extracted features, whereas our filter applies unchanged to any model that outputs a probability vector, linear or otherwise.

3

Problem Statement

In this work, we consider the problem of classification model adaption to label removal conditions, where a pretrained classification model must be updated to accommodate one or more label removal requests. The adapted model should retain both the knowledge of the remaining labels and the ability to apply that knowledge, while matching the performance of a newly trained model on the updated label set. The adaptation process must also be realistic, considering aspects such as instancy, efficiency, compatibility and the level of access availability to the model and data. Let f : X → [0, 1]n be a classification model that has been trained using a dataset T D with labels in L = {1, · · · , n}. Let D be the reference data for the label removal process. Suppose that the set of labels to be removed are Ldel ⊆ L. Define Lr = L\Ldel . We aim to develop a procedure Π such that given f, reference data D, the removed labels Ldel , and an input x, it produces the confidence vector of the adjusted model of length n − 1 corresponding to labels in Lr with input x. Specifically, given f, D and Ldel , Π produces a classification model f ′ : X → [0, 1]n−|Lr | such that f ′ approximates f ∗ : X → [0, 1]n−|Lr | , a classifier model retrained from scratch using Dr ≜ {(x, y) ∈ DT : y ∈ Lr }. Here the measure of approximation is defined based on commonly used machine learning metrics including the accuracy difference and the KL divergence between the two models.

OSR for Adaptive Label Removal in Classification Models

5

We assume that the pre-trained model f outputs a confidence vector. Define fˆ as a related function that outputs final predicted decision by taking the label corresponding to the highest confidence level of f . We further assume some performance guarantee requirements on the pre-trained model f. We require that f is sufficiently well trained to learn information regarding the training data. This is represented by the ability of fˆ to predict the correct label with high enough accuracy. Furthermore, it can also be observed from the fact that for any input x, if fˆ(x) outputs the correct label, its corresponding confidence is sufficiently high. This requirement ensures the variance of the predicted confidence in f to be small. These constrain the variance of predicted confidence in f . By the assumptions we have made, we can assume that the confidence vectors output by M P for such data points have a smaller variance around their averages. This provides a sharper estimate of the projection space. To approach the problem, we first consider how classification training is done in general. A classical classification model training uses training data with all possible n labels simultaneously to optimize a function that outputs a real-valued vector of length n, containing probability of the data being from each of the n labels. Here we note that under some smoothness and convexity assumptions of the loss function, a similar model can be obtained through the use of class incremental learning (CIL) (see for example [32]) where labels are introduced sequentially, which is an implication of basic convex optimization theory. This also implies that models established by using CIL in different label introduction orders will converge to the same global minimum. One phenomenon that we need to consider in CIL is the catastrophic forgetting. This may happen when training using data with the next label without reconsidering the data from the previously learned labels. To avoid such phenomenon, CIL typically requires some information about the data from previously learned labels to influence the current iteration of training. In this work, we are particularly interested in the effect of introducing a new label in CIL from the perspective of the evolution of label boundary, which interpretation is formalized in the following section 4.1.

4

Methodology

Our method treats adaptation to label removal as a boundary reorganization problem. Specifically, it considers how a new boundary expands during the training phase of classification, and how to reverse that process back to the previous boundary state. The following section 4.1 shows how boundary expansion occurs when a new label is learned. Section 4.2 explains how the boundary shrinks to erase the label’s effect and its residual influence, under the condition of a single label removal. Given the lightweight nature and low complexity of this procedure, as demonstrated in section 4.3, the classification model handles multi-label removal through iteration of single-label removal. A visualization and workflow of the OSR procedure can be found in Fig. 3. 4.1

Boundary Expansion considered CIL

We consider the following strategy of how CIL happens. For i = 2, · · · , n, we start from fi−1 : X → [0, 1]i−1 , trained using data points with labels belonging to {1, · · · , i − 1}. For

6

M. Peng et al.

j = 1, · · · , n, we denote by D j the subset of D containing data points with label j and Sj D(c) j = i=1 D j . At step i, we consider the training that updates fi−1 to fi . Intuitively, this is done by first embedding the output space of fi−1 to [0, 1]i , which is then trained further using Di to obtain fi . This process can be summarized in the following 4 steps. – Stage 1. For each x ∈ X, denote c(i−1) = fi−1 (x) as a confidence vector of length x i − 1. – Stage 2. Given c(i−1) , we initialize a small constant α0 (x) > 0, and normalize c(i−1) x x to have sum 1 − α0 (x). This produces a new vector ((1 − α0 (x))c(i−1) ∥α0 (x)) ∈ [0, 1]i x of length i with sum of entries being equal to 1. – Stage 3. Train fi : X → [0, 1]i which is a classification model with i labels using Di as the train data set with further information obtained from fi−1 (x) for x ∈ D(c) i−1 to (trans) optimize the values of α0 (x) and c(i−1) . More specifically, we train α(x) and c x x where fi (x) = ((1 − α(x))c(trans) ∥α(x)), here α(x) is initialized as α0 (x) and c(trans) x x is initialized as c(i−1) . x – Stage 4. Once the training in stage 3 is done, we have the final values of α(x) ) (i) and c(trans) which we denote by αT (x) and c(T x x . We define c x ≜ fi (x) = ((1 − (T ) αT (x))c x ∥αT (x)). Visualization of the process is reflected in Figure 1. Note that for x ∈ Di , assuming that fi has a sufficiently high accuracy for input x, we can see fi (x) as approximately (c) ′ ′ orthogonal to fi−1 (x′ ) for x′ ∈ D(c) i−1 . Hence, in order to recover fi−1 (x ) for x ∈ Di−1 , we may consider the projection of fi (x) to the vector space orthogonal to fi (x) for x ∈ Di .

Fig. 1: Transition Within the Output Space When Learning a New Label

4.2

Reverse Output Semantics of One label

To recover the semantics of the classification model fi−1 (x), we reverse the update from fi (x) to fi−1 (x) in the output vector space. To achieve this, we adopt the learning transition described in Section 4.1 and model its reverse process, as illustrated in Fig. 2. Starting from fi (x), which has output dimension n, we first construct a reference space U that approximates the output vector space of fi−1 (x). – Projection with Reference Space U Given a group of test data Dtest with to-be-removed label i ∈ L, an average coni fidence vector c̄i is generated with respect to classification model predictions fi (x) where x ∈ Dtest i . The null space of the average vector c̄i is computed with GaussJordan elimination to get basis vectors {vi,1 , · · · , vi,n−1 } ⊆ Rn of solution space

OSR for Adaptive Label Removal in Classification Models

7

Fig. 2: Reversing the Transition Within the Output Space to Remove An Existing Label

c̄Ti · v = 0. Further, Gram-Schmidt orthogonalisation is performed to obtain a set of orthonormal basis {qi,1 , · · · , qi,n−1 } based on the intermediate basis {vi,1 , · · · , vi,n−1 }. This new basis {qi,1 , · · · , qi,n−1 } constructs a matrix Ai corresponding to the Reference Space U with its projection matrix Pi = Ai · ATi . For any vector v projected to the reference space U, a projected vector should be obtained by v p = Pi · v. Based on our assumption, the average confidence vector c̄i should capture the characteristics of the removed label i. The reference space U orthogonal to c̄i scrubs most information off from it. When the confidence vectors ci output by classification model fi for such data points have a smaller variance around their averages, this should provide a sharper estimate of the reference space. This reference plane U indicated by matrix Ai resembles the decision plane of fi−1 (x), as shown in Stage 1 in Section 4.1. For any confidence vector c x output by classification model fi (x), we perform projection to the reference space to obtain its projected confidence c xp . Here after, we split the confidence vector into scalar of the removed label i and sub-vector of the remaining labels L \ {i}. ) This is to emulate the concatenate vector (1 − αT (x))c(T x ∥αT (x) as denoted in Stage 2 to 4 to further tackle them separately. We first consider the scalar confidence score c xp of the removed label i from the projected vector c xp . It is relatively small after projection and resembles α0 (x) defined in Stage 2. Consider the initialization of α0 (x), it is a small ratio coming from the confidence vector c(i−1) output by classification model fi−1 (x). Consider from another x perspective of classification model fi (x), c xp indicates the residual impact of the removed label i. Therefore, one more step is made to further distribute this residual probability . c xp back to the original semantic vector c(i−1) x – Redistribution of the Removed label Probability Given the same group of test data Dtest that generates matrix Ai corresponding to i the Reference Space U above, all of their predictions y = fi (x) are projected to the space U, returning a list of projected removed label confidences y xp . For every entry in the projected confidence y xp , we take its absolute value to form a new vector z xp and further normalize it to have sum 1, denoted as vector z̃ xp . By removing its i-th p entry, it results a new list of vectors z̃ x,r . The distribution ratio vector c̄i,r for any p p residual probability c x is then obtained by averaging the list of z̃ x,r . p P For any residual probability c x , we have its removed result c x · c̄i,r . We then consider the sub-vector crx of the confidence vector c x of length i − 1. It indicates the confidence values of the remaining labels L \ {i}. We reckon the output

8

M. Peng et al.

Fig. 3: Overview of our OSR filter, which applies a post-hoc output-space redistribution comprising a projection step and a redistribution step. It removes the influence of the removed labels, reweights the remaining labels, and reassigns the residual probability mass associated with the removed labels, using a classifier trained on the updated label set as reference. This adjustment revises the original predicted confidence while preserving semantic consistency.

semantics of the remaining i − 1 dimensions not undergoing an obvious change, thus the redistribution is about adjusting its ratio back to the portion of ci−1 x in Stage 2. – Redistribution of the Remaining label Probability The proportion of the remaining label probability crx in classification model fi (x) is denoted by 1 − cUx , where cUx is the i-th entry of the predicted confidence vector c x from classification model fi (x). When the output semantics returns to the initialized stage - Stage 2, the ratio of crx turns back to 1 − α0 (x). This ratio is approximated by 1 − c xp , corresponding to the projected confidence value of the removed label. 1−cP For any remaining label probability crx , we have its redistributed result 1−cUx · crx . x

The complete predicted result of reversing output semantics of a label is to add the 1−cP separate result of its removed label cPx · c̄i,r and the remaining labels 1−cUx · crx together, x normalize to have sum 1. With this entire operation, the reversed output semantics is expected to preserve the original distribution of classification model fi (x), while ensembles the predicted decisions of classification model fi−1 (x). Details of the algorithm are provided in Algorithm 1. 4.3

Asymptotic Complexity Analysis

We analyze the asymptotic complexity of our proposed procedure. We assume complexity is measured in terms of the number of real number multiplication or division operations, and that real number addition has negligible complexity by comparison. We denote by CDiv the complexity of a single real number division operation, measured

OSR for Adaptive Label Removal in Classification Models

9

Algorithm 1 OSR v ← OSR( f, i, Di , x) 1: Initiate c̄i ← 0n where 0n is the zero vector of length n; 2: for x ∈ Di do 3: Calculate y x ← f (x); 4: c̄i ← c̄i + y x ; 5: c̄i ← |D1i | c̄i ; 6: Perform Gauss-Jordan elimination to find {vi,1 , · · · , vi,n−1 } ⊆ Rn , the basis for the solution space of c̄Ti · v = 0; 7: Perform Gram-Schmidt Orthogonalization from {vi,1 , · · · , vi,n−1 } to obtain an orthonormal basis {qi,1 , · · · , qi,n−1 } of the solution space of c̄Ti · v = 0; 8: Define Ai ∈ Rn×n−1 with qTi, j as its j-th column and Pi = Ai · ATi ; 9: for x ∈ Di do 10: Calculate y xp ← Pi · y x ; 11: Calculate z xp , by taking the absolute value of each entry of y xp , and z̃ xp , obtained by normalizing z xp to have sum 1; p 12: Construct z̃ x,r from z̃ xp by removing its i-th entry; P p ; 13: Calculate c̄i,r ← N1i x∈X z̃ x,r 14: Calculate c x ← f (x). Denote by cux , its i-th entry and crx , the vector obtained from c x by removing its i-th entry; 15: Calculate cPx ← Pi · c x and retrieve cPx , its i-th entry; 16: Calculate c1 ← cPx · c̄i,r ; 1−cP 17: Calculate c2 ← 1−cUx · crx and b ← cUx + 1 − cPx ; x

18: Output 1b (c1 + c2 ) ;

in terms of the number of real number multiplications, and by C( f ) the complexity of one call to f , likewise measured in terms of real number multiplications. We further note that normalization and averaging can be done by first calculating X1 where X is the denominator before multiplying it to the vector being averaged or normalized. This allows the reduction of number of divisions (which is the more expensive operation) by replacing it with 1 division operation and multiple multiplication operations. Table 1 summarizes the complexity of each step in our Algorithm 1. Summing over all steps, the overall complexity of our proposed method is C =   (Ni + 1) · C( f ) + (Ni + 4) · CDiv + O n3 + Ni n2 . Since N > Ni ≫ n in our setting, this simplifies toC = (Ni + 1) · C( f ) + (Ni + 4) · CDiv + θ(Ni n2 ). Based on this approximation of the overall complexity, asymptotically we require the same number of inference computation (referring to the computation of f ) as to the division operations. Similarly, we asymptotically require O(n2 ) times more multiplication operations than the call to f. However, in practice, we can assume that one call to the the model computation is much more complex than requiring just 1 division operation or n2 multiplications. Hence, we can expect that the dominant cost of our proposed solution is O(Ni )C( f ).

5

Experiments and Analyses

In this section, we consider the performance of our proposed solution along with its comparison to existing baseline methods.

10

M. Peng et al.

Table 1: Asymptotic complexity of steps in Algorithm 1.

Step (of algorithm 1) Complexity (in number of multiplication operations)

5.1

Steps 2– 4 Step 5 Step 6–7 Step 8 Steps 9–13 Step 14 Step 15 Step 16 Step 17 Step 18

Ni · C( f ) CDiv + n O(n3 ) O(n3 ) (Ni + 1)CDiv + O(Ni n2 ) C( f ) O(n2 ) n−1 CDiv + (n − 1) CDiv + (n − 1)

Total

  (Ni + 1) · C( f ) + (Ni + 4) · CDiv + O n3 + Ni n2 ≈ O(Ni )C( f )

Dataset

In our experiment, we consider the implementation of our proposed solution in the following datasets, image classification models, and sets of removed-labels. 1. CIFAR-10 [15] that is trained from scratch using the image classification models ResNet18 [13] and AllCNN [21]. In our experiment, among the 10 labels {0, · · · , 9}, we consider four cases for the sets of the labels to be removed, namely {0}, {0, 4}, {0, 4, 7, 9}, and {0, 2, 3, 4, 6, 7, 9}. We call these forget sets F1 , · · · , F4 respectively. 2. CIFAR-100 [15] fine-tuned from the image classification models ResNet18 and MobileNetV2 [20] which are pre-trained using the ImageNet-1K [8]. For CIFAR100 dataset with labels denoted by {0, · · · , 99}, we consider four cases for the sets of the labels to be removed, {0, 1, · · · , i − 1} for i ∈ {1, 20, 40, 60}. We call these forget sets F1 , · · · , F4 respectively. 3. VGGFace-1001 fine-tuned from the image classification models ResNet18 and Vision Transformer B16 [9] which are pre-trained with ImageNet-1K. Similar to the sets of removed labels in CIFAR-100, the 100 labels are denoted by {0, · · · , 99} and we consider four different sets of the removed labels {0, · · · , i − 1} for i ∈ {1, 20, 40, 60}. We call these forget sets F1 , · · · , F4 respectively. 5.2

Baseline Methods

In our experiments, we compare OSR with state-of-the-art machine unlearning methods, namely UNSIR [25] and NHLE-CWM [26]. We consider the desired target behavior to be the model obtained from the approach of retraining the model with the whole dataset without those with their true label belonging to the set of removed labels. For UNSIR and NHLE-CWM, we implement their proposed algorithms following the parameters stated in their respective works [25, 26] prioritizing our implementation to reproduce their best performance with the following remark. 1

A 100-identity subset of VGGFace2 [5], constructed by randomly sampling 100 identities and their associated images from the full dataset.

OSR for Adaptive Label Removal in Classification Models

11

Table 2: Comparison of accuracy metrics on CIFAR10, CIFAR100, and VGGFace-100 between base model (B), Retrained, (R), OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). Bolded entries show the best performance in each setting. Forget Classes(AR ) ↑ Forget Classes (AF ) ↓ Dataset Model Method F1 F2 F3 F4 F1 F2 F3 F4

ResNet18

B R OSR [25] [26]

0.9296 0.9314 0.9352 0.8891 0.9044

0.9263 0.9325 0.9386 0.8773 0.9035

0.9173 0.9243 0.9425 0.8242 0.8448

0.9423 0.9823 0.9953 0.9523 0.9377

0.929 0 0 0 0

0.9425 0 0 0 0

0.9478 0 0 0.0005 0

0.924 0 0 0.0014 0.0047

AllCNN

B R OSR [25] [26]

0.9181 0.9187 0.922 0.8737 0.892

0.9139 0.9193 0.9284 0.8629 0.906

0.9062 0.919 0.9312 0.8353 0.751

0.9347 0.975 0.9943 0.917 0.882

0.861 0 0 0.004 0

0.9065 0 0 0.0025 0

0.9218 0 0 0.005 0.0003

0.9029 0 0 0.0007 0.01529

ResNet18

B R OSR [25] [26]

0.8171 0.8123 0.8176 0.8148 0.8041

0.8179 0.834 0.8426 0.8204 0.8154

0.8167 0.8517 0.8708 0.807 0.8313

0.8165 0.8855 0.9183 0.6745 0.8593

0.94 0 0 0 0

0.82 0 0 0.0445 0

0.8208 0 0 0.0638 0

0.8195 0 0 0.0848 0

B R MobileNet V2 OSR [25] [26]

0.8066 0.8080 0.8070 0.8063 0.7898

0.8054 0.8233 0.834 0.8018 0.8074

0.8045 0.8418 0.8603 0.7652 0.8115

0.805 0.8795 0.9068 0.5928 0.7988

0.9 0 0 0 0

0.816 0 0 0.065 0

0.812 0 0 0.0635 0

0.8092 0 0 0.0435 0

ResNet18

B R OSR [25] [26]

0.9415 0.9417 0.9419 0.9380 0.9341

0.9451 0.9429 0.9530 0.9273 0.9399

0.9462 0.9427 0.9602 0.9215 0.9261

0.9423 0.9413 0.9629 0.9049 0.6413

0.9882 0 0 0 0

0.9280 0 0 0 0

0.9349 0 0 0 0

0.9419 0 0 0 0

Vision Transformer B16

B R OSR [25] [26]

0.9524 0.9482 0.9527 0.9460 0.9514

0.9547 0.9511 0.9605 0.8935 0.0927

0.9554 0.9571 0.9652 0.8898 0.0435

0.9520 0.9481 0.9665 0.8008 0.0213

0.9882 0 0 1 0

0.9444 0 0 0.7995 0

0.9485 0 0 0.7619 0

0.9535 0 0 0.4885 0

CIFAR10

CIFAR100

VGG Face-100

– For our implementation of UNSIR: • Instead of the use of Adam [14] as the optimizer, we consider the use of the more powerful SGD [24] as the optimizer to provide a fair comparison with our approach that uses SGD optimizer. Without such alteration, it causes the performance of UNSIR to be significantly lower. • For the implementation on VGGFace-100, we reduce the noise parameter from 0.1 to 0.001 to account for the 49 times larger L2 term for 224× 224 inputs compared to that for 32×32 inputs. • For the implementation on CIFAR-100, we increase the sample size used for each batch from 50 to 100 while for the implementation on VGGFace-100, the batch size is increased from 100 to 500. This alteration is needed to ensure that the repair step of UNSIR covers sufficient amount of data to perform in an acceptable level. – For our implementation of NHLE-CWM: • We set the learning rates to the following 0.001 for CIFAR-10, 0.0001 for CIFAR-100, and 0.0001 on VGGFace100. This is different from the implementation described in [26], which implies the use of 0.01 for all implementations. This change is done because using the original learning rate causes runaway gradient problem in our experiment.

12

M. Peng et al.

Table 3: Comparison of coverage metric on CIFAR10, CIFAR100, and VGGFace-100 between base model (B), Retrained, (R), OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). Bolded entries show the best performance in each setting Forget Classes(CovR ) ↓ Forget Classes (CovF ) ↑ Dataset Model Method F1 F2 F3 F4 F1 F2 F3 F4

ResNet18

B R OSR [25] [26]

0.5222 0.5432 0.4321 0.5889 0.5556

0.5125 0.6094 0.375 0.7 0.725

0.55 0.6944 0.4444 0.7167 0.7

0.4667 0.5556 0.4444 0.8333 0.8667

0.6 ∞ ∞ 0.9 0.7

0.6 ∞ ∞ 0.85 0.65

0.5 ∞ ∞ 0.85 0.6

0.5571 ∞ ∞ 0.8429 0.3

AllCNN

B R OSR [25] [26]

0.6333 0.6049 0.4198 0.7778 0.6222

0.625 0.5625 0.3906 0.675 0.625

0.6333 0.6667 0.4444 0.85 0.8167

0.6333 0.6667 0.4444 0.8667 0.9

0.7 ∞ ∞ 0.9 0.7

0.7 ∞ ∞ 0.9 0.65

0.65 ∞ ∞ 0.875 0.575

0.6429 ∞ ∞ 0.8857 0.3

ResNet18

B R OSR [25] [26]

0.2016 0.1923 0.1749 0.2118 0.2173

0.2059 0.1936 0.0506 0.2403 0.1960

0.1827 0.1675 0.0436 0.2495 0.1423

0.1788 0.1875 0.0388 0.3830 0.1155

0.04 ∞ ∞ 0.99 0.51

0.1765 ∞ ∞ 0.919 0.4555

0.2260 ∞ ∞ 0.9003 0.3688

0.2142 ∞ ∞ 0.8458 0.2670

B R MobileNet V2 OSR [25] [26]

0.1959 0.1670 0.1750 0.2036 0.217

0.2020 0.2069 0.0516 0.2504 0.1861

0.1763 0.1781 0.0433 0.2732 0.1737

0.1808 0.1938 0.0444 0.4033 0.2803

0.04 ∞ ∞ 0.9900 0.52

0.1635 ∞ ∞ 0.7990 0.451

0.2213 ∞ ∞ 0.8698 0.3618

0.2033 ∞ ∞ 0.871 0.2662

ResNet18

B R OSR [25] [26]

0.0741 0.0915 0.0691 0.0817 0.0855

0.0733 0.0883 0.0339 0.1028 0.0715

0.0763 0.1017 0.0319 0.1247 0.0675

0.0758 0.1456 0.0375 0.1515 0.2890

0.01 ∞ ∞ 0.98 0.52

0.0745 ∞ ∞ 0.8295 0.4510

0.0693 ∞ ∞ 0.9063 0.3638

0.072 ∞ ∞ 0.9213 0.2767

Vision Transformer B16

B R OSR [25] [26]

0.0727 0.0725 0.0659 0.0811 0.073

0.0813 0.0684 0.0363 0.1333 0.5169

0.0755 0.1 0.0347 0.1483 0.5505

0.0693 0.1356 0.0419 0.1908 0.4138

0.01 ∞ ∞ 0 0.5

0.0355 ∞ ∞ 0.1045 0.5375

0.067 ∞ ∞ 0.1813 0.4655

0.074 ∞ ∞ 0.3382 0.3603

CIFAR10

CIFAR100

VGG Face-100

• When considering removal of more than 20 labels, to avoid the significant increase in the loss value, we run NHLE-CWM for 3 epochs instead of 10 epochs. Our experiments are conducted on a virtual machine equipped with two NVIDIA A100 (80 GB) GPUs, AMPD EPYC 7763 64-core, and 64GB RAM running Ubuntu 22.04. 5.3

Evaluation Metrics

We briefly discuss the evaluation metrics defined in the works on machine unlearning [25, 26] and multilabel learning [11] that will be used as our evaluation metrics. 1. Accuracy. Based on the output after the application of the solutions, we may take the label with the highest confidence value to be the prediction of the classifier. Hence, for each dataset D and each possible label value i, we measure the proportion of Di that is correctly predicted to have label i. We are mainly interested in finding the average of such proportions for the labels retained, denoted by AR , as well as the average of such proportions for the removed labels, denoted by AF . It is clear that the better solution should have higher AR while AF should be 0.

OSR for Adaptive Label Removal in Classification Models

13

Table 4: Comparison of KL Divergence on CIFAR10, CIFAR100, and VGGFace-100 between retrained model (R) against OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). Bolded entries show the best performance in each setting. For each metric, the arrow indicates whether better solutions have larger (↑) or smaller (↓) value. Forget Classes(KLRR ) ↓ Forget Classes (KLFR ) ↓ Dataset Model Method F1 F2 F3 F4 F1 F2 F3 F4 ResNet18

OSR [25] [26]

0.1784 0.5683 0.4157

0.1729 0.6290 0.444

0.2316 1.281 1.386

0.0888 0.3616 0.9566

1.082 1.6559 2.1313

1.2932 1.9389 2.7903

1.6925 3.4369 4.8446

2.1947 0.8137 8.0864

AllCNN

OSR [25] [26]

0.1102 0.4003 0.2450

0.1020 0.4504 0.1633

0.1324 0.6379 1.8146

0.0851 0.4836 1.2182

0.6176 0.8151 1.2961

0.5881 1.0849 1.0497

0.6663 1.3144 3.4006

1.2840 1.2034 5.4858

ResNet18

OSR [25] [26]

0.1733 0.1818 0.2445

0.1961 0.2 0.2349

0.2217 0.2585 0.2693

0.2261 0.5685 0.3231

0.3272 0.3926 0.6308

0.6041 2.4210 0.6554

0.9072 5.1018 0.9578

1.1154 8.9134 1.4806

OSR MobileNet V2 [25] [26]

0.1987 0.1981 0.3014

0.2237 0.26 0.2712

0.2540 0.4296 0.3911

0.2704 1.2407 0.938

0.2853 0.3177 0.4268

0.7122 2.601 0.7965

1.0202 4.8616 1.3189

1.2177 8.4283 2.7282

ResNet18

OSR [25] [26]

0.1631 0.1946 0.2060

0.1405 0.1881 0.2277

0.1545 0.2076 0.5316

0.1580 0.2337 3.7939

0.8174 0.9632 0.9889

0.8886 5.13 1.2484

0.9698 6.4925 1.7372

1.1304 8.2943 3.8979

Vision Transformer B16

OSR [25] [26]

0.0996 0.1244 0.1002

0.0907 0.7858 7.3222

0.0943 0.8738 7.89

0.1191 2.2414 8.7059

1.1588 9.0109 1.0712

0.9416 5.5759 4.6728

1.0542 5.6367 4.5615

1.3507 4.0039 6.4601

CIFAR10

CIFAR100

VGG Face-100

2. Coverage. Coverage is a metric that is similar to the accuracy metric. However, instead of focusing only on the label with the highest confidence, we are more interested to measure the position of the true label when the labels are sorted based on its confidence value in the output of the approach. It is clear that if the average of such position is small, then even when the label with highest confidence is not the true label, the true label is still predicted with high probability. More specifically, for each x ∈ D with true label i which is not removed, for an approach F, we define rF,x,i ∈ Z>0 such that F(x) has its i-th value to be the rF,x,i -th largest value among the entries of F(x). We further define rF,i = max x∈Di rF,x,i , the maximum value of rF,x,i among all x ∈ Di . Lastly, the coverage of F, denoted by rF is defined to be the average of rF,i for all i ∈ Lr . To normalize the value so that a perfect F gives CovR (F) = 0, we define CovR (F) to be rF − 1. That is: P CovR (F) = |L1r | i∈Lr max{rF,x,i : x ∈ Di } − 1. We extend the definition of coverage to also consider the removed labels. More specifically, we define CovF (F) similarly where instead of considering the non-removed labels Lr , we consider the removed labels L f . Here for solutions that simply remove the forgotten labels, we simply set it as ∞. 3. KL divergence [16] of confidence vectors. Beyond the top label, we quantify the similarity of full output vectors to the retraining baseline; smaller average divergences indicate closer approximation. We report two metrics: (a) retained-label inputs and (b) forgotten-label inputs. A good label-removal method should preserve information about non-removed labels (low divergence from the base model) while removing information about removed labels (high divergence when the input is from a removed label). 4. Processing Time. We are interested in measuring the time required by different approaches in completing the label removal process. It is clear that the smaller the

14

M. Peng et al.

Table 5: Comparison of KL Divergence on CIFAR10, CIFAR100, and VGGFace-100 between base model (B) against OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). Bolded entries show the best performance in each setting. For each metric, the arrow indicates whether better solutions have larger (↑) or smaller (↓) value. Forget Classes(KLRB ) ↓ Forget Classes (KLFB ) ↑ Dataset Model Method F1 F2 F3 F4 F1 F2 F3 F4 ResNet18

OSR [25] [26]

0.0030 0.4629 0.1955

0.0087 0.6060 0.2739

0.0154 1.1698 1.0356

0.0077 0.4203 0.6971

8.9727 9.9941 9.4887

9.3584 10.3125 11.5819 10.588 11.7677 12.1166 10.1972 11.6639 11.4442

AllCNN

OSR [25] [26]

0.0072 0.3336 0.1061

0.0132 0.4902 0.0749

0.0195 0.6867 1.7802

0.0142 0.7073 1.3557

5.1187 5.4451 5.4616

6.4038 6.9287 6.5652

7.8772 8.5991 9.8556

9.2597 9.6862 9.4157

ResNet18

OSR [25] [26]

0.0004 0.0268 0.0652

0.0341 0.0412 0.0662

0.064 0.089 0.0828

0.0880 0.4346 0.1235

4.1952 4.2453 4.4348

3.938 4.5003 3.9725

4.7698 6.1877 4.6271

5.6617 8.0276 5.3763

OSR MobileNet V2 [25] [26]

0.0005 0.0213 0.0793

0.0357 0.0832 0.0767

0.0688 0.2314 0.1459

0.0986 1.0471 0.6394

4.6331 4.646 4.8304

3.9890 4.1023 4.0432

4.7484 5.7880 4.6947

5.5058 7.978 5.8235

ResNet18

OSR [25] [26]

0.0001 0.0316 0.0340

0.0058 0.0459 0.0771

0.011 0.0565 0.3467

0.0156 0.0821 3.9801

4.3195 4.5182 4.4576

4.2046 7.9142 4.4413

4.8145 9.6922 5.2942

5.7763 10.9717 7.1377

Vision Transformer B16

OSR [25] [26]

<0.0001 0.0029 0.0241 0.7553 0.0013 7.3263

0.0056 0.8393 7.9734

0.0079 2.5364 8.9262

5.2295 0.0187 5.2324

4.6097 1.2922 7.4741

5.2415 1.5609 7.6715

6.0458 4.2886 9.6077

CIFAR10

CIFAR100

VGG Face-100

processing time, the better from the perspective of its feasibility to be applied in a real-life application. Here, for different approaches, we measure the time required for each step to better understand the bottleneck of different approaches. We note that we omit two metrics considered in [25, 26], namely the retraining time and weight distance. We recall that we consider the problem of label removal from the perspective of the output distribution instead of the perspective of the model producing such output. So instead of modifying the trained model to approximate the retraining approach, our approach serves as a post-hoc procedure to process the output, which can be seen as a filter to be applied to existing models. This leads to the following observations. Firstly, since our approach applies a filter to the output of the original model, to recover the original model, we can simply remove the filter. Hence, retraining time is trivial. Similarly, the weights for the parameters of the model never change, hence the distance is 0. We note that because of this, our approach does not suffer from the risk of catastrophic forgetting that may happen in other approaches, especially when the number of removed labels increases. 5.4

Experimental Results

In this section, we discuss some of the measurement results from our experiments. Accuracy As has been previously discussed, we first consider the comparison of both retain accuracy AR and forget accuracy AF in different datasets and models. We also consider four different forget set with increasing size for each dataset where the forgotten set is sampled randomly with the requirement that F1 ⊆ F2 ⊆ F3 ⊆ F4 . As can be observed in Table 2, OSR outperforms the state-of-the-art solutions UNSIR [25] and

OSR for Adaptive Label Removal in Classification Models

15

Table 6: Comparison of Processing Time (in seconds) on VGGFace-100 between retrained model (R), OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). For Retraining method, Ave: Average per Epoch, Tot (n): Total for n Epochs. For OSR, Pro: Projection, Red: Redistribution, Tot: Total. For UNSIR, NG: Noise Generation, RC: Retain Collection, Im: Impair, Re: Repair, Tot: Total. For NHLE-CWM, FT: Finetune, CWM: Class Weight Masking, Tot: Total Forget Classes(Time) ↓ Model Method Component F1 F2 F3 F4 R

Ave Tot(5)

130.4148 652.074

109.0052 545.026

84.2198 421.099

57.9094 289.547

OSR

Pro Red Total

0.145 0.022 0.167

0.306 0.021 0.327

0.494 0.039 0.533

0.594 0.018 0.612

[25]

NG RC Im Re Tot

52.435 19.409 651.557 563.656 1287.057

1044.551 18.87 1753.912 271.413 3088.746

2088.334 20.14 1977.476 175.297 4261.247

3132.282 20.342 2588.275 176.551 5917.45

[26]

FT CWM Tot

3.662 0.0 3.662

16.283 0.002 16.285

30.635 0.001 30.636

44.172 0.002 44.174

R

Ave Tot(5)

25.5474 127.737

22.0204 110.102

17.8888 89.444

12.7862 63.931

OSR

Pro Red Total

0.137 0.021 0.158

0.2 0.02 0.22

0.229 0.023 0.252

0.242 0.021 0.263

[25]

NG RC Im Re Tot

3.807 18.747 206.522 42.992 272.068

61.821 20.786 969.596 21.597 1073.8

124.053 19.795 1005.548 80.029 1229.425

185.884 19.733 1216.515 9.531 1431.663

[26]

FT CWM Tot

3.226 0.0 3.226

12.571 0.001 12.572

23.646 0.001 23.647

34.595 0.002 34.597

ResNet18

Vision Transformer B16

NHLE-CWM [26] in both AR and AF , which shows that after applying our solutions, the resulting output still preserves very high accuracy in retained labels and it has 0 probability to still predict the forgotten labels. We note that in most cases, our accuracy in AR is higher than both the base model (before forgetting) and the retrained model (where a fresh model is trained just using the training data belonging to the retained labels). This can be argued in the following manner. Compared to the baseline model, since some labels are removed, data from a remaining label that is close to the removed label may no longer be misclassified as the label that is being removed. Because of this, we can expect a better separation between remaining labels. This can also be observed in the fact that most retrained models have better average accuracy than the base model. Secondly, OSR outperforms the accuracy of the retrained model due to the fact that the information that can be learned from the removed label will still help our solution in refining the prediction of the remaining labels better, which can be observed from the higher AR of OSR compared to that of the retrained model. Coverage Next we consider the coverage metric. Recall that this provides the average rank of the true label in the output of the label removal process. In our experiment, since the number of removed labels grows, in order to allow for comparison to be done between settings with different size of labels, we consider a normalized coverage, CovR

16

M. Peng et al.

Table 7: Comparison of Processing Time (in seconds) on CIFAR10 between retrained model (R), OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). For Retraining method, Ave: Average per Epoch, Tot (n): Total for n Epochs. For OSR, Pro: Projection, Red: Redistribution, Tot: Total. For UNSIR, NG: Noise Generation, RC: Retain Collection, Im: Impair, Re: Repair, Tot: Total. For NHLE-CWM, FT: Finetune, CWM: Class Weight Masking, Tot: Total Forget Classes(Time) ↓ Model Method Component F1 F2 F3 F4 R

Ave Tot(40)

7.9859 319.437

7.8332 313.328

7.3648 294.592

5.7423 5.7423

OSR

Pro Red Total

0.004 0.004 0.008

0.037 0.016 0.053

0.005 0.004 0.009

0.003 0.044 0.007

[25]

NG RC Im Re Tot

1.628 1.082 2.624 1.46 6.794

1.94 1.09 3.049 1.333 7.412

3.974 1.08 4.475 0.879 10.408

6.821 1.155 6.203 0.386 14.565

[26]

FT CWM Tot

0.809 0.0 0.809

0.458 0.0 0.458

0.531 0.0 0.531

0.685 0.0 0.685

R

Ave Tot(40)

5.1727 206.909

5.6598 226.391

5.1892 207.566

4.4128 176.511

OSR

Pro Red Total

0.004 0.032 0.036

0.03 0.004 0.034

0.003 0.004 0.007

0.003 0.044 0.007

[25]

NG RC Im Re Tot

0.951 1.104 1.614 0.807 4.476

0.996 1.112 1.505 0.832 4.445

1.989 1.097 2.43 0.464 5.98

3.377 1.135 3.697 0.229 8.438

[26]

FT CWM Tot

0.826 0.0 0.826

0.408 0.0 0.408

0.489 0.0 0.489

0.596 0.0 0.596

ResNet18

AllCNN

which divides CovR by the number of possible labels the resulting model considers. Hence, this divides the coverage of Base Model, UNSIR, and NHLE-CWM by the original number of labels (10 for CIFAR-10, 100 for CIFAR-100 and VGGFace-100) while the coverage of retrained and OSR approach is divided by a smaller number since the output model no longer considers the removed labels. The summary of the result can be found in Table 3. As observed, OSR outperforms even the retrained model in terms of coverage where for retained labels, the true label is constantly predicted towards the top of the list while removed labels are never considered. The performance of OSR becomes significantly better when considering larger set of labels. This suggests that OSR constantly puts the true label towards the top of the list and the larger set of possible labels causes this to have a much smaller normalized coverage.

KL Divergence We next consider the KL divergence comparison. Recall that since our aim is to approximate the model obtained through retraining, in this section, we only consider the KL divergence between retraining and OSR as well as the two stateof-the-art solutions, UNSIR [25] and NHLE-CWM [26]. We consider the average KL divergence of the outputs in two groups: when the input belongs to one of the non-

OSR for Adaptive Label Removal in Classification Models

17

Table 8: Comparison of Processing Time (in seconds) on CIFAR100 between retrained model (R), OSR (ours), UNSIR ([25]), and NHLE-CWM ([26]). Bolded entries show the best performance in each setting. For each metric, the arrow indicates whether better solutions have larger (↑) or smaller (↓) value. For Retraining method, Ave: Average per Epoch, Tot (n): Total for n Epochs. For OSR, Pro: Projection, Red: Redistribution, Tot: Total. For UNSIR, NG: Noise Generation, RC: Retain Collection, Im: Impair, Re: Repair, Tot: Total. For NHLE-CWM, FT: Finetune, CWM: Class Weight Masking, Tot: Total Forget Classes(Time) ↓ Model Method Component F1 F2 F3 F4 R

Ave Tot(5)

37.8354 189.177

30.614 153.07

23.9884 119.942

16.8228 84.114

OSR

Pro Red Total

0.162 0.031 0.193

0.358 0.029 0.387

0.353 0.025 0.378

0.721 0.026 0.747

[25]

NG RC Im Re Tot

3.784 6.948 199.902 12.765 223.399

61.691 6.461 1156.747 107.521 1332.42

124.666 7.177 257.804 17.752 407.399

187.689 6.841 315.584 3.415 513.529

[26]

FT CWM Tot

3.577 0.0 3.577

9.966 0.001 9.967

15.513 0.001 15.514

18.982 0.002 18.984

R

Ave Tot(5)

57.3078 286.539

47.0098 235.049

36.2396 181.198

25.5374 127.687

OSR

Pro Red Total

0.164 0.03 0.194

0.429 0.041 0.47

0.421 0.056 0.477

0.402 0.048 0.45

[25]

NG RC Im Re Tot

5.539 7.031 221.252 77.738 311.56

97.707 6.921 782.43 82.331 969.389

194.632 6.561 646.702 7.078 854.973

294.216 7.338 905.26 4.546 1211.36

[26]

FT CWM Tot

3.086 0.001 3.087

8.925 0.001 8.926

15.918 0.001 15.919

19.688 0.002 19.688

ResNet18

MobileNet V2

removed labels KLRR and when the input belongs to one of the removed labels KLFR . The summary of the result can be found in Table 4. As can be observed, OSR only produces output with the highest similarity with the target output from the retrained model, which shows its capability to approximate the behavior of a freshly retrained model. It can also be observed that with the bigger number of removed labels, OSR exhibits much slower performance degradation rate in approximating the output of the retrained model, providing evidence of its robustness in handling multiple labels removal. We further report the summary of KL divergence measurement between the solutions and the base model, which are denoted by KLRB and KLFB respectively and can be found in Table 5. As can be observed, OSR consistently produces outputs that are similar to that of the base model, which shows that the amount of knowledge being removed is due to the label removal. This makes sense considering that OSR only produces a filter to modify the output without changing the model itself. This also explains the fact that OSR output still exhibits similar behaviour to the base model when fed a data point from the removed label.

18

M. Peng et al.

Processing Time Lastly, we compare the time required for solutions of label removal, namely, retraining, OSR, UNSIR [25], and NHLE-CWM [26]. As previously discussed, the time measurement for the different approaches are decomposed to main components while only the total time is being compared. The summary of the time measurement for datasets CIFAR10, CIFAR100, and VGGFace-100 can be found in Tables 7, 8, and 6 respectively. It is easy to observe that in all cases, the retraining method, UNSIR, and NHLECWM are finetuning methods while OSR is an output processing filter, causing the processing time for OSR to be significantly lower than any of the three other solutions.

6

Conclusion

Label removal is a common challenge when label taxonomies evolve and categories must be updated or discarded. Most existing methods suffer from several limitations: reliance on access to the original data, high computational and storage costs, inconsistent results, limited scalability, and potential degradation of model utility. To this end, we proposed a novel paradigm that redistributes probability mass in the output space to approximate the post-removal behavior of a retrained model. Viewed as a modular output-filter, it avoids any feature-space adjustments or loss-function convergence, addressing scalability concerns. Because it operates only on the remaining labels and the model’s prior output confidences, it also mitigates privacy issues typical of datacentric solutions. Extensive experiments across multiple classification tasks demonstrate competitive performance to full retraining, with notable gains in efficiency and privacy preservation. Disclosure of Interest The authors declare that they have no conflict of interest.

Bibliography

[1] Aldaghri, N., Mahdavifar, H., Beirami, A.: Coded machine unlearning. IEEE Access 9, 88137–88150 (2021) [2] Baumhauer, T., Schöttle, P., Zeppelzauer, M.: Machine unlearning: Linear filtration for logit-based classifiers. Machine Learning 111(9), 3203–3226 (2022) [3] Bendale, A., Boult, T.E.: Towards open set deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1563–1572 (2016) [4] Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C.A., Jia, H., Travers, A., Zhang, B., Lie, D., Papernot, N.: Machine unlearning. In: 2021 IEEE symposium on security and privacy (SP). pp. 141–159. IEEE (2021) [5] Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: VGGFace2: A dataset for recognising faces across pose and age. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). pp. 67–74. IEEE (2018) [6] Chen, C., Sun, F., Zhang, M., Ding, B.: Recommendation unlearning. In: Proceedings of the ACM Web Conference 2022 (WWW). pp. 2768–2777 (2022) [7] Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., Zhang, Y.: Grapheraser: An efficient algorithm for graph unlearning. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS). pp. 499– 513 (2022) [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A largescale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 248–255. IEEE (2009) [9] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR) (2021) [10] Foster, J., Schoepf, S., Brintrup, A.: Fast machine unlearning without retraining through selective synaptic dampening. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 12043–12051 (2024) [11] Gibaja, E., Ventura, S.: A tutorial on multilabel learning. ACM Computing Surveys (CSUR) 47(3), 1–38 (2015) [12] Gross, L., et al.: Cross-platform e-commerce product categorization and recategorization: A multimodal hierarchical classification approach. arXiv e-prints (2025) [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016) [14] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015) [15] Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Tech. rep., University of Toronto (2009) [16] Kullback, S., Leibler, R.A.: On information and sufficiency. The Annals of Mathematical Statistics 22(1), 79–86 (1951)

20

M. Peng et al.

[17] Li, G., Hsu, H., Chen, C.F., Marculescu, R.: Fast-ntk: Parameter-efficient unlearning for large-scale models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 227–234 (2024) [18] Mehta, R., Pal, S., Singh, V., Ravi, S.N.: Deep unlearning via randomized conditionally independent hessians. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10422–10431 (2022) [19] Nguyen, T.T., Huynh, T.T., Ren, Z., Nguyen, P.L., Liew, A.W.C., Yin, H., Nguyen, Q.V.H.: A survey of machine unlearning. ACM Trans. Intell. Syst. Technol. 16(5) (2025) [20] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4510–4520 (2018) [21] Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014) [22] Statistics Canada: Notice of release of the north american industry classification system (naics) canada 2022 version 1.0. https://www.statcan.gc.ca/ en/subjects/standard/naics/2022/v1/notice (2022) [23] Suriyakumar, V., Wilson, A.C.: Algorithms that approximate data removal: New results and limitations. Advances in Neural Information Processing Systems 35, 18892–18903 (2022) [24] Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning (ICML). pp. 1139–1147 (2013) [25] Tarun, A.K., Chundawat, V.S., Mandal, M., Kankanhalli, M.: Fast yet effective machine unlearning. IEEE transactions on neural networks and learning systems 35(9), 13046–13055 (2023) [26] Wang, J., Bie, H., Jing, Z., Zhi, Y.: Feature-indistinguishable machine unlearning via negative-hot label encoding and class weight masking. Scientific Reports (2026) [27] Warnecke, A., Pirch, L., Wressnegger, C., Rieck, K.: Machine unlearning of features and labels (2023), https://arxiv.org/abs/2108.11577 [28] Wu, G., Hashemi, M., Srinivasa, C.: Puma: Performance unchanged model augmentation for training data removal. In: Proceedings of the AAAI conference on artificial intelligence. vol. 36, pp. 8675–8682 (2022) [29] Yan, H., Li, X., Guo, Z., Li, H., Li, F., Lin, X.: ARCANE: An efficient architecture for exact machine unlearning. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI). pp. 4006–4013 (2022) [30] Zhang, Y., Hu, Z., Bai, Y., Wu, J., Wang, Q., Feng, F.: Recommendation unlearning via influence function. ACM Transactions on Recommender Systems 3(2), 1–23 (2024) [31] Zhao, B., Cui, Q., Song, R., Qiu, Y., Liang, J.: Decoupled knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11953–11962 (2022) [32] Zhou, D.W., Wang, Q.W., Qi, Z.H., Ye, H.J., Zhan, D.C., Liu, Z.: Classincremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

Related documents

• Privacy Leakage via Output Label Space and Differentially Private Continual Learning#163068
• Information Theoretic Adversarial Training of Large Language Models#160754
• Learning from Noisy Label Distributions#478222
• Adaptive Regression by Mixing#168009
• Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints#665644