One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

arXiv:2608.31096v1 [cs.CV] 31 Aug 2026

Yunxiang Fu Meng Lou Yizhou Yu School of Computing and Data Science The University of Hong Kong

Abstract Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while preserving the ability to recognize all seen classes. Recently, pretrained-model-based approaches have become prevalent by adapting a frozen backbone with additional lightweight trainable modules. Existing methods, however, exhibit limitations: task-specific adapters learn explicit per-task representations but are parameter- and computationinefficient, while LoRA-based merging methods combine per-task LoRA parameters into a single model whose static aggregated weights cause representation interference during inference. To address these problems, we present FACET: task-conditioned FeAture transformation with ConditionEd feature consisTency, achieving excellent parameter efficiency while producing highly discriminative features during inference. When continually trained on a task sequence, FACET learns a single shared adapter that employs a dynamic task-conditioned feature transformation, shaping the overall feature distribution of the adapter into a mixture of overlap-reduced task-specific components. On the other hand, we propose an efficient replay-free task-conditioned feature consistency loss, aiming to mitigate catastrophic forgetting of the learned mixture distribution in the adapter’s feature space. Even when maintaining only a single adapter, FACET demonstrates robust scalability. On both very long task sequences (e.g., 200 tasks) and standard short task sequences (e.g., 20 tasks), our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs. The code will be made open source upon acceptance.

1

Introduction

In real-world applications, training data may arrive sequentially from continuously changing environments, which requires a learning system to acquire new concepts while retaining previous knowledge [15, 25]. However, standard deep neural networks generally suffer from catastrophic forgetting: when trained on new data, their performance on previously learned tasks can severely degrade [17, 13, 27]. Continual learning (CL) aims to address this problem by enabling models to learn from continuous data streams without forgetting past knowledge. Among the many CL settings, class-incremental learning (CIL) is one of the most challenging because the model must learn new classes over time and, at test time, discriminate among all classes seen so far [23]. Traditional continual learning methods address catastrophic forgetting through regularization [13, 17], replay [26, 19],architectural expansion [27, 38], and parameter isolation/sub-networks [2, 21]. More recently, pretrained-model (PTM) based approaches have become prevalent for CIL, as large pretrained backbones provide general-purpose, powerful representations, and require relatively few additional parameters when adapted to new tasks [35, 28, 42, 43, 33, 30, 12]. A common strategy attaches task-specific learnable adapters to a frozen vision backbone [42, 33, 12, 32], which produces explicit task-specific feature perturbations but requires a separate adapter per task, resulting in Preprint.

Figure 1: Comparison of feature distributions. (a) t-SNE of block 1 features, colored based on task. FACET shows well-separated task-specific distributions, while SD-LoRA’s are mixed. (b) Class-averaged silhouette score (higher values mean better intra-class compactness and inter-class separation) across blocks. FACET achieves higher scores than representative LoRA-based (SD-LoRA) and per-task adapter based (TUNA) methods, producing more class-discriminative features that are beneficial for classification. parameter and computation inefficiency. Another popular solution trains separate LoRA modules [11] by task and merges all learned modules before inference, exemplified by InfLoRA [18] and SDLoRA [36]. Although such methods improve parameter efficiency at inference time, an input is passed to the aggregated weights, which give rise to representation interference and diluted features that are suboptimal for classification. To address these issues, we present FACET, a novel PTM-based CIL method that achieves parameter efficiency by using a single continually optimized adapter with an explicit task-conditioning mechanism. The core idea is to obtain task-dependent features by applying a dynamic task-conditioned transformation to the adapter’s output. Specifically, we predict a task-dependent context vector for the input, and map it to a matrix that performs task-dependent channel-wise scaling and cross-channel mixing in the latent space of the adapter. This substantially improves the separability of task-specific feature distributions in the adapter’s feature space, thereby reducing representation interference. However, training the shared adapter on a new task might distort the conditional feature distributions of previous tasks and make them less discriminative. To this end, the adapter is required to preserve the conditional feature distribution of any earlier task. That is, given any training sample from the new task, the adapter should be able to reproduce its feature representation as if conditioned on any earlier task. This goal is achieved via a simple task-conditioned feature consistency objective. We compare the representation quality of FACET with representative methods in Fig. 1. In early adapter blocks, FACET’s task-specific feature distributions are widely separated, while those of SDLoRA are completely mixed (Fig. 1 (a)). As CIL ultimately requires discrimination among classes, Fig. 1 (b) compares the block-wise class-averaged Silhouette score 1 . A higher score suggests more class-discriminative features. Driven by dynamic task conditioning, FACET achieves a significantly higher score in the last block, which indicates that features have transitioned from task-scaffolded to class-discriminative as depth increases. We evaluate FACET on standard class-incremental learning benchmarks. FACET achieves the highest final accuracy on ImageNet-R, ImageNet-A, CIFAR-100, and ObjectNet. Compared with SD-LoRA [36], FACET improves the final accuracy from 75.26% to 79.97% on ImageNet-R with 20 tasks and from 55.96% to 64.87% on ImageNet-A. Surprisingly, on OmniBenchmark-1K [20] with 200 tasks, FACET significantly improves over MIN [12] and TUNA [33] by 3.45% and 6.81% in final accuracy, respectively. FACET is also 3.85× faster and reduces GFLOPs by 62.69% relative to MIN (Table 3). Note that MIN and TUNA learn a task-specific adapter for every task. Our main contributions are as follows: A novel single-adapter design with task-conditioned transformations. We introduce a single, continually trained adapter that incorporates a dynamic task-conditioned transformation. This design achieves parameter efficiency and improves the separability of task-specific feature distributions, thus reducing representation interference. 1 The Silhouette score of a class is (b − a)/ max(a, b), where a is the mean pairwise feature distance within the class and

b is the mean pairwise distance between any feature in the class and any feature in the nearest different class.

2

Figure 2: Illustration of FACET. (a) Task-conditioned feature transformation within a single adapter. We predict the task context c and use it to generate a transformation matrix Mc to obtain taskdependent features. Activation functions are not shown for simplicity. (b) Task-conditioned feature consistency objective when training on a new task t with dataset Dt . Conditioned feature consistency. We propose a conditioned feature consistency loss that aligns the feature representation of the latest adapter with its frozen snapshot under earlier task conditions, effectively mitigating forgetting. State-of-the-art results. FACET achieves new state-of-the-art experimental results on class-incremental learning benchmarks with 10 to 200 tasks.

2

Related Work

Pre-trained model-based Continual Learning. Early continual learning methods combat catastrophic forgetting through regularization [13, 39], knowledge distillation [17], replay [26, 19], parameter isolation/sub-networks [2, 21, 24, 1], and architectural expansion [27, 38, 37]. Based on the powerful generalization ability of pretrained models (PTMs), recent class-incremental learning (CIL) methods attach a small number of learnable parameters for each task. For example, Prompt-tuning approaches such as L2P [35], DualPrompt [34], and CODA-Prompt [28] store a pool of task-specific tokens and retrieve them at inference. Recent methods provide stronger learning capacity by incrementally appending task-specific adapters, i.e., EASE [42] builds task-specific subspaces by incrementally tuning adapters: TUNA [33], MOS [30], and MIN [12] assign a dedicated adapter to each task to prevent forgetting. However, these methods are parameter inefficient and computationally heavy as they must activate or aggregate all stored task-specific adapters at test time. In contrast, we use a single shared adapter with a dynamic task-conditioned feature transformation, producing separable task-specific feature distributions without maintaining per-task adapters. Parameter Efficient CIL. Some works strive for a single set of parameters in PTM-based CIL. SLCA [40] and FeCAM [6] focus on classifier alignment and covariance-aware classifiers, respectively. Closer to our design, LoRA-based merging methods (InfLoRA [18], SD-LoRA [36]) and model merging techniques (MagMax [22], OPCM [31]) first train separate LoRA/adapters per task and then combine them into a single model during inference. These methods apply a static merged model during inference, where the aggregated weights may cause representation interference and dilute the task-specificity of feature maps. In contrast, our method directly trains a single adapter for all tasks with explicit task conditioning, eliminating such interference without per-task adapters.

3

Method

3.1

Problem Formulation

In class-incremental learning, a model learns from a data stream D = {D1 , D2 , . . . , DT } consisting t of T tasks arriving sequentially. Each task t provides a training set Dt = {(xti , yit )}ni=1 , where xti t is an image, yi ∈ Yt is its class label, and nt is the number of training samples. The label sets of different tasks are disjoint: Ys ∩ Yt = ∅ for all s ̸= t. We follow the exemplar-free setting [42, 33]. 3

When task t is learned, only the latest training data Dt are accessible. We denote Ynew = Yt as the St−1 set of new classes and Yold = s=1 Ys as the set of previously seen classes. After being trained on task t, the model is evaluated on all classes seen so far, Yold ∪ Ynew . We decompose the model into a pretrained ViT backbone F, an adapter with parameters θ [4] inserted in parallel to the FFN in every transformer block, and a classifier Wt that grows by appending a new class vector for each task. The backbone F is frozen throughout training. Only θ and Wt are updated. The goal is to learn all T tasks from the data stream and correctly classify any test sample ST into one of the classes in t=1 Yt without access to past data. 3.2

Single Adapter with Dynamic Task-Conditioned Transformations

As shown in Fig. 2 (a), a frozen pretrained Vision Transformer (ViT) [5] backbone F is equipped with adapter modules placed in parallel to the feed-forward network (FFN) in every transformer block. For an input x to the FFN, the output of the block o is o = FFN(x) + Adapter(x, c) + x,

(1)

where c denotes a context vector that reflects the task-related information of x. By conditioning the adapter’s output explicitly on c, we obtain task-dependent features during inference while using only a single shared set of adapter weights for all tasks, which is both parameter- and computation-efficient. The hidden layer of a standard adapter [4] operates in a latent space, which often has much lower dimensionality than the ViT backbone. To increase learning capacity without substantially increasing trainable parameters, we extend the standard adapter with one additional non-linear hidden layer. The revised adapter first computes a latent representation z ∈ Rr as follows,
z = σ σ(xWdown ) Wmid , Wdown ∈ Rd×r , Wmid ∈ Rr×r , (2) where σ is the GELU [7] activation, d is the ViT’s feature dimensionality, and r < d is the latent dimensionality of the adapter. This is common computation for all tasks. Next, before projecting the latent representation back to the ViT feature space, we apply a task-conditioned linear transformation to z, zc = zMc , (3) r×r where Mc ∈ R is a transformation matrix dynamically generated from the context vector c and zc is the resulting task-dependent latent representation. Context vectors related to different tasks give rise to different matrix transformations. As a result, the same latent representation z can be transformed to different task-dependent latent features. The adapter output is obtained by projecting zc up to the ViT feature space through the shared up-projection Wup ∈ Rr×d : Adapter(x, c) = (zc )Wup .

(4)

Since the up-projection is linear, applying the task-conditioned transformation in the latent space is sufficient to make the final adapter output task-dependent, while all adapter weights remain shared across tasks. Let us now discuss more details about context vectors and task-conditioned transformations. The context vector reflects information on the task identity of the given input. A simple option is to set it to be the task identity itself, i.e., c ∈ {0, 1}Kmax , where Kmax is a sufficiently large upper bound on the total number of tasks (ablation of Kmax is given in Appendix Table 11). During training, the task identity is known by counting the number of trained tasks at present, while lightweight task-wise classifiers predict the context vector during inference (Sec. 3.4). Given a context vector, we construct the task-conditioned transformation Mc in a low-rank decomposition form to maintain parameter efficiency. Mc = Diag(s) + U V ⊤ ,

(5)

where s ∈ Rr determines task-conditioned scaling, which independently rescales each feature dimension in a task-dependent manner, and U, V ∈ Rr×k with k ≪ r implement cross-dimension mixing, which redistributes activations across dimensions so that different tasks produce different linear combinations of the latent feature dimensions. The above matrix components are directly mapped from the context vector via linear projections, s = Ws c,

U = reshape(WU c), 4

V = reshape(WV c),

where Ws ∈ Rr×Kmax and WU , WV ∈ Rrk×Kmax are learnable projection matrices, and reshape(·) converts a vector of length rk into a r × k matrix. Thus, a single shared adapter, without any task-specific weights, produces task-conditioned feature transformations and, in turn, task-dependent final representations entirely on the basis of the context vector. This is the core difference between our method and other LoRA-based merging methods. To improve training stability, we initialize Ws , WU , WV to zero so that the adapter output is zero during the first training step. We also add small Gaussian noise to the context vector during training, aiming to improve robustness against imperfect task identity predictions during inference and softly enable backward knowledge transfer during training. Remark As a single adapter is shared among all learned tasks and each task has its own feature distribution, the overall distribution of the adapter’s final output features can be viewed as a mixture of task-specific components. Task-conditioned feature transformations are capable of reducing overlap among these task-specific components by altering their location and shape. Our experiments confirm that introducing task-conditioned transformations better separates task-specific components (Fig. 1). We also considered adding explicit losses to improve the separability of the mixture components, such as maximizing their cosine distance or maximizing the orthogonality of task-conditioned transformations. Nevertheless, this reduces the learning capacity of the adapter and leads to suboptimal results, as shown in our ablation studies (Table 5). Using context signals that reflect task identity as conditioning has been previously studied in continual learning. For instance, using them as learnable context tokens concatenated to the input of the model [35, 34, 28], as a gating mechanism to select subnetworks [2, 1, 24], or to route inputs to task-specific adapters [33, 30]. Distinct from previous works, FACET applies context signals to transform the adapter’s latent representation, making task-specific feature distributions more separable, which is a direction that has not been explored. 3.3

Task-Conditioned Feature Consistency

Since the adapter weights are tuned on every new task, there is an increasing risk of gradually shifting the adapter’s latent feature distribution. This may lead to forgetting of the learned mixture distribution because the task-specific mixture components are dependent on the adapter weights. To this end, we introduce a simple task-conditioned feature consistency objective that preserves the learned task-specific mixture components without resorting to any historical data. This allows our adapter to produce reliable task-dependent features for continual learning. Specifically, after learning task t − 1, we make a copy of the latest adapter weights, θ− ← θ, and keep the copy frozen when tuning the adapter on task t. For each mini-batch, we uniformly sample the index of a previous task j ∼ Uniform{1, . . . , t − 1} and construct the corresponding context vector cj ∈ RKmax . Then the training samples for task t pass through the following two paths: Main path: a training sample x ∈ Dt is passed to the ViT backbone equipped with the adapter (θ) currently being trained on task t and use cj as the context vector. The [CLS] token from the last transformer block is the main feature, and is denoted as CLSθj Reference path: x is passed to the ViT backbone equipped with the frozen adapter snapshot (θ− ) and use cj as the context vector again. The [CLS] token from the last transformer block is taken as − the reference feature, and is denoted as CLSθj Evaluating both paths on the same input under the same task conditioning enables us to directly measure how much the latest adapter has drifted from the learned feature distribution of task j. This does not require x to be a sample from task j as the context vector cj drives the adapter to process x as a sample from task j regardless of its true task identity. Our experiments demonstrate that this simple strategy can preserve the feature distribution of historical tasks without using their original data (Appendix D) and mitigates forgetting substantially (Table 4). The training loss is the squared Euclidean distance between conditioned [CLS] tokens averaged uniformly over all previous tasks:   2 − Lcon = Ej∼Uniform{1,...,t−1} Ex∼Dt CLSθj − CLSθj . (6) 2

5

In practice, each mini-batch draws a single pair (j, x), yielding an unbiased stochastic estimate of this expectation and providing uniform coverage of all earlier tasks over the course of training. We sample only one previous task per iteration rather than considering all previous tasks simultaneously because the latter would over-constrain θ and reduce the adapter’s capacity to learn the latest task. The overall training loss is L = Lcls + λcon Lcon , (7) where Lcls is a weighted cross-entropy loss combined with classifier alignment as in [40, 33, 30] (applied to learn a new conditional feature distribution for task t), and λcon = 2 in our experiments. Note that our task-conditioned feature consistency differs from standard distillation methods such as LwF [17], which aligns global predictions or features [10, 16], but cannot preserve the task-specific feature distributions that are central to our design. 3.4

Oracle-Free Inference

Generating task-dependent features from FACET at each block requires a task context vector c. We therefore introduce a straightforward strategy to estimate c during inference. Specifically, we attach a linear classifier to the output of the attention layer within each transformer block (Fig. 2). This auxiliary classifier receives the same supervision alongside the final classification head during the continual training phase. During inference, this classifier outputs a probability distribution P pt for all classes Yt for all learned tasks t ∈ {1, ..., T }. We compute the entropy Ht = − y∈Yt pt (y) log pt (y) for each task and select the task context with the highest prediction certainty (lowest entropy) t̂ = arg mint Ht . The final classifier predicts from the last block’s P[CLS] feature. Notably, thePauxiliary classifiers contain task-specific parameters with a total of (d × t |Yt |) parameters, where t |Yt | is the total number of classes in the dataset. Table 3 shows that FACET significantly reduces the task-specific costs compared to existing task-specific adapter methods. Section 4.4 compares alternative context-prediction strategies.

4

Experiments

We evaluate FACET on standard CIL benchmarks covering short to very long task sequences. Despite employing a single adapter, FACET outperforms strong baselines, especially on long task sequences. Ablation studies validate the effectiveness of all components. 4.1

Experimental Setting

Datasets. We benchmark on four widely used datasets [35, 28, 42, 18, 12], including CIFAR100 [14], ImageNet-R [8], ImageNet-A [9], and ObjectNet [3], plus long task sequences with up to 200 tasks in a recent benchmark, OmniBenchmark-1K [20] . Together, these benchmarks cover short, medium-length, and long task sequences in standard and distribution-shifted visual domains. Baselines. We compare FACET against state-of-the-art prompt-based and adapter-based PTM CIL methods with task-specific parameters (L2P [35], DualPrompt [34], CODA-Prompt [28], EASE [42], APER [43], TUNA [33], MOS [30], and MIN [12]), classifier-based methods (SLCA [40], FeCAM [6]), and LoRA-based weight-merging methods (InfLoRA [18], SD-LoRA [36], MagMax [22], and OPCM [31]). Implementation Details. We run all our experiments on a single NVIDIA L40S GPU. Following prior work [42, 43, 33, 12, 30], we employ the representative pre-trained vision backbone ViT-B/16-IN21K [5]. FACET adds a single adapter per transformer block (latent dimension 384, hidden depth 2) with k = 8 and Kmax = 200 for the task-conditioned transformation matrix Mc . For training, we use SGD (momentum 0.9, weight decay 5 × 10−4 ), cosine learning-rate decay with base learning rate 0.02 and batch size 16, and conditional feature consistency weight λcon = 2.0. The classification head follows [42, 33]. More details about datasets and settings are in the Appendix. Evaluation Protocol and Performance Metrics. We follow standard settings [35, 33, 12] to partition a dataset into T tasks with the learning order determined by the random seed 1993. We denote Inc n as each task having n classes. For FACET, we report the average performance over 3 runs. We report 6

Table 1: Performance comparison with state-of-the-art methods on standard benchmarks. Method

ImageNet-R 10 Tasks (Inc20) 20 Tasks (Inc10) Ā AT Ā AT

CIFAR-100 10 Tasks (Inc10) 20 Tasks (Inc5) Ā AT Ā AT

ImageNet-A 10 Tasks (Inc20) Ā AT

ObjectNet 10 Tasks (Inc20) Ā AT

L2P(CVPR’22) DualPrompt(ECCV’22) CODA-Prompt(CVPR’23) EASE(CVPR’24) SEMA(CVPR’25) MOS(AAAI’25) TUNA(ICCV’25) MIN(NeurIPS’25) SLCA(ICCV’23) FeCAM(NeurIPS’23) APER-Adapter(IJCV’25) MagMax-LoRA(ECCV’24) OPCM-LoRA(NeurIPS’25) InfLoRA(CVPR’24) SD-LoRA(ICLR’25) FACET (Ours)

75.46 73.10 77.97 81.74 81.39 81.74 84.22 85.18 81.17 79.02 75.82 84.59 84.37 80.82 82.04 85.41

85.92 89.65 91.05 92.11 91.60 93.83 94.85 95.12 92.67 93.23 92.22 94.76 94.90 91.70 92.54 95.18

49.39 53.71 53.54 65.34 63.83 69.13 72.96 72.89 68.66 56.04 60.53 66.64 69.67 58.50 64.95 73.31

66.77 64.31 66.53 71.04 67.95 74.75 76.46 72.56 74.12 68.68 69.24 70.85 71.99 70.67 70.37 77.13

69.77 67.18 72.27 76.17 77.84 76.17 79.42 79.75 77.00 72.53 67.95 77.68 79.38 75.65 77.34 80.66

63.75 66.52 70.45 81.18 77.84 82.96 82.54 83.69 81.85 77.84 72.35 81.8 84.08 77.28 80.22 84.73

55.78 61.77 64.68 74.62 69.60 77.93 77.45 78.08 76.63 71.05 64.33 73.40 77.62 71.01 75.26 79.97

79.19 84.89 86.44 87.72 86.75 90.19 91.71 92.12 89.30 89.05 87.45 91.41 91.92 86.51 88.01 92.27

85.94 87.87 89.11 91.51 92.23 93.30 94.44 94.31 93.52 91.86 90.65 94.31 94.60 89.13 90.90 94.62

79.93 81.15 81.96 85.80 87.84 89.25 90.74 91.03 90.07 87.04 85.15 90.94 91.18 81.46 85.18 91.31

41.71 41.67 42.73 55.04 52.21 59.12 64.38 64.32 58.74 46.41 49.57 54.64 59.78 46.28 55.96 64.87

55.16 52.99 56.80 59.37 54.92 65.10 66.32 61.36 63.23 57.39 57.41 59.15 60.28 58.04 58.54 67.38

Table 2: Performance comparison on benchmarks with long task sequences. InfLoRA and SD-LoRA failed to converge on the 200 task setting. Method

ImageNet-R 50 Tasks (Inc4) Ā AT

ImageNet-A 50 Tasks (Inc4) Ā AT

OmniBenchmark-1K 100 Tasks (Inc10) 200 Tasks (Inc5) Ā AT Ā AT

L2P(CVPR’22) DualPrompt(ECCV’22) CODA-Prompt(CVPR’23) EASE(CVPR’24) SEMA(CVPR’25) MOS(AAAI’25) TUNA(ICCV’25) MIN(NeurIPS’25) APER-Adapter(IJCV’25) MagMax-LoRA(ECCV’24) OPCM-LoRA(NeurIPS’25) InfLoRa(CVPR’24) SD-LoRA(ICLR’25) FACET (Ours)

69.16 64.00 62.43 78.11 67.80 75.15 80.93 82.26 72.43 81.87 82.49 71.68 68.40 82.70

49.89 43.85 38.24 59.86 52.99 63.72 67.82 66.15 61.29 65.40 66.58 50.17 54.72 69.16

60.91 62.18 64.16 65.00 56.55 76.80 71.93 75.46 73.23 69.93 60.19 51.53 53.97 76.88

63.45 56.33 57.57 70.63 59.32 66.80 75.48 75.72 64.83 74.97 73.70 62.62 63.28 76.63

36.41 29.95 26.60 47.53 40.68 51.22 58.72 57.08 48.58 52.58 54.02 38.70 41.28 59.58

48.87 49.45 51.75 53.54 33.96 64.27 60.04 63.60 62.24 41.85 33.73 27.01 28.15 66.17

57.53 59.13 60.22 67.23 45.06 75.92 71.45 74.94 72.49 61.30 58.32 77.30

45.25 45.62 47.56 55.13 19.95 63.51 59.14 62.50 61.53 30.27 28.43 65.95

PT the average incremental accuracy Ā = T1 t=1 At and final accuracy AT , where At is the average accuracy over all classes seen so far after the adapter has been trained on task t. Due to space limit, results on the ViT-B/16-IN1k backbone and other random seeds are provided in the Appendix B.1. 4.2

Main Results

Tables 1 and 2 compare FACET with SOTA PTM-based CIL methods. Our experimental results clearly demonstrate two key advantages. First, our single adapter yields superior performance on standard benchmarks in comparison to computationally heavy methods based on a set of task-specific adapters. Second, this design surprisingly achieves leading performance on long task sequences of up to 200 tasks when compared to other strong baselines. More details are provided below. Comparison with State-of-the-Art Methods. Table 1 shows that FACET achieves the highest final accuracy AT on all four datasets that have 10 to 20 tasks. For instance, on ImageNet-R with 20 tasks, FACET improves AT from 78.08% to 79.97% compared to MIN, one of the strongest baselines that stores one adapter-like module for every task. On the more challenging ImageNet-A benchmark, where adversarially filtered images make class discrimination more difficult, the strongest baselines with task-specific adapters, TUNA and MIN, substantially outperform LoRA-based merging methods, showing that explicit task-specific representations are important. Nevertheless, FACET achieves the highest AT (64.87%) and competitive Ā (73.31%) with a single adapter, improving over the strongest LoRA-based merging method by 3.64% in Ā and 5.09% in AT . OPCM-LoRA remains competitive on ImageNet-R and CIFAR-100, but FACET still improves AT by 2.35% on ImageNet-R with 20 tasks. Scaling to Long Task Sequences. Table 2 evaluates whether FACET remains effective as the number of tasks in a sequence increases to 100 or even 200, when storing, searching over, or merging many task-specific modules becomes increasingly difficult. On ImageNet-R with 50 tasks, FACET achieves the highest Ā and AT . It improves AT from 75.72% to 76.63% compared to MIN, and by an absolute 7

Table 3: Comparison of parameter, computation, and GFLOPs among SOTA methods. Latency is the median latency at batch size 32. ImageNet-R (50 tasks)

OmniBenchmark-1K (200 tasks)

Method

Params(M)

Latency(ms)

GFLOPs

Ā

AT

Method

Params(M)

Latency(ms)

GFLOPs

Ā

Ours MoS TUNA MiN

10.20 15.37 15.37 135.15

28.61 1431.30 1308.28 44.02

37.37 1798.57 1798.05 55.39

82.70 76.63 75.15 66.80 80.93 75.48 82.26 75.72

Ours MoS TUNA MiN

22.88 61.63 61.63 1831.59

36.64 5717.77 6306.79 141.42

40.23 7088.81 7086.94 107.84

77.30 65.95 75.92 63.51 71.45 59.14 74.94 62.50

AT

2.93% compared to OPCM-LoRA, the strongest LoRA-based merging method. On ImageNet-A with 50 tasks, FACET again achieves the highest Ā and AT , improving over TUNA by 1.34% in Ā and 0.86% in AT . This is significant because ImageNet-A is the setting where LoRA-based merging methods struggle the most: FACET improves AT over OPCM-LoRA by 5.56% while preserving the single-adapter design. The scaling advantage becomes clearer for 100 and 200 tasks. On OmniBenchmark-1K with 100 tasks, FACET improves over the strongest baseline by 1.90% in AT , while achieving the highest Ā of 76.88. When the number of tasks further increases to 200, FACET remains the top performer among all baselines, improving Ā from 75.92% to 77.30% and AT from 63.51% to 65.95% over MOS. These results show that FACET is capable of scaling to a large number of tasks while maintaining a single adapter, and task-conditioned feature transformations remain effective even when the number of tasks reaches 200. 4.3

Parameter, Computation and GFLOPs

Table 3 compares our method with the best-performing baselines (i.e., MIN, MOS, and TUNA) in trainable parameter count, inference latency, and GFLOPs per image on long task sequences. Our method is among the most efficient and achieves a 156× speedup in inference latency and only use 0.57% GFLOPs compared to MoS when there are 200 tasks. Furthermore, FACET requires only 1.25% of the cumulative trainable parameters of MIN. These findings demonstrate that our proposed method maintains leading performance while delivering better parameter efficiency and substantially lower latency than per-task adapter methods such as TUNA/MOS. 4.4

Ablation Studies

We extensively verify the effectiveness of all components in FACET with ablation studies conducted on the ImageNet-R Inc10 (20 tasks) dataset, which provides a challenging setting. The baseline is our Table 4: Incremental component ablation on ImageNet-R (Inc10). full FACET method presented in Table 1. Variant Mc cond Consistency Other designs Ā AT ×

Plain Single Adapter

×

×

57.75

48.75

+ Mc Transformation ✓ × × 70.06 60.02 Individual contribu+ Cond. Feat. Consis. ✓ ✓ × 83.90 78.23 tions of components. FACET (Full) ✓ ✓ ✓ 84.70 79.97 Table 4 decomposes the Feat. Consis. Only × ✓ ✓ 72.94 65.39 gains of FACET. A plain single adapter reaches only 48.75% final accuracy, confirming that simply sharing weights across tasks is insufficient. Adding task-conditioned feature transformations (Mc transformation) improves final accuracy to 60.02%, demonstrating that explicit task conditioning and conditioned transformations yield a substantial benefit by making task-specific feature distributions more separable. The performance jump to 78.23% after adding task-conditioned feature consistency, showing that once FACET creates separable mixture components, they are fragile to forgetting and must be explicitly preserved. Note that non-conditioned feature consistency alone achieves an AT of 65.39%, indicating that the interplay between conditioned feature transformation and consistency is crucial. Finally, other designs such as adding gaussian noise during training together contributes a further improvement to 79.97%.

Variants of conditioned feature transformation. Table 5 studies alternative forms of taskconditioned transformation and the effect of an explicit separability loss (Sec. 3.2), reporting Ā, AT , and the silhouette scores computed from last-block features as a measure of class discriminability. 8

Transforming the adapter input instead of Table 5: Ablation on conditioned feature transformation its output causes a sharp drop of AT to and separability loss. 69.22% and a negative silhouette score, Form Ā AT Silhouette suggesting that intra-class distances exNo transformation 72.49 65.39 -0.11 ceed inter-class distances. This confirms Concat c to adapter input 69.46 66.42 -0.086 that conditioning the adapter output via Mc at adapter input 72.27 69.22 -0.057 Mc = U V ⊤ 84.50 78.58 0.026 Eq. (3) is essential. Diagonal-only and Mc = Diag(s) 84.46 79.60 0.040 low-rank-only variants of Mc , already raise Mc = Diag(s)+U V ⊤ 84.70 79.97 0.042 performance clearly above the plain sin+Orthogonal M loss 82.19 78.63 0.035 c gle adapter, showing that the primary gain +zc cos distance loss 81.83 78.02 0.014 comes from explicit task conditioning. Finally, adding explicit losses to further separate task-specific feature distributions hurts performance since it reduces the learning plasticity for new tasks. Design of conditioned feature consistency. Table 6 studies how to best preserve previously learned task- Table 6: Ablation on conditioned feature specific feature distributions. Only enforcing consistency consistency. for the most recent task under-protects earlier task-specific Strategy Ā AT distributions (AT = 77.03%), while considering all previMost recent (t − 1) 83.31 77.03 ous tasks simultaneously over-constrains the adapter and All tasks simultaneous 84.01 78.37 limits learning plasticity (AT = 78.37%). Uniformly samUniform, EMA teacher 83.96 78.85 Uniform, snapshot (ours) 84.70 79.97 pling one previous task per mini-batch achieves the best trade-off, covering all prior task-specific distributions withAll features (Last Block) 80.56 70.70 out hindering new task learning. A hard adapter snapshot CLS in every block 84.35 79.80 outperforms an EMA teacher, indicating that the exact Cosine Distance 84.53 79.46 post-training adapter state is a more reliable reference. Alternative designs in computing the loss lead to suboptimal results, such as using CLS tokens from every block (rather than just the last block), all tokens from the last block, and using cosine distance (1-cosine similarity) in Eq. 6 instead of MSE. Task Context Prediction. Table 7 investigates alternative methods to predict task context during inference. Table 7: Ablation on context prediction. Assembling the context vector using the highest logit in Predictor Ā AT each task from the final classification head results in subFinal classifier highest logits 84.07 79.33 optimal performance, while a context with the highest Aux. classifier highest logits 84.19 78.63 logits from auxiliary classifiers sees a 1.34% drop in AT , Entropy as task context 84.23 77.85 Min entropy 84.70 79.97 demonstrating the advantage of entropy for task identity Oracle 81.62 prediction. Directly filling the context using the entropy value for each task leads to a larger decline of AT to 77.85%, confirming that the use of sparse context vectors is beneficial. Notably, using ground truth task context during inference improves final accuracy to 81.62%, implying that FACET can benefit from more accurate task context predictors for short task sequences. Additional analysis on task context prediction and robustness of FACET to task context prediction errors are given in the Appendix.

5

Limitations

FACET is designed for exemplar-free class-incremental learning with a frozen pretrained vision backbone, disjoint task label sets, and known task boundaries during training. These assumptions make the method directly comparable to PTM-based CIL methods, but they also leave several directions open. While FACET substantially reduces the task-specific costs compared to state of the art methods for long task sequences with 200 tasks, task-specific cost is kept small in the auxiliary classifiers but does not fully disappear. Moreover, the single adapter capacity may saturate. Although we achieve strong performance up to 200 tasks, longer task sequences or highly heterogeneous tasks may eventually exceed what one adapter can represent. Extending our method to true lifelong continual learning over thousands of tasks is non-trivial. In addition, the replay-free objective preserves old task-specific feature distributions by matching a frozen snapshot on current-task inputs, 9

which is efficient and avoids storing old data. However, it is not equivalent to matching the full old data distribution and may be weaker when the data distribution of the new task is highly different from earlier tasks. Promising directions include extending the method to large language models and studying alternative task context prediction methods.

6

Conclusion

This work has introduced FACET, a single-adapter approach for pretrained-model-based classincremental learning. By transforming the adapter’s latent representation in a task-conditioned manner while preserving previously learned task-specific feature distributions, FACET makes continual learning more efficient and scalable. Future work could extend the proposed method’s principles of sharing weights across tasks with dynamic conditioned transformation to extract task-specific features to other domains, such as large language models, vision-language models, and other scenarios where efficient continual learning is useful.

References [1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3931–3940, 2020. [2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3366–3375, 2017. [3] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019. [4] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664–16678, 2022. [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [6] Dipam Goswami, Yuyang Liu, Bartłomiej Twardowski, and Joost Van De Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. Advances in Neural Information Processing Systems, 36:6582–6595, 2023. [7] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [8] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340–8349, 2021. [9] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262–15271, 2021. [10] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1921–1930, 2019. [11] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022. 10

[12] Kai Jiang, Zhengyan Shi, Dell Zhang, Hongyuan Zhang, and Xuelong Li. Mixture of noise for pre-trained model-based class-incremental learning. Advances in neural information processing systems, 39, 2025. [13] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. [14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.(2009), 2009. [15] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information fusion, 58:52–68, 2020. [16] Songze Li, Tonghua Su, Xu-Yao Zhang, and Zhongjie Wang. Continual learning with knowledge distillation: A survey. IEEE Transactions on Neural Networks and Learning Systems, 36(6): 9798–9818, 2024. [17] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017. [18] Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024. [19] David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017. [20] Meng Lou, Yunxiang Fu, and Yizhou Yu. Scaling continual learning with bi-level routing mixture-of-experts. arXiv preprint arXiv:2602.03473, 2026. [21] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018. [22] Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. Magmax: Leveraging model merging for seamless continual learning. In European Conference on Computer Vision, pages 379–395. Springer, 2024. [23] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5): 5513–5533, 2022. [24] Nicolas Y Masse, Gregory D Grant, and David J Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44):E10467–E10475, 2018. [25] Yuan Meng, Zhenshan Bing, Xiangtong Yao, Kejia Chen, Kai Huang, Yang Gao, Fuchun Sun, and Alois Knoll. Preserving and combining knowledge in robotic lifelong reinforcement learning. Nature Machine Intelligence, 7(2):256–269, 2025. [26] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017. [27] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. 11

[28] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11909–11919, 2023. [29] Hai-Long Sun, Da-Wei Zhou, De-Chuan Zhan, and Han-Jia Ye. Pilot: A pre-trained modelbased continual learning toolbox. SCIENCE CHINA Information Sciences, 68(4):147101, 2025. [30] Hai-Long Sun, Da-Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model surgery for pre-trained model-based class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 20699–20707, 2025. [31] An Quang Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. Advances in neural information processing systems, 39, 2025. [32] Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. Self-expansion of pre-trained models with mixture of adapters for continual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10087–10098, 2025. [33] Yan Wang, Da-Wei Zhou, and Han-Jia Ye. Integrating task-specific and universal adapters for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 806–816, 2025. [34] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference on computer vision, pages 631–648. Springer, 2022. [35] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149, 2022. [36] Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. In International Conference on Learning Representations, 2025. [37] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3014–3023, 2021. [38] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017. [39] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. Pmlr, 2017. [40] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19148–19158, 2023. [41] Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In European conference on computer vision, pages 594–611. Springer, 2022. [42] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554–23564, 2024. [43] Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting classincremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 133(3):1012–1032, 2025. 12

A

Additional Dataset and Implementation Details

A.1

Datasets

We evaluate FACET on five datasets, including ImageNet-R [8], ImageNet-A [9], CIFAR100 [14], ObjectNet [3, 43] , and OmniBenchmark-1K [41, 20]. The datasets used in the main comparison are summarized below: ImageNet-R [8] has 200 classes with 30000 images. It consists of ImageNet-compatible classes rendered in artistic and non-standard visual styles. ImageNet-A [9] contains 200 classes with 7475 images. The dataset consists of naturally adversarial images with ImageNet-compatible categories. CIFAR100 [14] is a standard image classification benchmark that contains 100 classes and 60000 images. Following [33, 30, 43], the ObjectNet [3, 43] dataset we refer to for CIL contains 200 classes with 33137 images. It a subset of the original ObjectNet [3] dataset that contains 50000 images of 313 distinct objects with different backgrounds, rotations, and imaging viewpoints. OmniBenchmark-1K [41, 20] has 188569 images and 1000 classes with non-overlapping data with ImageNet-1k. It is a subset of the clean version of OmniBenchmark [41] with at least 100 images per class. A.2

Implementation Details

We use the PILOT [29] codebase to run our experiments. Following [35, 34, 42, 30, 43, 33, 12, 18, 36], our main experiments use the ViT-B/16 backbone pretrained on ImageNet-21k. FACET inserts one task-conditioned adapter shared across all tasks in each transformer block, with two 384-dimensional hidden layers. The linear projections for generating context vectors (Ws , WU , WV ) are initialized to zeros for the first step of training. The auxiliary classifier is a linear layer that is trained to predict class labels directly. A zero-mean Gaussian noise with a standard deviation of 0.3 is added to the context vector during training to improve its robustness against imperfect context predictions during inference. Optimization uses SGD with a cosine learning-rate schedule, a weight decay of 5 × 10−4 , a batch size of 16, a learning rate of 0.04, and 20 training epochs. Classifier alignment following [33] is included during training. For the short task sequence benchmarks with 10 to 20 tasks, we ensemble the predictions when using task context and when disabling the adapter by setting task context to zero. For long task sequences, predictions use only the FACET probabilities from a single backbone pass.

B

Additional Experiments

Table 8: Performance comparison on ImageNet-R (Inc10) using ViT-B/16-IN21k backbone and 5 random seeds. Metric is Ā / AT . Method \ Seed SD-LoRA TUNA MIN FACET

49

127

982

1523

2026

Mean ± Std

78.50 / 70.58 82.51 / 77.78 84.41 / 78.90 85.63 / 80.80

77.27 / 70.73 81.55 / 76.97 84.46 / 77.80 85.14 / 79.32

77.82 / 71.85 82.37 / 77.73 85.22 / 79.40 85.89 / 80.98

76.58 / 69.95 82.44 / 77.68 84.38 / 78.10 84.66 / 79.52

77.22 / 70.77 82.44 / 77.15 83.65 / 77.43 84.77 / 79.28

77.48 ± 0.72 / 70.78 ± 0.68 82.26 ± 0.40 / 77.46 ± 0.37 84.42 ± 0.56 / 78.33 ± 0.81 85.22 ± 0.53 / 79.98 ± 0.84

Table 9: Performance comparison on ImageNet-R (Inc10) using ViT-B/16-IN1k backbone and 5 random seeds. Metric is Ā / AT . Method \ Seed SD-LoRA TUNA MIN FACET

49

127

982

1523

2026

Mean ± Std

81.40 / 74.65 83.98 / 78.53 85.01 / 79.35 86.05 / 80.87

81.19 / 75.72 83.37 / 77.97 84.26 / 77.95 84.82 / 79.90

80.45 / 74.77 84.08 / 78.78 85.75 / 80.03 86.65 / 80.88

80.03 / 74.75 83.36 / 78.31 83.73 / 78.47 86.68 / 80.67

79.82 / 73.32 83.45 / 78.22 83.86 / 78.28 85.50 / 80.07

80.58 ± 0.70 / 74.64 ± 0.86 83.65 ± 0.35 / 78.36 ± 0.31 84.52 ± 0.85 / 78.82 ± 0.85 85.94 ± 0.79 / 80.48 ± 0.46

13

B.1

Robustness to Task Order and Pretrained Backbone

To evaluate the robustness and generalizability of our approach, we shuffle the task order with 5 random seeds and experiment with two widely used pretrained backbones, ViT-B/16-IN21k and ViT-B/16-IN1k [5]. Tables 8-9 present our results. FACET outperforms strong baselines in all task orders on both backbones. This indicates that the advantage of FACET is consistent rather than an artifact of a particular task order. B.2

Detailed Ablations on Single Adapter Design

Random initialization starts each new task with a random transformation of the adapter feature space and reduces AT to 76.68%. Zero initializa- Table 10: Impact of initialization and noise on tion avoids this unstable start by making a newly ImageNet-R (Inc10). Config. Ā AT activated task-conditioned mixture component initially fall back to the pretrained representation, Random, no noise 82.74 76.68 improving AT to 78.80%. Adding noise to the Zero, no noise 83.95 78.80 Zero, +noise 84.70 79.97 context vector further improves AT to 79.97%, supporting its role in making the model robust to imperfect task identity prediction during inference. Finally, Table 12 varies the bottleneck dimension and adapter depth. The results are stable across a range of capacities, and deeper adapters do not consistently improve performance, indicating that FACET’s gains are not simply due to adding more adapter parameters. As shown in Table 11, FACET is robust to different values of Kmax , with smaller values achiev- Table 11: Effect of Kmax on ImageNet-R (Inc10). ing better performance for ImageNet-R (Inc10) Metric \ Dim 50 100 200 400 with 20 tasks. We select 200 since it is a sufAT 80.20 80.14 79.97 79.64 ficiently large upper bound on the number of Ā 84.77 84.73 84.70 84.44 tasks in CIL. Table 12: Ablation of adapter capacity on ImageNet-R (Inc10). Metric is (AT ). Hidden depth \ Dim

16

64

256

384

1 2 3

79.08 79.39 79.37

79.10 79.30 78.97

79.38 79.57 79.62

79.49 79.97 79.15

Table 13: Impact of the consistency weight λcon on ImageNet-R (Inc10). Metric

0.5

1.0

2.0

4.0

8.0

Ā AT

83.94 77.12

84.75 79.28

84.70 79.97

84.34 78.72

84.13 78.50

B.3 Impact of the Coefficient for Task-Dependent Feature Consistency Loss Table 13 shows that λcon = 2.0 achieves the highest final accuracy, achieving an optimal balance between mitigating forgetting of task-specific feature distributions and learning new task.

C

Analysis of Task Prediction Accuracy

We further evaluate the robustness of FACET to wrong task-context predictions during inference. Without retraining, we compare with a randomly assigned context and a deliberately incorrect context on the same checkpoint. Random and forced-wrong results are mean ± sample standard deviation over five tries. Task prediction accuracy is 5.3 to 9.8 times above random guesses. More importantly, using an incorrect context reduces final accuracy by at most 4.48 (79.97 to 75.49 for ImageNet-R 20 tasks). 14

Table 14: Robustness of FACET to wrong task-context predictions at inference (same checkpoint, no retraining). Random and forced-wrong results are mean ± sample standard deviation over five tries. Setting ImageNet-R Inc20 ImageNet-R Inc10 ImageNet-R Inc4

Task-context pred. acc. (random choice)

Final acc. w/ predicted ctx.

Final acc. w/ oracle ctx.

Final acc. w/ random ctx.

Final acc. w/ forced-wrong ctx.

53.83% (10%) 49.05% (5%) 15.21% (2%)

80.66 79.97 76.63

86.38 81.62 78.72

78.07 ± 0.19 75.80 ± 0.24 76.59 ± 0.13

77.54 ± 0.15 75.49 ± 0.28 76.47 ± 0.09

We note that 75.49 is comparable to the performance of the strong baseline SD-LoRA (75.26). This shows that FACET is robust to wrong task-context predictions during inference. We note that providing ground-truth task context during inference is beneficial (ablation Table 7), and can improve final accuracy from 79.97 to 81.62 on ImageNet-R 20 tasks. Therefore, the performance of FACET is not only positively correlated with task-context prediction accuracy, but also robust against wrong task-context predictions. More advanced mechanisms to predict task context can further improve performance.

D

Additional Analysis of Task-Conditioned Feature Consistency

Figure 3: How well our single adapter preserves the feature distribution of the first task as it is sequentially trained on new tasks from task 2 to 100. Top: Mean square error as the distance metric. Bottom: Cosine distance as the distance metric. We compute the distance between the CLS feature produced by the model after it has learned the first task with the corresponding CLS feature after the model has sequentially learned t ∈ {2, ..., 100} tasks. The original training data and context for the first task are used for a direct comparison. Here we evaluate how well our conditioned feature consistency loss preserves the feature distributions of historical tasks. Specifically, we measure how well the sample features from the first task are preserved after learning new tasks. The CLS token from the last transformer block is selected as the feature representation since it is passed to the classifier for prediction. Note that the task-conditioned feature consistency loss is computed using data from task t (x ∈ Dt ), which is the latest task the adapter is being trained on. Results are shown in Fig. 3, where the solid line denotes the mean distance and the light shaded area illustrates the standard deviation. It can be seen that our conditioned feature consistency loss (blue line) effectively preserves the original features from the first task, even after the adapter has 15

been trained on 99 subsequent tasks from the OmniBenchmark-1K dataset. Without our conditioned feature consistency loss (orange line), the MSE and Cosine distances are much larger, which indicates forgetting of the task-specific distribution. For instance, after task 100, the mean cosine distance and mean squared error when using the consistency loss are 0.0588 and 0.0866, respectively. Without the consistency loss, cosine distance and MSE increases to 0.249 (+323%) and 0.427 (+393%), respectively. These results confirm the effectiveness of the conditioned feature consistency loss in preserving task-specific feature distributions in continual learning even though it relies on data from the latest task instead of those from the original tasks. We provide more details on how cosine distance and MSE are computed below: We first extract the CLS features of the data samples from the first task using the adapter trained on its dataset D1 . This defines the original feature distribution of the first task. Then, after learning t tasks, we extract CLS features using the same data samples from D1 and the same context vector for the first task, but using the adapter sequentially trained on all t tasks. We evaluate t ∈ {2, ..., 100} using the OmniBenchmark-1K dataset. Because the input images and context vectors are the same, any difference between corresponding CLS features directly measures how much an original feature from the first task has drifted after learning later tasks. We quantify this drift using cosine distance and mean squared error (MSE). For each training image x ∈ D1 , cosine distance is computed between the original CLS feature and the corresponding CLS feature extracted after learning t tasks: dcos = 1 −

⟨CLSθ11 , CLSθ1t ⟩ , ∥CLSθ11 ∥2 ∥CLSθ1t ∥2

where CLSθ11 is the original CLS feature from the model trained on the first task, and CLSθ1t is the CLS feature from the model after learning t tasks using the same context. We average the cosine distance over all data samples. We also compute the MSE distance: 2 1 CLSθ11 − CLSθ1t , N 2 where N is the number of images in the first task. These two metrics are complementary: cosine distance captures directional drift, while MSE captures overall feature-space displacement.

dMSE =

Table 15: Task-1 forgetting versus CLS feature drift under different λcon on ImageNet-R Inc10. λcon

Task-1 forgetting

Cosine distance

MSE

0 0.5 1 2 4 8

35.96 36.59 31.23 18.61 15.46 13.32

0.2376 0.0888 0.0538 0.0285 0.0167 0.0113

0.5072 0.1578 0.0961 0.0502 0.0296 0.0206

We also directly test whether feature drift is related to old-task classification forgetting. On ImageNetR Inc10, we train six models that differ only in λcon . After task 20, we compare the CLS features of the task-1 images with their features immediately after task 1, always using the task-1 context, and measure task-1 forgetting. Across the six settings, cosine distance and MSE have Pearson correlations of 0.752 and 0.711 with task-1 forgetting. Greater drift is strongly associated with greater old-task classification forgetting.

E

Justification of Using Current-Task Data for Conditioned Feature Consistency

In Equation 6, we use the data from the current task Dt to preserve the old feature distributions. Here we provide a justification on why it works. Current-task images are probe inputs used to detect whether new-task training changes the shared adapter under a previous context. They are not treated as approximations of old-task images in the consistency loss. 16

After task t−1, we freeze a copy of the adapter. While learning task t, a current image is processed by the frozen and updated adapters under the same sampled old context cj . Because the input and context are identical, their feature difference isolates the effect of the adapter update. Minimizing this difference therefore constrains new-task training from changing the adapter’s behavior under cj . The backbone is frozen and all tasks use the same low-dimensional adapter. Consequently, current probes can protect old features when the same adapter weights affect both current and old features. In that case, a weight change that damages old features also changes the current probe features and is penalized by the consistency loss. This does not require the tasks to have identical images, labels, or feature distributions. Local Justification. Let ∆θ = θ − θ− be one adapter update and let hθ (x, cj ) be the conditioned feature. For a small update, hθ (x, cj ) − hθ− (x, cj ) ≈ Jj (x)∆θ,

(8)

where Jj (x) is the feature Jacobian with respect to the shared-adapter parameters. The current-input consistency loss and old-input feature drift are locally Ljcon ≈ ∆θ⊤ Gjt ∆θ,

Dj ≈ ∆θ⊤ Gjj ∆θ,

(9)

where

  Gja = Ex∼Da Jj (x)⊤ Jj (x) . (10) If every small adapter change that affects old features is also detected by the current probes, up to a constant κ, Gjj ⪯ κGjt , (11) then Dj ≤ κLjcon . (12) Thus, minimizing consistency on current inputs also limits old-task feature drift when current and old inputs overlap and respond to the same adapter changes.

Empirical justification for sufficient overlap. Overlap means that changing the same adapter weights affects both sets of conditioned features in similar ways. It does not require current and old images, or even their frozen-backbone features, to be similar one by one. Jj (x) records the effect of small adapter-weight changes for one input, and Gjt and Gjj summarize these effects over current and old inputs. We test this overlap directly with feature-Jacobian products over all shared-adapter parameters. We use ImageNet-R (Inc10) and analyze a checkpoint obtained while training the second task. At this fixed checkpoint, we construct two input sets. The first contains old inputs, namely images from the first task. The second contains current probes, namely images from the second task. Both sets are processed using the old-task context c1 . Therefore, the only difference between G01 and G02 is the input distribution,     G01 = Ex∼D1 J1 (x)⊤ J1 (x) , G02 = Ex∼D2 J2 (x)⊤ J2 (x) . (13) For each image set, we measure how the 768-dimensional conditioned feature responds to all sharedadapter parameters. The auxiliary classification heads are excluded because they do not produce the conditioned feature matched by the consistency loss. These measurements support the required coverage interpretation. Current-task images, when evaluated under the old-task context, activate most adapter directions that are important for first-task features. The experiment does not claim that old- and current-task images are semantically similar. We verified that this behavior remains after the full 20-task sequence. At the final checkpoint, we use images from the last task as current probes and compare them with images from old tasks 0, 4, 8, 12, and 16. For each old task j, both image sets are evaluated under that task’s context cj . Across these five comparisons, the average similarity between current-task directions and old-task directions is 0.869±0.049, and 99.58% of sampled old-active directions satisfy the directional relation with κ=5.

17

Table 16: Feature-Jacobian overlap between current probes and old inputs under the old-task context c1 on ImageNet-R (Inc10). Measurement

Result

Interpretation

Similarity between adapter 0.809±0.041 Normalized similarity between G01 and G02 . A value of changes detected by current and 1 means identical sensitivity structure up to scale, and 0 old inputs means orthogonal structure. The result indicates substantial overlap. Rank-16 old-gradient energy cap- 85.2±3.5% We learn the 16 strongest adapter directions from current tured by current directions probes and test them on held-out old inputs. They capture 85.2% as much old-input Jacobian variance as directions learned directly from old inputs. Sampled old-active directions satFor 96.88% of sampled directions that affect old features, 96.88% isfying v ⊤ G01 v ≤ 2 v ⊤ G02 v the old-input effect is no more than twice the effect detected by current probes. 100% Sampled old-active directions satEvery sampled old-active direction is detected by current isfying v ⊤ G01 v ≤ 5 v ⊤ G02 v probes within a factor of 5.

18

Related documents

• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models#31266
• NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts#975078
• Geometry of Forgetting: Representation Flux in Continual Learning#616986
• Geometry of Forgetting: Representation Flux in Continual Learning#616486
• Convergence of Continual Learning in Homogeneous Deep Networks#321826