arXiv CSopen access

Data-Free Reservoir Features for Efficient Long-Horizon Cold-Start Continual Learning

arXiv:2606.27095v1 [cs.LG] 25 Jun 2026

Data-Free Reservoir Features for Efficient Long-Horizon Cold-Start Continual Learning

Augustinas Jučas Department of Computer Science University of Oxford [email protected]

Yangchen Pan Department of Engineering Sciences University of Oxford [email protected]

Abstract Cold-start exemplar-free class-incremental learning requires learning a growing set of classes without replay, external pretraining, or a large initial task. Existing cold-start methods typically either train the backbone throughout the stream and compensate for semantic drift, or freeze a backbone after the first task, producing features biased toward the initial classes. These choices also create a computational tension: drift-compensation methods require repeated backbone training and increasingly expensive updates as the task horizon grows, while frozenbackbone methods are cheap but weak under cold start. We study a third option: a feature extractor that is never fit to image data at all. We propose CIRCLE, a class-incremental classifier built from fixed bidirectional two-dimensional reservoir features, adapted from BiRC2D for image classification, and streaming linear discriminant analysis heads. CIRCLE groups multiple random reservoir instantiations into feature ensembles and averages the softmax outputs of independent SLDA heads, yielding a tunable bias-variance tradeoff between richer random features and prediction-level ensembling. Because the feature extractor is fixed and the head admits streaming closed-form updates, CIRCLE performs sample-wise training without replay, task-boundary information, or backbone backpropagation. On CIFAR-100, TinyImageNet, ImageNet-Subset, and ImageNet-1k, CIRCLE is competitive at 10-20 task splits and substantially outperforms strong CS-EFCIL baselines at 50, 100, and 500 task splits, while training much faster than trainedbackbone drift-compensation methods. Ablations show that the BiRC2D-style extractor, SLDA head, and balanced feature/prediction ensembling each contribute to the final performance.

1

Introduction

Continual learning (CL) studies models trained on non-stationary data streams, where naive updates can overwrite prior knowledge and cause catastrophic forgetting [McCloskey and Cohen, 1989, French, 1999]. In CL, class-incremental learning (CIL) [Zhou et al., 2024] considers a sequence of tasks with disjoint class sets, requiring prediction over all seen classes at test time. We focus on the cold-start exemplar-free (CS-EFCIL) setting [Zhou et al., 2024]. Exemplar-free prohibits storing or replaying past data, while cold-start further disallows pre-trained backbones or large initial tasks, preventing reliance on pre-trained representations. Hence, CS-EFCIL exposes a representation-learning tradeoff. To make this tradeoff precise, view a classifier as the composition of a feature extractor and a classification head: the feature extractor maps inputs to representations, while the head maps these representations to class predictions. Since exemplar-free methods cannot store old samples, the head often retains compact information about past classes, such as means and covariances. Most cold-start methods continue training the feature extractor throughout the stream and compensate for semantic drift, i.e., changes in the feature space that make head’s Preprint.

stored old-class statistics stale [Yu et al., 2020]. Such methods work well at moderate horizons, but drift control is approximate and can accumulate error over many task transitions. Alternatively, one can freeze the feature extractor after training on the first task T1 and further only update a closed-form head, avoiding drift and repeated backbone training. However, in cold-start, the data available before freezing is only a small first task, resulting in non-transferable feature extractors biased towards initial classes. Thus, existing approaches face a tradeoff between trained feature extractors that drift and frozen ones that are first-task biased and less transferable. This tradeoff also has a computational dimension. Trained-backbone methods repeatedly backpropagate through the feature extractor and often remap old-class statistics after each task, making long-horizon streams potentially expensive [Magistri et al., 2025, Rypeść et al., 2024]. A natural third option is therefore to use a feature extractor that is frozen from the beginning and never fitted to image data. Such a feature extractor is neither pretrained externally nor biased by first-task training, and it avoids backbone backpropagation and expensive drift estimation entirely. The open question is whether such data-free features can be strong enough for nontrivial image classification. We answer this question with CIRCLE (Class-Incremental Reservoir Classifier with sLDA Ensemble), a cold-start exemplar-free classifier built from an untrained frozen feature extractor and streaming linear discriminant analysis (SLDA) heads [Hayes and Kanan, 2020]. We adapt BiRC2D [Nakanishi and Tokunaga, 2024], originally developed for spectral image anomaly detection, into such a feature extractor; ensemble multiple random instantiations of the extractor through both feature concatenation and prediction averaging; and update each head closed-form using additive sufficient statistics. CIRCLE therefore supports sample-wise learning without replay, task-boundary information, or backbone optimization. Our contributions are the following: • We study a data-free frozen-feature design for CS-EFCIL and propose CIRCLE, which combines a fixed random feature extractor with an SLDA ensemble. • We show that CIRCLE is competitive at CS-EFCIL on standard 10–20 task splits and considerably stronger at 50–500 task splits on CIFAR-100 [Krizhevsky, 2009], TinyImageNet [Wu et al., 2017], ImageNet-Subset, and ImageNet-1k [Russakovsky et al., 2015], with substantially lower training time than trained-backbone baselines and no backbone-training. • We ablate the feature extractor, analytic head choice, and grouped ensemble design, showing that each component substantially contributes to performance.

2

Related Work

Exemplar-free class-incremental learning [Zhou et al., 2023] can be categorized along two axes: warm- vs. cold-start settings, and trained vs. frozen feature extractors. Warm-start methods rely on external pretraining or a large initial task, whereas cold-start methods learn from comparable task sizes without external data. Independently, trained-backbone methods update the feature extractor throughout the stream, while frozen-backbone methods update only the classifier head. Trained-backbone methods update the feature extractor ϕ jointly with the head throughout the stream, so head’s old-class statistics must remain compatible with a feature space that changes over time; here warm- vs. cold-start differs only by pre-training. Their main limitation is semantic drift [Yu et al., 2020]: as ϕ changes, old-class features shift and stored head statistics become outdated. Existing methods mitigate this via rehearsal/augmentation (PASS [Zhu et al., 2021a], IL2A [Zhu et al., 2021b]), drift estimation (SDC [Yu et al., 2020], ADC [Goswami et al., 2024], LDC [Gomez-Villa et al., 2024], AdaGauss [Rypeść et al., 2024], EFC [Magistri et al., 2024], EFC++ [Magistri et al., 2025]), or distillation-style regularization (LwF [Li and Hoiem, 2016], AdaGauss, EFC, EFC++). They are effective on the short and moderate horizons most commonly studied in prior CS-EFCIL work. However, because most prior evaluations focus on short horizons (T ∈ {5, 10, 20}), the behavior of these methods at much longer horizons remains underexplored. In Section 4, we evaluate T ∈ {50, 100} and an ImageNet-1k T = 500 stress test, and find that performance degrades sharply for trained-backbone drift-compensation methods in this regime. They are also computationally expensive, with per-task cost growing with the number of seen classes [Magistri et al., 2025]. Warm-start frozen-backbone methods freeze the feature extractor ϕ after pre-training or after a large initial task T1 . A fixed ϕ enables closed-form head updates over frozen features, ensuring equivalence between continual training and joint training on all seen classes. Examples include ACIL [Zhuang et al., 2022], DS-AL [Zhuang et al., 2024], FeCAM [Goswami et al., 2023], RanPAC [McDonnell 2

et al., 2023], SLDA [Hayes and Kanan, 2020], and FeTrIL [Petit et al., 2023]. Their key requirement is a transferable ϕ, typically obtained via pre-training or large T1 ; without this, performance degrades. Frozen-ϕ methods are more efficient – no backpropagation or drift control after T1 are needed, however the initial T1 training can still be expensive. Cold-start frozen-backbone methods are largely under-explored. There is little work that freezes ϕ under cold-start; existing baselines, primarily built for warm-start, simply train ϕ on a small T1 then freeze ϕ [Goswami et al., 2023], which could hurt performance due to ϕ’s strong representational bias towards a small T1 . This highlights a trade-off within cold-start EFCIL: training ϕ avoids representational bias but incurs drift and cost, while freezing ϕ enables efficient, drift-free updates but suffers from first-task bias. An ideal cold-start approach thus would achieve a non-biased, transferable and fixed ϕ without pre-training, implying a backbone ϕ that is not fit to any task-specific data. Reservoir computing (RC) offers such a data-free ϕ: reservoirs are fixed random nonlinear dynamical systems that embed input sequences into high-dimensional spaces, with theoretical guarantees, with only a trained readout [Jaeger, 2001, Maass et al., 2002]. Due to recurrent nature of RC, existing RC-based CL mainly considers time-series data [Bereska and Gavves, 2022]; image reservoirs are harder because naive pixel flattening loses spatial structure [López-Ortiz et al., 2024, Chang and Futagami, 2020]. BiRC2D [Nakanishi and Tokunaga, 2024] addresses this by applying bidirectional 2D reservoirs over image patches. We adapt this image-reservoir idea to class-incremental image classification. To our knowledge, this is the first study of data-free, reservoir-based CS-EFCIL.

3

CIRCLE: Bidirectional Reservoir Ensembles with Streaming LDA

We now describe CIRCLE. The method has three components: (1) a fixed reservoir-based feature extractor adapted from BiRC2D [Nakanishi and Tokunaga, 2024] for classification (Section 3.1); (2) a feature ensembling scheme, which concatenates embeddings from several independent reservoir instantiations (Section 3.2); (3) a prediction ensembling scheme, which averages the probability outputs of several independent SLDA heads (Sections 3.2 and 3.3). 3.1

Data-free Reservoir Features

Let ϕs : X → Rdr denote a random reservoir-based feature extractor instantiated with a random seed s. The seed determines all random convolutional, reservoir, and projection weights, which remain fixed throughout the stream. We adapt BiRC2D [Nakanishi and Tokunaga, 2024] from image anomaly detection to image-level classification. Each image is first mapped by a small fixed random convolutional stem. The resulting feature map is divided into patches and processed by bidirectional 2D reservoir blocks, which scan spatial locations in four directions and concatenate the directional states. We then aggregate the final spatial map into a single image embedding and apply a fixed random projection to obtain the reservoir feature ϕs (x). Compared with the original BiRC2D, our classification variant uses random convolutional preprocessing, ReLU reservoir activations with Kaiming initialization, spatial flattening into an image-level vector, and a final random upscaling layer. Full architectural details are given in Appendix A. 3.2

Grouped Feature and Prediction Ensembling

A single random reservoir can yield a noisy feature space, so CIRCLE uses multiple independent reservoir draws. Given n random seeds, we partition them into k equal-size disjoint groups, denoted Gi , i = 1, ..., k, such that | ∪ki=1 Gi | = n and |Gi | = n/k = m (with n, k, m integers by design). Feature ensembling then concatenates their embeddings: ΦGi (x) = ||s∈Gi ϕs (x) where ||s∈Gi denotes concatenation of features generated by the seeds in Gi . This provides each classifier with a richer random-feature representation without training additional backbones. CIRCLE also ensembles at the prediction level. We partition n reservoirs into k disjoint groups, attach one SLDA head to each group, and average their probabilities: p̂(x) =

k
1X softmax hj (ΦGj (x)) , k j=1

3

where hj denotes the jth head. This grouped design interpolates between two extremes: using all reservoirs in one feature vector gives a rich representation but no prediction averaging, while using one reservoir per head gives many predictions but weak individual representations. Bias-variance insight into the two-level ensembles. Feature and prediction ensembling act on complementary axes, corresponding to a bias–variance tradeoff between feature richness and ensemble diversity. Concatenating all n reservoirs yields a rich representation but a single classifier (no variance reduction), while assigning one reservoir per classifier yields diverse predictions but weak features. Grouping reservoirs into k sets of size m = n/k interpolates between these extremes: withingroup concatenation reduces bias, while averaging across groups reduces variance. Proposition B.1 (Appendix B) formalizes this as a bias-variance tradeoff and motivates intermediate group sizes – the optimal m is typically neither 1 nor n; empirically, Section 4.3.3 confirms that balanced grouping performs best. 3.3

Streaming LDA Head

Each group of feature extractors is paired with a streaming linear discriminant analysis (SLDA) head [Hayes and Kanan, 2020]. For class c, the head models features with mean µc , shared covariance Σ, and empirical prior πc . With ridge regularization Σλ = Σ + λI, the class logit for feature z is 1 ⊤ −1 [h(z)]c = z ⊤ Σ−1 λ µc − µc Σλ µc + log πc . 2 The head is updated using additive sufficient statistics: class counts nc , class sums Tc = P and the global second moment M = i zi zi⊤ . On observing (z, y), we update ny ← ny + 1,

Ty ← Ty + z,

P

i:yi =c zi ,

M ← M + zz ⊤ .

At evaluation time, these statistics materialize the LDA classifier in closed form. Given fixed features and the same regularization rule, the sufficient-statistic update is order-invariant and yields the same classifier as LDA trained jointly on all seen samples. Thus CIRCLE has no representation drift and no order-induced approximation error in the head. Full update details are given in Appendix A. Section 4.3.2 shows that SLDA performs uniformly best on top of reservoir features, compared to alternative classification methods. Overall Algorithm. For each incoming sample, CIRCLE computes the feature representation for each reservoir group and updates the corresponding SLDA sufficient statistics. At test time, each head produces logits from its current closed-form LDA parameters; probabilities are averaged across heads. Detailed pseudocode is provided in Appendix A.

4

Experiments

Our experiments test four questions within the cold-start EFCIL regime. First, how does CIRCLE compare with trained-backbone and freeze-after-first-task baselines as the task horizon increases? Second, does the use of many untrained reservoir draws make CIRCLE computationally expensive? Third, we ask whether the same fixed-feature analytic design remains viable in an extreme longhorizon setting, using a 500-task ImageNet-1k stream. Finally, which components of CIRCLE are necessary: the BiRC2D-style feature extractor, the SLDA head, and the grouped feature/prediction ensemble? We first describe the common setup, then present the main comparison and computation study in Section 4.1, the ImageNet-1k long-horizon stress test in Section 4.2, and ablations in Section 4.3. Datasets and protocol. We evaluate on CIFAR-100, TinyImageNet, and ImageNet-Subset with time horizon T ∈ {10, 20, 50, 100}, and evaluate additional long-horizon T = 500 experiments on full ImageNet-1k. All experiments follow the cold-start exemplar-free setting: no replay, no pre-training, and classes split evenly across T tasks. Results are averaged over 5 seeds unless specified. Baselines. We compare against representative EFCIL methods across three categories: driftcompensation / distillation (EFC++ [Magistri et al., 2025], AdaGauss [Rypeść et al., 2024], ADC [Goswami et al., 2024], LwF [Li and Hoiem, 2016]), prototype rehearsal / augmentation (PASS [Zhu et al., 2021a], IL2A [Zhu et al., 2021b]), and frozen-backbone analytic methods 4

Table 1: Cold-start EFCIL results on CIFAR-100, TinyImageNet, and ImageNet-Subset for T ∈ {10, 20, 50, 100}. Each cell reports mean ± std over 5 seeds for mean incremental accuracy (Ā) and final accuracy (AT ). Best mean per data setting in bold, second-best underlined. For our CIRCLE, AT is invariant across task splits because continual training for CIRCLE is equivalent to joint training.

ImageNet-Subset

TinyImageNet

CIFAR-100

Algorithm

T=10

T=20

T=50

T=100

Ā

AT

Ā

AT

Ā

AT

Ā

AT

AdaGauss EFC++ ADC ACIL FeCAM FeTRIL IL2A PASS LwF DS-AL

62.53 ± 0.2 66.61 ± 0.2 61.98 ± 0.2 55.36 ± 0.5 49.71 ± 0.3 52.06 ± 0.2 43.23 ± 4.3 52.85 ± 0.3 50.92 ± 0.4 56.43 ± 0.2

46.57 ± 0.4 52.68 ± 0.4 44.71 ± 0.2 40.73 ± 0.4 33.08 ± 0.4 36.60 ± 0.3 27.24 ± 3.3 36.76 ± 0.3 28.92 ± 0.4 41.76 ± 0.2

56.11 ± 0.3 55.74 ± 0.1 49.86 ± 0.4 44.54 ± 0.4 37.02 ± 0.6 42.25 ± 1.4 15.70 ± 2.1 40.73 ± 0.6 37.15 ± 0.5 45.79 ± 0.1

40.80 ± 0.3 42.10 ± 0.5 31.89 ± 0.5 33.16 ± 0.5 23.09 ± 0.5 27.71 ± 1.2 5.04 ± 1.9 24.85 ± 0.4 18.06 ± 0.4 34.89 ± 0.3

40.30 ± 0.6 38.93 ± 0.2 30.57 ± 0.3 37.37 ± 0.4 34.05 ± 0.4 32.76 ± 0.2 26.97 ± 4.9 33.76 ± 0.6 19.41 ± 0.4 38.17 ± 0.5

15.64 ± 2.5 26.95 ± 0.4 16.22 ± 0.4 24.95 ± 0.6 20.97 ± 0.4 20.24 ± 0.2 15.31 ± 3.8 19.23 ± 0.3 6.03 ± 0.5 25.84 ± 0.5

21.26 ± 0.4 26.73 ± 0.6 8.11 ± 0.1 23.13 ± 0.6 23.44 ± 1.1 22.15 ± 0.2 13.86 ± 0.8 28.70 ± 0.6 5.13 ± 0.3 23.86 ± 0.8

6.87 ± 0.3 16.40 ± 0.3 3.35 ± 0.2 14.04 ± 0.6 13.85 ± 0.7 12.04 ± 0.2 4.43 ± 0.6 15.48 ± 0.1 1.02 ± 0.1 15.02 ± 0.5

CIRCLE

55.32 ± 0.3

45.34 ± 0.3

55.90 ± 0.3

45.34 ± 0.3

56.97 ± 0.3

45.35 ± 0.3

57.22 ± 0.3

45.34 ± 0.3

AdaGauss EFC++ ADC ACIL FeCAM FeTRIL IL2A PASS LwF DS-AL

50.67 ± 0.2 52.34 ± 0.3 47.54 ± 0.2 39.86 ± 0.3 48.02 ± 0.3 37.02 ± 0.2 37.72 ± 0.3 38.71 ± 0.3 37.11 ± 0.4 37.92 ± 0.3

37.97 ± 0.4 39.49 ± 0.3 31.40 ± 0.3 27.26 ± 0.2 34.33 ± 0.2 22.72 ± 0.3 20.30 ± 0.4 25.88 ± 0.4 20.61 ± 0.4 26.38 ± 0.2

44.46 ± 0.2 47.51 ± 0.1 37.21 ± 0.1 31.34 ± 0.1 40.26 ± 0.2 28.10 ± 0.6 27.13 ± 0.2 31.05 ± 0.9 26.37 ± 0.1 30.66 ± 0.3

31.44 ± 0.2 35.16 ± 0.4 21.14 ± 0.2 20.88 ± 0.2 27.81 ± 0.2 15.37 ± 0.7 9.99 ± 0.4 18.38 ± 0.6 11.16 ± 0.1 20.94 ± 0.2

37.02 ± 0.4 36.24 ± 0.2 26.54 ± 0.2 23.52 ± 0.3 25.56 ± 0.5 22.26 ± 0.2 23.94 ± 0.2 24.10 ± 0.3 18.53 ± 0.2 22.81 ± 0.5

21.94 ± 0.3 24.37 ± 0.4 13.05 ± 0.3 14.64 ± 0.3 14.57 ± 0.2 11.96 ± 0.3 13.05 ± 0.4 12.92 ± 0.4 5.96 ± 0.3 14.22 ± 0.4

24.56 ± 0.3 25.17 ± 0.1 18.59 ± 0.3 16.53 ± 1.2 16.36 ± 0.4 16.11 ± 0.1 16.57 ± 0.4 19.57 ± 0.4 8.88 ± 0.1 18.54 ± 0.2

9.71 ± 0.2 13.95 ± 0.3 9.29 ± 0.4 8.47 ± 0.8 8.14 ± 0.1 8.13 ± 0.2 7.39 ± 0.1 8.30 ± 0.4 2.32 ± 0.1 10.11 ± 0.2

CIRCLE

39.72 ± 0.2

30.77 ± 0.2

40.91 ± 0.2

30.78 ± 0.2

41.78 ± 0.2

30.77 ± 0.2

42.17 ± 0.2

30.75 ± 0.2

AdaGauss EFC++ ADC ACIL FeCAM FeTRIL IL2A PASS LwF DS-AL

62.53 ± 0.3 67.97 ± 0.4 58.36 ± 0.4 59.77 ± 0.2 58.38 ± 0.6 55.60 ± 0.5 48.55 ± 0.7 49.72 ± 4.4 56.97 ± 0.3 60.48 ± 0.4

45.84 ± 0.4 53.46 ± 0.4 37.44 ± 0.8 44.46 ± 0.3 42.32 ± 0.7 39.46 ± 0.7 29.52 ± 1.0 33.73 ± 2.0 29.34 ± 0.9 45.56 ± 0.5

53.76 ± 0.4 58.54 ± 0.6 49.32 ± 0.4 45.70 ± 0.2 44.86 ± 0.4 44.82 ± 0.2 41.76 ± 0.4 40.62 ± 1.8 40.89 ± 0.4 46.68 ± 0.3

33.87 ± 0.7 40.86 ± 0.6 28.62 ± 0.4 32.18 ± 0.3 29.24 ± 0.6 28.94 ± 0.7 23.73 ± 1.4 23.18 ± 1.6 14.88 ± 0.2 32.91 ± 0.2

33.76 ± 0.3 40.45 ± 0.3 29.10 ± 0.6 36.60 ± 0.5 21.39 ± 0.4 31.14 ± 0.3 22.85 ± 0.5 25.95 ± 2.5 18.11 ± 0.4 37.57 ± 0.3

14.28 ± 0.3 23.10 ± 0.6 12.87 ± 0.1 22.09 ± 0.3 9.88 ± 0.2 16.98 ± 0.1 10.07 ± 0.4 8.15 ± 1.2 3.74 ± 0.4 23.02 ± 0.3

22.02 ± 0.3 28.47 ± 0.6 7.42 ± 0.2 26.39 ± 0.6 27.22 ± 0.4 24.02 ± 0.4 13.71 ± 1.4 18.87 ± 6.0 4.79 ± 0.5 26.71 ± 0.7

6.93 ± 0.3 13.92 ± 0.5 1.02 ± 0.0 14.96 ± 0.5 14.20 ± 0.5 11.56 ± 0.4 2.11 ± 0.7 5.31 ± 1.9 0.73 ± 0.2 15.35 ± 0.5

CIRCLE

52.18 ± 0.4

39.68 ± 0.3

53.54 ± 0.4

39.66 ± 0.3

54.56 ± 0.4

39.65 ± 0.3

54.71 ± 0.4

39.66 ± 0.3

(ACIL [Zhuang et al., 2022], DS-AL [Zhuang et al., 2024], FeCAM [Goswami et al., 2023], FeTrIL [Petit et al., 2023]), adapted to cold-start by training on T1 and freezing thereafter. All trained-backbone baselines use ResNet-18. We report mean incremental accuracy Ā and final accuracy AT . For efficiency, we report end-to-end wall-clock training time, total parameter count, and learnable parameter count. All methods are tuned separately for each (T, dataset) configuration, which is important because the long-horizon splits induce substantially different task dynamics. Tuning protocol, selected hyperparameters, and implementation details are given in Appendices C and D. 4.1 4.1.1

Main Comparison and Computational Efficiency Performance for T ∈ {10, 20, 50, 100}

Table 1 compares CIRCLE with all baselines on CIFAR-100, TinyImageNet, and ImageNet-Subset across task horizons T ∈ {10, 20, 50, 100}. At T = 10, drift-compensation methods (EFC++, AdaGauss) perform best across all datasets, as limited task transitions keep drift-estimation errors small. CIRCLE does not match them in mean incremental accuracy Ā, but, despite no image-trained backbone, remains competitive with other cold-start baselines, often matching or exceeding ACIL, FeCAM, FeTRIL, PASS, and LwF on at least one metric. At T = 20, CIRCLE’s relative performance improves. It achieves the best final accuracy AT on CIFAR-100 and second-best Ā (just below AdaGauss); and ranks second in AT on 5

ImageNet-Subset while staying close in Ā. TinyImageNet is less favourable: EFC++ and AdaGauss lead, but CIRCLE still outperforms the other baselines in AT and remains competitive in Ā. As the task horizon grows, trained-backbone methods (e.g., EFC++, AdaGauss) degrade sharply, with accuracies dropping by around 10-30 percentage points from T = 10, potentially due to accumulated drift-estimation errors. Freeze-after-first-task methods also deteriorate, as the initial task likely becomes too small to yield transferable features, inducing representation bias. CIRCLE avoids both failure modes: its feature extractor is fixed and never fitted to the first task, and its SLDA head updates via additive sufficient statistics, making continual training equivalent to batch training. As a direct result, AT is invariant to task splits and Ā slightly improves with T (averaging artefact), making CIRCLE the strongest method across all datasets at T = 50 and T = 100. Takeaway. The comparison shows that relative performance changes with the task horizon. Driftcompensation methods are strongest at short horizons, where few feature-space shifts have occurred. As the number of task transitions grows, their advantage disappears and then reverses. CIRCLE is therefore not a short-horizon universal SOTA method; its advantage is specifically in long-horizon cold-start EFCIL, where avoiding both representation drift and first-task feature bias becomes decisive. 4.1.2

Training Time and Computation Cost

To assess computation cost, we compare CIRCLE against representative strong baselines from different design families: EFC++ and AdaGauss as trained-backbone drift-compensation methods, ACIL and FeCAM as freeze-after-first-task analytic methods, and FeTRIL as a freeze-after-firsttask method with an SGD-trained classification head. Table 2 reports wall-clock training time and parameter counts at T = 20. Table 2: Wall-clock training time at T = 20, mean ± std over 3 seeds, combined with parameter counts. Times are in minutes, parameter counts are in millions. Total denotes the total parameter count, Learn denotes the learnable parameter count. Fastest method in bold, second-fastest underlined. CIFAR-100

TinyImageNet

ImageNet-Subset

Method

Time

Total

Learn.

Time

Total

Learn.

Time

Total

Learn.

ACIL FeCAM FeTrIL AdaGauss EFC++

9.27 ± 0.05 15.53 ± 0.24 177.62 ± 8.96 119.63 ± 4.07 98.39 ± 2.66

16.18M 11.22M 11.31M 11.20M 11.22M

11.99M 11.22M 11.31M 11.20M 11.22M

52.89 ± 0.12 26.11 ± 0.37 560.09 ± 10.46 200.17 ± 1.57 310.04 ± 5.04

17.10M 11.27M 11.37M 11.20M 11.27M

12.90M 11.27M 11.37M 11.20M 11.27M

87.26 ± 0.43 107.07 ± 1.70 288.70 ± 0.68 858.50 ± 9.59 390.31 ± 6.44

21.20M 11.22M 11.32M 11.21M 11.23M

12.81M 11.22M 11.32M 11.21M 11.23M

CIRCLE

2.04 ± 0.03

11.69M

7.04M

11.51 ± 0.23

11.46M

10.04M

39.22 ± 0.17

11.05M

6.48M

Despite using multiple reservoirs, CIRCLE is the fastest method on all three datasets: 4.5× faster than the second-fastest method on CIFAR-100, 2.3× on TinyImageNet, and 2.2× on ImageNet-Subset. Relative to the faster drift-compensation baseline on each dataset, the gap is roughly an order of magnitude. CIRCLE also uses fewer learnable parameters than all compared methods, while keeping total parameter count comparable to the ResNet-18-based baselines. Thus, the ensemble is not a hidden training-cost advantage: the additional reservoirs are fixed random draws, not separately trained backbones. Figure 3 (Appendix C.2) provides an additional view of the accuracy–time trade-off, which shows that CIRCLE can finish the whole stream before some baselines finish early training. Takeaway. The reservoir ensemble does not make CIRCLE expensive to train. Although CIRCLE may use many fixed reservoir draws, these draws require no optimization. Training is therefore dominated by forward passes and additive SLDA statistic updates, not backbone backpropagation. 4.2

Long-horizon Behaviour: ImageNet-1k at T = 500

We use ImageNet-1k with T = 500 as an extreme long-horizon stress test for CIRCLE. Each task contains only two classes, so the method must accumulate information through many small updates while retaining early classes. This setting directly tests whether fixed data-free reservoir features with analytic SLDA updates remain useful beyond the standard short- and moderate-horizon regimes. We compare CIRCLE with representative methods from the two main baseline families: EFC++ and AdaGauss as trained-backbone methods, and ACIL, DS-AL, and FeCAM as frozen-feature analytic 6

methods. To complement incremental accuracy, we also track accuracy on the first task T1 . This gives a metric of whether the earliest class statistics remain useful after hundreds of later updates. Figure 1 shows that CIRCLE remains stable throughout the stream and achieves the highest incremental accuracy at every task index. Its first-task accuracy never reaches zero, indicating that early-class statistics remain usable even after hundreds of updates, which matches CIRCLE’s construction of continual training being equivalent to joint training.

100

CIRCLE AdaGauss EFC++

80

100

ACIL FeCAM DS-AL

Task-1 accuracy (%)

Incremental accuracy (%)

The trained-backbone baselines are less stable in this regime: EFC++ and AdaGauss reach zero first-task accuracy after roughly 30 and 75 tasks, respectively. They are also difficult to run at this horizon: EFC++ is truncated after a near 5-day budget, while AdaGauss fails after about 110 tasks due to numerical instability. Thus, the T = 500 experiment supports the main long-horizon claim of CIRCLE: fixed reservoir features with analytic updates remain accurate and practical when the task stream becomes very long.

60 40 20

80

CIRCLE AdaGauss EFC++

ACIL FeCAM DS-AL

400

500

60 40 20 0

0 0

100

200

300

400

0

500

100

200

300

Task index

Task index

Figure 1: ImageNet-1k at T = 500. Accuracy is tracked over the task stream and averaged over 2 seeds (single std shaded). EFC++ is truncated after a near 5-day budget, and AdaGauss after failing at ∼ 110 tasks due to numerical instability. Left: incremental test accuracy. Right: mean accuracy tracked for the first task, showing how quickly each method forgets the start of the stream. Takeaway. CIRCLE remains stable in very long task horizons, outperforming both frozen- and trained- backbone baselines, while trained-backbone baselines collapse in such a regime. 4.3

Ablations

We isolate the contributions of CIRCLE’s three main design choices: the reservoir feature extractor, the analytic classification head, and the two-level ensembling scheme. All ablations are performed at the T = 20 split on each dataset, using 5 seeds. Unless stated otherwise, all components not under study are kept fixed to the default CIRCLE configuration. Hyperparameters specific to each variant, including feature- and prediction-ensembling sizes, are tuned separately for every variant using the same protocol as in the main experiments. 4.3.1

Feature Extractor

We first evaluate whether BiRC2D is essential, or whether similar performance can be obtained with other non-trained feature extractors. Therefore, we replace the BiRC2D extractor with six alternatives: (i) a randomly-initialised ResNet-18 with BatchNorm, matching the architecture used by the trainedbackbone baselines; (ii) a randomly-initialised ResNet-18 with LayerNorm instead of BatchNorm, controlling for sensitivity of BatchNorm at random initialisation; (iii) a randomly-initialised VGG-13, a comparable-capacity convolutional architecture without normalization layers; (iv) Patch-RNN, a natural extension of pixel-flatten-into-ESN approaches [López-Ortiz et al., 2024], where the image is divided into patches and processed sequentially by a single ESN; (v) Conv+Patch-RNN, which adds a small fixed random convolutional stack before the Patch-RNN, in the spirit of Chang and Futagami [2020]; and (vi) the scattering transform, an untrained, theory-driven CNN-style filter bank [Bruna and Mallat, 2012]. Additional details on these extractors are given in Table 5. For each extractor, we keep the rest of the CIRCLE pipeline unchanged: features are ensembled, followed by SLDA heads, and combined through prediction ensembling. Ensembling sizes and extractor-specific hyperparameters are tuned per variant. Table 3 reports the results at T = 20 on all three benchmarks. 7

Table 3: Feature extractor ablation at T = 20 on CIFAR-100, TinyImageNet, and ImageNet-Subset. All extractors are non-trained and combined with SLDA heads; ensembling sizes and any extractorspecific hyperparameters are tuned per variant. Mean ± std over 5 seeds. TinyImageNet

ImageNet-Subset

Ā

CIFAR-100 AT

Ā

AT

Ā

AT

Random ResNet-18 (BN) Random ResNet-18 (LN) Random VGG-13 Patch-RNN Conv + Patch-RNN Scattering transform

39.5 ± 0.2 40.2 ± 0.2 40.0 ± 0.2 21.4 ± 0.1 26.9 ± 0.1 34.7 ± 0.0

31.0 ± 0.2 28.8 ± 0.2 28.3 ± 0.2 13.6 ± 0.1 17.4 ± 0.1 22.9 ± 0.0

18.4 ± 0.3 21.5 ± 0.2 26.8 ± 0.2 9.7 ± 0.1 12.6 ± 0.1 23.2 ± 0.0

11.7 ± 0.1 14.0 ± 0.1 17.5 ± 0.2 5.0 ± 0.1 6.5 ± 0.1 14.4 ± 0.0

38.7 ± 0.4 41.1 ± 0.3 38.5 ± 0.2 12.7 ± 0.2 15.7 ± 0.4 28.9 ± 0.0

28.3 ± 0.5 27.4 ± 0.3 24.2 ± 0.2 5.9 ± 0.1 7.3 ± 0.3 16.2 ± 0.1

BiRC2D (CIRCLE)

55.9 ± 0.3

45.3 ± 0.3

40.9 ± 0.2

30.8 ± 0.2

53.5 ± 0.4

39.7 ± 0.3

Extractor

The results show that BiRC2D is the only tested untrained extractor that consistently supports strong cold-start EFCIL. Random CNNs produce non-trivial but much weaker features, simple Patch-RNN variants show that sequentializing image patches is insufficient, and scattering features remain below BiRC2D despite being the strongest non-reservoir alternative. Thus, CIRCLE’s gains come from spatially structured reservoir features, not merely from using any fixed random representation. 4.3.2

Classification Head

We ablate the classification head while keeping the BiRC2D feature extractor fixed. We compare SLDA with several streaming analytic alternatives: Euclidean NCM, cosine NCM, ridge regression with recursive least squares (RLS), diagonal LDA, a FeCAM-style Mahalanobis classifier, and QDA with class-specific covariance estimates. These heads are described in the Appendix C.4. SLDA is consistently the strongest analytic head; full results are in Table 4. Prototype heads lose second-order information, diagonal LDA is too restrictive, and class-specific covariance methods such as FeCAM-Mahalanobis and QDA are less stable because per-class covariance estimation is data-hungry under cold start. Shared-covariance SLDA pools covariance across classes and gives the best balance between second-order modeling and statistical stability. Table 4: Classification head ablation at T = 20 on CIFAR-100, TinyImageNet, and ImageNet-Subset. All heads are paired with the same BiRC2D feature extractor and ensembling scheme. Mean ± std over 5 seeds. CIFAR-100

4.3.3

TinyImageNet

ImageNet-Subset

Ā

AT

Ā

AT

Ā

AT

Euclidean NCM Cosine NCM Ridge RLS Diagonal LDA FeCAM Mahalanobis QDA (class-spec. cov)

23.56 ± 0.2 23.48 ± 0.2 50.97 ± 0.1 24.72 ± 0.1 49.36 ± 0.1 48.04 ± 0.1

14.96 ± 0.1 14.90 ± 0.1 37.76 ± 0.2 15.90 ± 0.0 39.40 ± 0.1 37.58 ± 0.2

14.09 ± 0.1 14.06 ± 0.0 13.60 ± 7.4 15.19 ± 0.1 27.41 ± 0.9 30.78 ± 0.3

7.91 ± 0.0 7.90 ± 0.0 5.49 ± 3.3 8.63 ± 0.1 19.34 ± 0.7 21.89 ± 0.3

25.29 ± 0.7 25.26 ± 0.7 48.55 ± 0.1 25.92 ± 0.8 21.58 ± 2.2 34.51 ± 1.2

13.83 ± 0.4 13.81 ± 0.4 32.96 ± 0.4 14.31 ± 0.6 11.18 ± 1.4 21.14 ± 0.7

SLDA (CIRCLE)

55.9 ± 0.3

45.3 ± 0.3

40.9 ± 0.2

30.8 ± 0.2

53.5 ± 0.4

39.7 ± 0.3

Head

Ensembling: Feature Dimension vs. Prediction Dimension

CIRCLE uses a fixed budget of n = km reservoir instances, partitioned into k groups of size m. The group size m controls feature ensembling: the m reservoir embeddings within each group are concatenated and passed to one SLDA head. The number of groups k controls prediction ensembling: the k independently trained heads are averaged at the probability level. Thus, increasing m gives each head a richer representation, while increasing k increases prediction-level diversity. We ablate both axes on CIFAR-100 at T = 20 to verify that the two ensembling mechanisms are complementary and that balanced groupings perform best under a fixed reservoir budget. Figure 2 shows that both ensembling axes contribute. For fixed feature-ensembling size m, increasing the number of heads k improves incremental accuracy; for fixed k, increasing m also improves performance. The two are complementary: feature ensembling enriches each head’s representation, while prediction ensembling reduces variance across independently drawn reservoir groups. 8

50

45

40 1

3

5

7

m=1 m=4

m=8 m = 12

9

13

11

Incremental accuracy (%)

Incremental accuracy (%)

55

55

50

45 k=1 k=4

40

15

Heads k

1

3

5

7

9

11

m

k

4 1 16

4 52.97 ± 0.31 16 49.35 ± 0.11 1 45.37 ± 0.99

Ā (%)

k=8 k = 12

13

15

Feature ensemble size m

Figure 2: Ensembling ablation on CIFAR-100 at T = 20. Left: varying the number of prediction heads k for fixed feature-ensembling sizes m. Right: varying m for fixed k. The table compares three allocations with the same total reservoir budget mk = 16. The fixed-budget comparison confirms the axes are not interchangeable. With the same total budget mk = 16, the balanced setting m = 4, k = 4 outperforms both m = 1, k = 16 and m = 16, k = 1. Many small heads leave each classifier under-represented, while a single large feature vector removes the benefit of prediction averaging. A balanced allocation thus provides the best trade-off between representation quality and ensemble diversity, consistent with our theoretical results (Appendix B). Takeaway from ablations. CIRCLE’s superior performance is not explained by any single component such as freezing, random features, or ensembling. CIRCLE needs all three design choices: a spatially structured reservoir feature extractor, an SLDA head, and a balanced allocation between feature and prediction ensembling.

5

Discussion and Limitations

CIRCLE targets a specific regime: long-horizon cold-start exemplar-free class-incremental learning. It is not intended as a universal continual learning method. In short-horizon settings, where only a small number of distribution shifts occur, trained-backbone drift-compensation methods such as EFC++ and AdaGauss outperform CIRCLE. CIRCLE’s advantage appears when the task horizon is long enough that repeated representation updates and drift corrections become unreliable or expensive, and no sufficient data exists for strong initial representation learning. However, the method is designed primarily for low-cost streaming training, not necessarily minimal single-sample inference latency. At inference time, it evaluates multiple fixed reservoir instances and multiple SLDA heads; this cost is parallelizable and can be amortized by batching, but latency may be higher than single-backbone or single-head methods in small-batch or memory-constrained settings. The SLDA parameters can be materialized after any update, so the head computation is a set of linear scores; the dominant inference cost is the reservoir ensemble. Reducing this cost through ensemble compression or distillation into a smaller student model is a promising future direction. Our results show that not all data-free features are sufficient. Random CNNs, simple Patch-RNN reservoirs and scattering features do not match the adapted BiRC2D extractor. Thus, CIRCLE depends on a spatially structured reservoir architecture and careful tuning of reservoir dimensionality and ensemble allocation. Extending CIRCLE to other modalities or higher-resolution domains may require similarly structured data-free extractors rather than direct reuse of the current architecture. The theoretical analysis of grouped ensembling should be interpreted as motivation rather than a complete theory of BiRC2D reservoirs. Proposition B.1 formalizes a bias–variance tradeoff under independence and approximation assumptions, and it explains why intermediate group sizes can be preferable to pure feature concatenation or pure prediction averaging. However, the approximation condition is stylized and is not a proof that the actual BiRC2D feature distribution satisfies a particular rate. The empirical ensemble ablation therefore remains essential evidence for the grouping design. Finally, the ImageNet-1k T = 500 experiment is a stress test rather than a fully exhaustive benchmark. We run two seeds because trained-backbone baselines are very expensive at this horizon, and some baselines are truncated or fail numerically before completing the full stream. We report these failures explicitly because they are part of the long-horizon behaviour under study, but the results should be interpreted together with the broader T ∈ {10, 20, 50, 100} evaluation across three datasets. 9

Acknowledgement Augustinas Jučas and Yangchen Pan acknowledges the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-AI/1023] [McIntosh-Smith et al., 2024]. Yangchen Pan acknowledge the support from the Engineering and Physical Sciences Research Council (EPSRC) New Investigator Award under grant reference UKRI2775.

References Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, pages 109–165. Academic Press, 1989. Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, pages 128–135, 1999. Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Lu Yu, Bartłomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. Conference on Computer Vision and Pattern Recognition, 2020. Simone Magistri, Tomaso Trinci, Albin Soutif-Cormerais, Joost van de Weijer, and Andrew D. Bagdanov. EFC++: Elastic feature consolidation with prototype re-balancing for cold start exemplar-free incremental learning. CoRR, abs/2503.10439, 2025. Grzegorz Rypeść, Sebastian Cygert, Tomasz Trzcinski, and Bartłomiej Twardowski. Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning. Conference on Neural Information Processing Systems, 2024. Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. Conference on Computer Vision and Pattern Recognition Workshops, 2020. Keiichi Nakanishi and Terumasa Tokunaga. Bidirectional 2d reservoir computing for image anomaly detection without any training. Nonlinear Theory and Its Applications, IEICE, pages 838–850, 2024. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Jiaxiang Wu, Qing Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, Stanford University, 2017. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015. Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: a python toolbox for class-incremental learning. SCIENCE CHINA Information Sciences, 2023. Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and selfsupervision for incremental learning. Conference on Computer Vision and Pattern Recognition, pages 5871–5880, 2021a. Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. Advances in Neural Information Processing Systems, pages 14306–14318, 2021b. Dipam Goswami, Albin Soutif-Cormerais, Yuyang Liu, Sandesh Kamath, Bartłomiej Twardowski, and Joost Van De Weijer. Resurrecting old classes with new data for exemplar-free continual learning. Conference on Computer Vision and Pattern Recognition, 2024. Alex Gomez-Villa, Dipam Goswami, Kai Wang, Andrew D. Bagdanov, Bartlomiej Twardowski, and Joost van de Weijer. Exemplar-free continual representation learning via learnable drift compensation. ECCV, 2024. Simone Magistri, Tomaso Trinci, Albin Soutif, Joost van de Weijer, and Andrew D. Bagdanov. Elastic feature consolidation for cold start exemplar-free incremental learning. International Conference on Learning Representations, 2024.

10

Zhizhong Li and Derek Hoiem. Learning without forgetting. CoRR, 2016. Huiping Zhuang, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 2022. Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. Ds-al: a dual-stream analytic learning for exemplar-free class-incremental learning. AAAI Conference on Artificial Intelligence, 2024. Dipam Goswami, Yuyang Liu, Bartłomiej Twardowski, and Joost van de Weijer. FeCAM: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. Conference on Neural Information Processing Systems, 2023. Mark McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random projections and pre-trained models for continual learning. Conference on Neural Information Processing Systems, 2023. Gregoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning . Winter Conference on Applications of Computer Vision, pages 3900–3909, 2023. Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networks. Technical report, German National Research Center for Information Technology (GMD), 2001. Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14(11):2531–2560, 2002. Leonard Bereska and Efstratios Gavves. Continual learning of dynamical systems with competitive federated reservoir computing, 2022. E. J. López-Ortiz, M. Perea-Trigo, L. M. Soria-Morillo, F. Sancho-Caparrini, and J. J. Vegas-Olmos. Exploring deep echo state networks for image classification: a multi-reservoir approach. Neural Computing and Applications, 2024. Hanten Chang and Katsuya Futagami. Reinforcement learning with convolutional reservoir computing. Applied Intelligence, 2020. Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. CoRR, abs/1203.1513, 2012. Simon McIntosh-Smith, S. R. Alam, and C. Woods. Isambard-ai: a leadership class supercomputer optimised specifically for artificial intelligence. arXiv preprint arXiv:2410.11199, 2024. Zhiqiang Tong and Gouhei Tanaka. Reservoir computing with untrained convolutional neural networks for image recognition. International Conference on Pattern Recognition, 2018. Andrew R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, pages 930–945, 1993. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 2007. Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in Neural Information Processing Systems, 2008. Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in Neural Information Processing Systems, 2017.

11

The appendix includes the following contents. 1. Section A: provides complete description of our proposed CIRCLE method. 2. Section B: Proposition B.1 characterizes the bias-variance trade-off of our two-level ensemble design introduced in Section 3.2, along with its detailed proof. 3. Section C: provides additional details and results on the experiments. 4. Section D: discusses and lists the hyperparameters for the main experiment.

A

CIRCLE: Additional Details

Reservoir computing uses a fixed recurrent system to map signal inputs into a high-dimensional feature space, while only the readout layer is trained [Jaeger, 2001, Maass et al., 2002]. Most reservoirs, including the canonical instantiation known as the Echo State Network (ESN) [Jaeger, 2001], are suited to sequential data, but images are two-dimensional objects. A simplistic conversion of an image into a one-dimensional pixel stream to be passed into an ESN destroys the local spatial structure, which is why simple flattened reservoir classifiers [López-Ortiz et al., 2024] are weak on non-trivial image benchmarks, as we show in an ablation later in this study. Therefore, more sophisticated reservoir architectures are required for image data. However, the literature on such models is very limited. To the best of our knowledge, the most suitable and relevant reservoir architecture for our work is BiRC2D, introduced by Nakanishi and Tokunaga [2024]. BiRC2D description. We therefore base our feature extractor on BiRC2D [Nakanishi and Tokunaga, 2024] – a bidirectional two-dimensional reservoir architecture designed for image data. The model first divides the input image into patches of a fixed size and flattens the pixels within each patch into a feature vector, producing a feature map whose spatial locations correspond to patches rather than to raw pixels. This feature map is then passed through a stack of BiRC2D layers, each of which is a map RH×W ×Cin → RH×W ×Cout that gives every spatial location access to contextual information from multiple directions. In particular, a single layer reshapes its input into sequences along four directions – left-to-right, right-to-left, top-to-bottom, and bottom-to-top – runs each sequence through a fixed ESN reservoir, and concatenates the four directional outputs along the channel dimension to recover a spatial map. To improve spatial awareness, the full model runs several such stacks in parallel, each operating on a different patch size, and concatenates their outputs at the end. Our additions. The original BiRC2D model was proposed for image anomaly detection, with the output of the model keeping the spatial H × W feature map structure, required by the context of the original study. However, our task is different: we require a single image-level embedding vector (not a 2D grid) for class-incremental classification. Furthermore, our study concerns general images, as opposed to analyzing only spectral imaging in [Nakanishi and Tokunaga, 2024]. Based on these differences, as well as empirical experimentation, we adapt the architecture in four ways. First, before applying the reservoir blocks, we pass the image through a stack of small and random convolutional layers, inspired by Tong and Tanaka [2018]. This gives the reservoir local low-level features rather than raw pixels. Second, we use ReLU activations inside the reservoir blocks instead of hyperbolic tangent activations. Third, we initialize the fixed reservoir weights using Kaiming uniform initialization, matching the ReLU nonlinearity, as opposed to using Gaussian initialisations. Finally, after the final reservoir block, we concatenate across all spatial locations to obtain a single vector and apply a fixed random upscaling layer to obtain a high-dimensional embedding, similar to how it is done in RanPAC [McDonnell et al., 2023] and ACIL [Zhuang et al., 2022]. Multiple seeds. We denote ϕs : X → Rdr to be the reservoir feature extractor instantiated with random seed s. The seed determines all fixed random weights in the convolutional stack, reservoir blocks, and final upscaling layer. No parameter of ϕs is trained. Our model in detail. Algorithms 1, 2 and 3 give the training and prediction procedures of CIRCLE. Training takes a stream S = {(xt , yt )}N t=1 presented in any order. Given n reservoir seeds, CIRCLE partitions them into k disjoint groups G = {Gi }ki=1 , with |Gi | = m = n/k. The algorithms are written for single samples; in practice, feature extraction and accumulator updates are vectorised over minibatches. 12

Algorithm 1 CIRCLE – training Require: Stream S = {(xt , yt )}N t=1 Require: Reservoir groups G = {Gi }ki=1 , with |Gi | = m 1: for i = 1, . . . , k do 2: Initialise head hi

Ni,c ← 0, Ti,c ← 0 for all classes c Mi ← 0 3: end for 4: for (x, y) ∈ S do 5: for i = 1, . . . , k do 6: zi ← ΦGi (x) 7: Ni,y ← Ni,y + 1 8: Ti,y ← Ti,y + zi

▷ ΦGi (x) = ∥s∈Gi ϕs (x)

9: Mi ← Mi + zi zi⊤ 10: end for 11: end for

Algorithm 2 CIRCLE – prediction Require: Test sample x Require: Reservoir groups G = {Gi }ki=1 Require: Heads {hi }ki=1 with accumulators {Ni,· , Ti,· , Mi }ki=1 1: for i = 1, . . . , k do 2: Materialise {µi,c }, Σi,λ , and {πi,c } from (Ni,· , Ti,· , Mi ) ▷ Use Alg. 3 3: zi ← ΦGi (x) ▷ ΦGi (x) = ∥s∈Gi ϕs (x) 4: for each observed class c do 1 ⊤ −1 5: ℓi,c ← [hi (zi )]c ▷ ℓi,c ← zi⊤ Σ−1 i,λ µi,c − 2 µi,c Σi,λ µi,c + log πi,c 6: end for 7: pi ← softmax(ℓi ) 8: end for Pk 1 9: return p̂(x) = k i=1 pi

B

Proof for Proposition B.1

Definitions and notations. Let x be some arbitrarily fixed input and η(x) ∈ ∆C denote the target PC conditional label distribution over C classes, where ∆C = {p ∈ RC : pc ≥ 0, c=1 pc = 1}. Let n = km be the total number of reservoir instantiations, partitioned into k disjoint groups G1 , . . . , Gk , def each of size m. For each group Gj , define zj (x) = s∈G ϕs (x), where ϕs is the fixed random j reservoir feature extractor instantiated with seed s, and ∥ denotes feature concatenation. Let hj be the classification head trained on the features from group Gj , and define the group-level predicted def (m) probability vector pj (x) = softmax(hj (zj (x))) ∈ ∆C . The grouped ensemble prediction is h i Pk def (m) (m) p̄k,m (x) = k1 j=1 pj (x). Define mean prediction µm (x) = E pj (x) , where the expectation is over the random reservoir initializations. Proposition B.1. [Bias-variance decomposition for grouped reservoir ensembles] Assume that, (m) (m) conditional on the training data, the random vectors p1 (x), . . . , pk (x) are independent and identically distributed with respect to the reservoir initialization randomness. Define the squared def 2 bias Bm (x) = ∥µm (x) − η(x)∥2 . Then h i m 2 E ∥p̄k,m (x) − η(x)∥2 ≤ Bm (x) + . (1) n Furthermore, if the group-level squared bias obeys the approximation condition Bm (x) ≤ Am−2α 13

(2)

Algorithm 3 M ATERIALISE: obtaining closed-form LDA classifier from accumulated statistics Require: P Head accumulators (nc , Tc , M ) for c in observed classes 1: N ← c nc ; C ← |{c : nc > 0}| 2: µc ← Tc /nc ▷ class means 3: πc ← nc /N ▷ class priors P ⊤ 4: S ← M − ▷ pooled within-class scatter c nc µc µc 5: Σλ ← S/(N − C) + λI ▷ regularised covariance 6: return {µc }, Σλ , {πc }

for some constants A > 0 and α > 0, then i h m 2 E ∥p̄k,m (x) − η(x)∥2 ≤ Am−2α + . n 1

The right-hand side is minimized over positive real m at m⋆ = (2αAn) 2α+1 . Thus, under the approximation condition above, the bound is optimized by an intermediate group size rather than necessarily by either m = 1 or m = n. Proof. Fix x throughout the proof and suppress the explicit dependence on x to simplify notation. Write (m) pj = pj (x), p̄ = p̄k,m (x), µm = E[pj ], η = η(x). By definition, k

p̄ =

1X pj . k j=1

We decompose the prediction error as p̄ − η = (p̄ − µm ) + (µm − η). Taking squared Euclidean norm gives ∥p̄ − η∥22 = ∥p̄ − µm ∥22 + ∥µm − η∥22 + 2⟨p̄ − µm , µm − η⟩. Taking expectation over the reservoir randomness, E∥p̄ − η∥22 = E∥p̄ − µm ∥22 + ∥µm − η∥22 + 2 ⟨E[p̄ − µm ], µm − η⟩ . Now

 E[p̄] = E 

k 1X

k j=1



k

pj  =

1X E[pj ] = µm , k j=1

so E[p̄ − µm ] = 0. Therefore the cross term vanishes, and E∥p̄ − η∥22 = E∥p̄ − µm ∥22 + ∥µm − η∥22 . By definition,

∥µm − η∥22 = Bm .

It remains to compute the variance term. Define ξj = pj − µm . Then E[ξj ] = E[pj − µm ] = E[pj ] − µm = 0, and k

p̄ − µm =

1X ξj . k j=1

14

Therefore

2

k

1X E∥p̄ − µm ∥22 = E ξj k j=1

. 2

Expanding the squared norm, * + k k k k X X 1 1 XX E∥p̄ − µm ∥22 = 2 E  ξi , ξj  = 2 E [⟨ξi , ξj ⟩] . k k i=1 j=1 i=1 j=1 For i = j, def

2 E [⟨ξi , ξi ⟩] = E∥ξi ∥22 = E∥pi − µm ∥22 = σm , which is the single-group prediction variance at input x. For i ̸= j, independence of pi and pj implies independence of ξi and ξj . Hence " C # C X X E [⟨ξi , ξj ⟩] = E ξi,c ξj,c = E[ξi,c ξj,c ]. c=1

c=1

By independence, E[ξi,c ξj,c ] = E[ξi,c ]E[ξj,c ] = 0 · 0 = 0. Therefore, E [⟨ξi , ξj ⟩] = 0

for all i ̸= j.

E∥p̄ − µm ∥22 =

2 1 X 2 σm σ = . m k 2 j=1 k

Thus

k

Combining the bias and variance terms yields E∥p̄ − η∥22 = Bm +

2 σm . k

2 We now prove the bound σm ≤ 1. Since pj ∈ ∆C , every coordinate of pj is nonnegative and PC c=1 pj,c = 1. Therefore !2 C C X X 2 2 ∥pj ∥2 = pj,c ≤ pj,c = 1. c=1

Also,

c=1

2 σm = E∥pj − µm ∥22 .

Expanding this expression,   2 σm = E ∥pj ∥22 − 2⟨pj , µm ⟩ + ∥µm ∥22 . Because µm = E[pj ], E⟨pj , µm ⟩ = ⟨E[pj ], µm ⟩ = ⟨µm , µm ⟩ = ∥µm ∥22 . Hence

2 σm = E∥pj ∥22 − ∥µm ∥22 .

Since E∥pj ∥22 ≤ 1 and ∥µm ∥22 ≥ 0,

2 σm ≤ 1. 2 Also σm ≥ 0 because it is the expectation of a squared norm. Therefore 2 0 ≤ σm ≤ 1.

Substituting this into the exact decomposition gives E∥p̄ − η∥22 = Bm + 15

2 σm 1 ≤ Bm + . k k

Since n = km, we have k = n/m, and therefore 1 m = . k n Thus E∥p̄k,m (x) − η(x)∥22 ≤ Bm (x) +

m . n

Finally, suppose Bm (x) ≤ Am−2α for some A > 0 and α > 0. Then E∥p̄k,m (x) − η(x)∥22 ≤ Am−2α + Define R(m) = Am−2α +

m . n

m n

for m > 0. Its derivative is R′ (m) = −2αAm−2α−1 +

1 . n

Setting R′ (m) = 0 gives 1 . n

2αAm−2α−1 = Equivalently,

m2α+1 = 2αAn. Thus the unique stationary point is 1

m⋆ = (2αAn) 2α+1 . Moreover, R′′ (m) = 2α(2α + 1)Am−2α−2 > 0 for all m > 0. Therefore m⋆ is the unique minimizer of R(m) over positive real m. Remark B.2. The decomposition holds for any Bm (x). The bound Bm (x) ≤ Am−2α is a stylized model of how bias may decrease as more independent random reservoirs are concatenated. Such polynomial rates are standard in approximation theory and align with classical random-feature results, e.g., O(1/m) rates for shallow networks [Barron, 1993] and convergence bounds for random features in kernel methods [Rahimi and Recht, 2007, 2008, Rudi and Rosasco, 2017]. We use Am−2α only as a phenomenological model, not a theorem for BiRC2D. Interpretation. The result explains the need for both feature concatenation and prediction averaging. Increasing m enriches each head’s representation (captured by Bm (x)), but reduces the number of heads to k = n/m, so prediction averaging contributes at most 1/k = m/n variance. Thus, the grouped ensemble trades bias Bm (x) against variance ≤ m/n. Under Bm (x) ≤ Am−2α , the bound becomes Am−2α + m n : the first term decreases with m, the second increases. Hence, m = 1 (prediction-only) leaves high bias, m = n (feature-only) removes variance reduction, and an intermediate m balances both, explaining the benefit of a balanced ensemble.

C

Additional Experimental Details

This section provides additional details and results behind the performed experiments. All ablation baselines are designed to preserve the main online-learning setting of CIRCLE. In particular, the feature-extractor ablations replace only the frozen image representation, while keeping the same analytic SLDA head and the same ensembling mechanism. Conversely, the head ablations replace only the final analytic classifier, while keeping the BiRC2D feature extractor fixed. All variants are updated in a streaming manner and are trained without gradient-based optimisation of the feature extractor or classification head. The code for all experiments can be found at https://anonymous.4open.science/r/circle. 16

C.1

Detailed Experimental Setup

Datasets and protocol. We evaluate on three standard CIL benchmarks: CIFAR-100 (100 classes, 32×32) [Krizhevsky, 2009], TinyImageNet (200 classes, 64×64) [Wu et al., 2017], and ImageNetSubset (100 classes from ImageNet-1k at 224×224) [Russakovsky et al., 2015]. For long-horizon experiments, we additionally use the full ImageNet-1k. All experiments are conducted in the coldstart exemplar-free regime: classes are split evenly across T tasks, no replay buffer is used, and no pre-trained weights or external data are allowed. For the main comparison, we consider T ∈ {10, 20, 50, 100}. The settings T = 10 and T = 20 are standard in the EFCIL literature [Magistri et al., 2024, Rypeść et al., 2024]. The longer horizon T = 50 has appeared occasionally in warm-start studies (e.g., DS-AL [Zhuang et al., 2024]) but, to our knowledge, not in cold-start EFCIL on standard benchmarks. The T = 100 setting is likewise unexplored in cold-start CIL; for CIFAR-100 and ImageNet-Subset, it corresponds to one class per task, i.e., an extreme distribution-shift regime. For full ImageNet-1k, we further evaluate T = 500 (two classes per task), following Zhuang et al. [2024]. Results are averaged over 5 seeds, except for T = 500, where we use 2 seeds due to the high cost of training-based baselines. Our protocol for splitting a dataset into multiple tasks is as follows. Assume we wish to split a dataset into T tasks, where each task has C classes. We fix a data seed (we use seed 0) using which we shuffle the classes. Then we assign the samples from the first C classes from the list into T1 , the next C classes into T2 , etc. For different splits (i.e., different T values), we still use the same shuffling seed 0, this way the splits are consistent. Across multiple random instantiations of the same experiment we do not change the data seed, i.e., the underlying data split remains the same across re-runs of the experiment. The varied seeds vary mostly model parameters, so model weight initializations, but also some data parameters, namely the augmentations for models that use them. Baselines. We compare against representative EFCIL methods across key design axes. Driftcompensation / distillation: EFC++ [Magistri et al., 2025], AdaGauss [Rypeść et al., 2024], ADC [Goswami et al., 2024], LwF [Li and Hoiem, 2016]. Prototype rehearsal / augmentation: PASS [Zhu et al., 2021a], IL2A [Zhu et al., 2021b]. Frozen-backbone analytic methods (adapted to cold-start by training on T1 and freezing thereafter): ACIL [Zhuang et al., 2022], DS-AL [Zhuang et al., 2024], FeCAM [Goswami et al., 2023], FeTrIL [Petit et al., 2023]. All trained-backbone baselines use ResNet-18, following standard EFCIL practice. Implementations for ACIL, DS-AL, FeTrIL, IL2A, LwF, and PASS are from PyCIL [Zhou et al., 2023]; others use official code. Hyperparameter tuning. We tune all baselines for each (T, dataset) configuration. This is crucial at T = 50 and T = 100, where prior hyperparameters perform poorly. Our protocol thus provides a fair assessment at long horizons. Details and optimal values for all methods, including CIRCLE, are given in Appendix D. Metrics. We report average incremental accuracy (Ā) and final accuracy (AT ). For efficiency, we also report total wall-clock training time, measured by running each experiment in isolation on a single NVIDIA L40S GPU. Hardware and compute. The experiments in Table 1 were tuned and executed on NVIDIA A100 GPUs (40 GB memory per GPU) in a SLURM cluster, with the experiment lengths having strongly varied depending on the model that was used. However, no single experiment took more than 24 hours to finish. The hyperparameter tuning for the ablation experiments were performed by utilizing eight B200 GPUs (180 GB memory per GPU), used for less than 48 hours in total. Finally, the duration experiments were performed on 8 NVIDIA L40S GPUs (48 GB memory per GPU), each experiment taking exactly as long as specified in Table 2. Licensing of the data and code. CIFAR-100 does not include a specified license, ImageNet-1k, ImageNet-Subset and TinyImageNet are used under the ImageNet terms of access, which restrict use to non-commercial research and educational purposes. To obtain baseline results, we incorporate the codebases of the baselines into our project, which is allowed under MIT license which is used by the codebases of EFC++ [Magistri et al., 2025], ADC [Goswami et al., 2024], FeCAM [Goswami et al., 2023] and PyCIL (PyCIL covers LwF [Li and Hoiem, 2016], PASS [Zhu et al., 2021a], IL2A [Zhu et al., 2021b], ACIL [Zhuang et al., 2022], DS-AL [Zhuang et al., 2024] and FeTRiL [Petit et al., 2023]). The codebase of AdaGauss is the only one that provides no license, with the arxiv version of the paper being licensed under CC BY 4.0. 17

C.2

Additional Results on Performance vs Wall-clock Time

Figure 3 provides an additional view of the accuracy–time trade-off. For any fixed wall-clock budget, it shows how far each method can progress through the incremental stream and what accuracy it achieves at that point. In most settings, CIRCLE completes the entire T = 20 stream before most baselines have finished training even on the first task. This is particularly important when comparing against freeze-after-first-task methods: although these methods avoid later backbone updates, they still require an expensive first-task training stage before incremental learning can begin. CIRCLE has no such warm-up stage and can update incrementally from the very first sample obtained.

Incremental accuracy (%)

ACIL

FeCAM

CIFAR-100

FeTRIL

AdaGauss

ImageNet-Subset

EFC++

CIRCLE

Tiny-ImageNet

80 60 40 20 100

101

102

Time (min, log)

101

102

Time (min, log)

101

102

Time (min, log)

Figure 3: Accuracy-computation time trade-off at T = 20. Each panel plots incremental test accuracy up to the current task against average wall-clock time of training up to the current task. C.3

Feature-extractor Ablations

Table 5 summarises the feature extractors used in the ablation study. The feature-extractor ablations test whether CIRCLE’s performance comes from the BiRC2D representation itself or merely from using a frozen high-dimensional image embedding. The random ResNet-18, ResNet-18-LN, and VGG-13 baselines compare against standard convolutional architectures without training. Patch-RNN and Conv+Patch-RNN test simpler reservoir-style alternatives that sequentialise the image, with and without local random convolutional preprocessing. The scattering transform provides a deterministic, theory-driven non-trained baseline. C.4

Classification-head Ablations

Table 6 summarises the analytic classification heads used in the ablation study. All heads ignore classes that have not yet been observed. Head-specific hyperparameters, such as regularisation, shrinkage, and temperature, are tuned separately for each variant, together with the ensembling sizes. The classification-head ablations test whether the gains come from the SLDA readout or simply from using any analytic streaming classifier. Mean-based heads test whether class means alone are sufficient. Ridge regression tests a discriminative linear readout. Diagonal LDA tests a cheaper covariance model than regular LDA. FeCAM-style Mahalanobis and QDA test more flexible classspecific covariance models. SLDA provides the best trade-off in this setting because it uses a full covariance estimate shared across classes, which captures feature correlations while avoiding the cold-start instability of estimating a separate covariance matrix for every class.

D

Hyperparameters

D.1

Hyperparameter Tuning Process

We tune hyperparameters separately for each dataset–split–algorithm tuple. That is, each algorithm on each dataset split is assigned its own hyperparameter configuration, selected using validation set performance. This is motivated by the fact that the different extreme splits (such as T = 10 vs T = 100) have very differing task dynamics (one split has very large tasks, another has very small tasks), and this likely means that very different hyperparameter sets would be optimal for both cases, 18

Table 5: Feature-extractor baselines used in the ablation study. All feature extractors are frozen and are followed by the same SLDA classification head. Extractor

Description

Purpose of the ablation

Random ResNet-18 (BN)

A standard ResNet-18 with randomly initialised and frozen weights. BatchNorm layers are retained, matching the architecture used by the trained-backbone baselines.

Tests whether the gains of CIRCLE can be explained simply by using a high-capacity convolutional architecture, even without training.

Random ResNet-18 (LN)

A randomly initialised and frozen ResNet18 variant in which BatchNorm is replaced by LayerNorm.

Controls for possible pathologies of BatchNorm in randomly initialised networks. This separates the effect of the ResNet architecture from the effect of BatchNorm statistics.

Random VGG-13

A randomly initialised and frozen VGG13-style convolutional network.

Provides a comparable-capacity convolutional baseline without residual connections or BatchNorm. Also tests if a different type of CNN could perform better.

PatchRNN

The image is divided into patches, the patches are flattened, and the resulting sequence is processed by a single reservoir recurrent model. The final reservoir (hidden) state is used as the image embedding.

Tests a simple reservoir-based image representation obtained by sequentialising the image. This is the natural patch-level extension of earlier approaches that feed flattened image pixels into an ESN-style reservoir.

Conv+Patch- A small fixed random convolutional stack RNN is first applied to the image. The resulting spatial feature map is then divided into patches and passed through a reservoir recurrent model.

Tests whether adding local random convolutional preprocessing is sufficient to close the gap between a simple patch reservoir and BiRC2D. This is approach is very stronly motivated by convolutional reservoir-computing approaches that combine fixed random CNN features with reservoir dynamics [Chang and Futagami, 2020].

Scattering transform

Tests CIRCLE against a strong non-random, but non-trained image representation whose filters are chosen from wavelet theory rather than learned from data.

A deterministic wavelet-scattering feature extractor. It applies fixed wavelet filters, modulus nonlinearities, and averaging operations to produce a CNN-like representation without learned weights. Taken from the work by Bruna and Mallat [2012].

hence we tune across splits. We use 20% of data from each class for the validation dataset, on which the models are evaluated when selecting the results. After selection of the optimal hyperparameters, both the validation and train sets are merged into a new training set and the model is trained on the full set, and tested on the testing set. Hyperparameter tuning uses a held-out validation split for model selection only. After selecting hyperparameters, train and validation data are merged, and no test data are used for selection. For the baseline methods, we use a few-stage randomized grid-search procedure. In the first stage, we tune the main optimization hyperparameters together, such as learning rate, number of training epochs, and batch size, while keeping the remaining hyperparameters fixed to sensible values, typically chosen from the corresponding literature or recommended defaults from the implementations. After this stage, we retain a small set of the best-performing configurations according to validation performance for every data–split–algorithm tuple. Starting from these configurations, we then tune the remaining method-specific hyperparameters using an additional randomized grid search. In the cases where a method still substantially underperforms relative to other baselines or previously reported results, we perform an additional, narrower search around the few best configurations found so far. For our method, we follow the same general procedure, but separate the search into two stages. We first set the total ensemble size to one and tune the core hyperparameters of the model using randomized grid search. This allows us to identify strong single-model configurations without the additional cost of ensembling. After selecting the best-performing single-model configurations, we tune the ensemble-related hyperparameters, together with the reservoir’s output dimensionality. Throughout this process, we always ensure that the total number (summed over trainable and non19

Table 6: Classification-head baselines used in the ablation study. All heads consume the same BiRC2D features and are updated from streaming sufficient statistics. Let z ∈ Rd denote the frozen feature vector; nc the number of observed examples from class c; µc the empirical class mean; and πc the empirical class prior. Head

Description

Main modelling assumption

Euclidean NCM

Nearest-class-mean classifier using Euclidean distance. For each class, the head stores the class sum and count, computes µc , and scores a test feature by

Each class is represented only by its mean. Class covariance and feature correlations are ignored. Used by ADC [Goswami et al., 2024].

sc (z) = 2z ⊤ µc − ∥µc ∥22 , which is equivalent to negative squared Euclidean distance up to a class-independent constant. Cosine NCM

Nearest-class-mean classifier using cosine similarity. Features are first ℓ2 -normalised before being accumulated into class prototypes. The final prototype is normalised again, and the score is z ⊤ µc sc (z) = τ , ∥z∥2 ∥µc ∥2 where τ is a temperature parameter.

Only angular similarity to the class prototype matters. Feature norms and covariance structure are ignored.

Ridge RLS

Ridge-regression classifier with one-hot class targets. The head accumulates X X A = λI + zi zi⊤ , B= zi yi⊤ ,

Learns a linear readout by least squares, with ridge regularisation. It does not explicitly model class-conditional covariance. Used by ACIL and DS-AL [Zhuang et al., 2022, 2024].

i −1

and solves W = A Diagonal LDA

B. Scores are s(z) = z ⊤ W .

Shared-covariance LDA with a diagonal covariance approximation. The head stores class means and a pooled per-feature variance estimate. Scores use the diagonal precision: sc (z) = z ⊤ Σ−1 diag µc −

FeCAM Mahalanobis

QDA, classspecific covariance

i

Models per-feature variances but ignores correlations between feature dimensions.

1 ⊤ −1 µc Σdiag µc + log πc . 2

FeCAM-style Mahalanobis classifier with class-specific covariance estimates. Each class is represented by a mean and a regularised covariance matrix. Scores are based on the negative Mahalanobis distance, sc (z) = −α(z − µc )⊤ Σ−1 c (z − µc ).

Uses class-specific secondorder structure, but the covariance estimates can be noisy when few examples are available per class. Used by FeCAM [Goswami et al., 2023].

Quadratic discriminant analysis with one covariance matrix per class. The head estimates µc and Σc , applies ridge regularisation and shrinkage toward the shared covariance, and scores by

Allows each class to have its own full covariance, giving a more flexible but more datahungry classifier than SLDA.

1 1 sc (z) = − (z − µc )⊤ Σ−1 log |Σc | + log πc . c (z − µc ) − 2 2

trainable) parameters in our model does not differ by more than 5% from a ResNet-18 that the baselines use, so that the model remains exactly comparable against the baselines (see the parameter counts in Table 2). Here too, the final hyperparameter configuration for each dataset–split–algorithm tuple is the one achieving the best validation performance. 20

D.2

Selected Hyperparameters

Below, we list the selected optimal hyperparameters for the main results in Table 1. The remaining hyperparameters, for the ablation and other experiments can be directly found in our released codebase. CIFAR-100, T = 10. 1. AdaGauss: batch size 128; α = 1; λ = 10; learning rate 0.1; adapter learning rate 0.01; backbone learning rate 0.01; epochs 440; s = 64; singular-value fraction 0.95; τ = 2; weight decay 0.0005. 2. EFC++: batch size 128; balanced epochs 50; balanced learning rate 0.001; damping 0.1; first-task epochs 200; later-task epochs 200; λ = 10; prototype update -0.2. 3. ADC: batch size 256; epochs 800; initial epochs 800; initial learning rate 0.05; λ = 10; learning rate 0.05; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 128; initial batch size 64; buffer size 8192; decay 0.18; γ = 0.28; initial epochs 258; initial learning rate 0.0408; initial weight decay 0.0005. 5. FeCAM: batch size 128; α1 = 0.6; α2 = 0.6; β = 0.5; initial epochs 400; initial learning rate 0.02; initial weight decay 0.0005. 6. FeTRIL: batch size 32; epochs 200; initial epochs 200; initial learning rate 0.02; initial weight decay 0.0005; learning rate 0.02; temperature 2. 7. IL2A: batch size 128; epochs 200; γ = 0.1; learning rate 0.005; augmentation ratio 2.5; step size 45; temperature 2; softmax temperature 0.1. 8. PASS: batch size 16; epochs 200; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.001; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 128; epochs 220; initial epochs 220; initial learning rate 0.02; initial learning-rate decay 0.5; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.5; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 128; initial batch size 64; buffer size 8192; compensation ratio 0.6; γ = 0.336; compensation γ = 0.1; initial epochs 258; initial learning rate 0.04; initial weight decay 0.0005; Scheduler: multi-step, milestones 120 and 140, with scheduler γ = 0.18; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 480; head ensemble size 8; feature ensemble size 8; reservoir output dimension 1100; CNN-stem channels 16 and 32; CNN-stem kernels 3 and 3; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 2 and 4; sparsity 0.9; spectral radius 0.9. CIFAR-100, T = 20. 1. AdaGauss: batch size 256; α = 1; λ = 10; learning rate 0.1; adapter learning rate 0.01; backbone learning rate 0.01; epochs 220; s = 64; singular-value fraction 0.95; τ = 2; weight decay 0.0005. 2. EFC++: batch size 64; balanced epochs 25; balanced learning rate 0.0002; damping 0.1; first-task epochs 200; later-task epochs 200; λ = 10; prototype update -0.2. 3. ADC: batch size 256; epochs 400; initial epochs 400; initial learning rate 0.05; λ = 10; learning rate 0.05; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 128; initial batch size 128; buffer size 8192; decay 0.1; γ = 0.1; initial epochs 320; initial learning rate 0.02; initial weight decay 0.0005. 5. FeCAM: batch size 128; α1 = 1; α2 = 1; β = 0.5; initial epochs 400; initial learning rate 0.02; initial weight decay 0.0005. 6. FeTRIL: batch size 64; epochs 200; initial epochs 200; initial learning rate 0.02; initial weight decay 0.0005; learning rate 0.02; temperature 2. 21

7. IL2A: batch size 64; epochs 200; γ = 0.1; learning rate 0.001; augmentation ratio 2.5; step size 45; temperature 2; softmax temperature 0.1. 8. PASS: batch size 32; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.001; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 128; epochs 110; initial epochs 110; initial learning rate 0.02; initial learning-rate decay 0.5; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.5; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 128; initial batch size 128; buffer size 8192; compensation ratio 0.6; γ = 0.1; compensation γ = 0.1; initial epochs 320; initial learning rate 0.023; initial weight decay 0.0005; Scheduler: multi-step, milestones 120 and 140, with scheduler γ = 0.1; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 480; head ensemble size 8; feature ensemble size 8; reservoir output dimension 1100; CNN-stem channels 16 and 32; CNN-stem kernels 3 and 3; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 2 and 4; sparsity 0.9; spectral radius 0.9. CIFAR-100, T = 50. 1. AdaGauss: batch size 128; α = 1; λ = 10; learning rate 0.1; adapter learning rate 0.005; backbone learning rate 0.005; epochs 220; s = 64; singular-value fraction 0.95; τ = 2; weight decay 0.0005. 2. EFC++: batch size 16; balanced epochs 19; balanced learning rate 0.0002; damping 0.1; first-task epochs 150; later-task epochs 150; λ = 10; prototype update -0.2. 3. ADC: batch size 64; epochs 300; initial epochs 300; initial learning rate 0.025; λ = 10; learning rate 0.025; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 64; initial batch size 64; buffer size 8192; decay 0.18; γ = 0.28; initial epochs 218; initial learning rate 0.0432; initial weight decay 0.0005. 5. FeCAM: batch size 64; α1 = 1; α2 = 1; β = 0.7; initial epochs 272; initial learning rate 0.036; initial weight decay 0.0005. 6. FeTRIL: batch size 32; epochs 200; initial epochs 200; initial learning rate 0.005; initial weight decay 0.0005; learning rate 0.005; temperature 2. 7. IL2A: batch size 64; epochs 150; γ = 0.1; learning rate 0.00025; augmentation ratio 2.5; step size 45; temperature 2; softmax temperature 0.1. 8. PASS: batch size 32; epochs 75; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.00075; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 64; epochs 110; initial epochs 110; initial learning rate 0.015; initial learning-rate decay 0.25; initial weight decay 0.0005; λ = 3; learning rate 0.015; learningrate decay 0.25; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 64; initial batch size 64; buffer size 8192; compensation ratio 0.6; γ = 0.336; compensation γ = 0.1; initial epochs 218; initial learning rate 0.0432; initial weight decay 0.0005; Scheduler: multi-step, milestones 120 and 140, with scheduler γ = 0.18; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 480; head ensemble size 8; feature ensemble size 8; reservoir output dimension 1100; CNN-stem channels 16 and 32; CNN-stem kernels 3 and 3; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 2 and 4; sparsity 0.9; spectral radius 0.9. CIFAR-100, T = 100. 1. AdaGauss: batch size 96; α = 1; λ = 2; learning rate 0.125; adapter learning rate 0.01; backbone learning rate 0.01; epochs 55; s = 64; singular-value fraction 0.95; τ = 2.5; weight decay 0.0005. 22

2. EFC++: batch size 16; balanced epochs 24; balanced learning rate 0.00032; damping 0.05; first-task epochs 172; later-task epochs 112; λ = 8; prototype update -0.1. 3. ADC: batch size 256; epochs 360; initial epochs 360; initial learning rate 0.08; λ = 6; learning rate 0.08; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 64; initial batch size 32; buffer size 8192; decay 0.108; γ = 0.84; initial epochs 218; initial learning rate 0.0432; initial weight decay 0.0005. 5. FeCAM: batch size 64; α1 = 2; α2 = 0.7; β = 1.05; initial epochs 231; initial learning rate 0.018; initial weight decay 0.0005. 6. FeTRIL: batch size 16; epochs 150; initial epochs 260; initial learning rate 0.0035; initial weight decay 0.0005; learning rate 0.0025; temperature 2.5. 7. IL2A: batch size 32; epochs 150; γ = 0.16; learning rate 0.000175; augmentation ratio 1.5; step size 45; temperature 0.14; softmax temperature 0.14. 8. PASS: batch size 32; epochs 75; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.00075; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 128; epochs 110; initial epochs 110; initial learning rate 0.02; initial learning-rate decay 0.5; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.5; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 64; initial batch size 32; buffer size 8192; compensation ratio 0.6; γ = 1.008; compensation γ = 0.1; initial epochs 218; initial learning rate 0.0432; initial weight decay 0.0005; Scheduler: multi-step, milestones 120 and 140, with scheduler γ = 0.108; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 480; head ensemble size 8; feature ensemble size 8; reservoir output dimension 1100; CNN-stem channels 16 and 32; CNN-stem kernels 3 and 3; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 2 and 4; sparsity 0.9; spectral radius 0.9. TinyImageNet, T = 10. 1. AdaGauss: batch size 192; α = 0.6; λ = 8; learning rate 0.1; adapter learning rate 0.01; backbone learning rate 0.01; epochs 110; s = 64; singular-value fraction 0.95; τ = 2; weight decay 0.0005. 2. EFC++: batch size 80; balanced epochs 62; balanced learning rate 0.00075; damping 0.05; first-task epochs 75; later-task epochs 90; λ = 3; prototype update -0.1. 3. ADC: batch size 256; epochs 320; initial epochs 320; initial learning rate 0.08; λ = 15; learning rate 0.08; temperature 3; weight decay 0.0005. 4. ACIL: incremental batch size 192; initial batch size 256; buffer size 8192; backbone ResNet18 CBAM; decay 0.009; γ = 0.048; initial epochs 239; initial learning rate 0.08925; initial weight decay 0.0005; learning-rate milestones 143 and 203; scheduler MultiStep; warmup epochs 0. 5. FeCAM: batch size 256; α1 = 1; α2 = 0.6; β = 0.3; initial epochs 200; initial learning rate 0.5; initial weight decay 0.0005. 6. FeTRIL: batch size 16; epochs 50; initial epochs 300; initial learning rate 0.1; initial weight decay 0.0002; learning rate 0.0005; temperature 2; weight decay 0.001. 7. IL2A: batch size 64; epochs 100; γ = 0.3; feature-distillation weight 1.5; prototype weight 0.5; learning rate 0.0005; augmentation ratio 1; step size 45; temperature 0.8; softmax temperature 0.8; weight decay 0.0002. 8. PASS: batch size 32; epochs 100; γ = 0.1; feature-distillation weight 15; prototype weight 0.5; learning rate 0.001; step size 45; temperature 0.05; softmax temperature 0.05; weight decay 0.0002. 9. LwF: batch size 64; epochs 320; initial epochs 320; initial learning rate 0.01; initial learningrate decay 0.5; initial weight decay 0.0005; λ = 3; learning rate 0.01; learning-rate decay 0.25; temperature 2.5; weight decay 0.0005. 23

10. DS-AL: incremental batch size 512; initial batch size 128; buffer size 8192; compensation ratio 1; γ = 0.06; compensation γ = 0.13; initial epochs 260; initial learning rate 0.108375; initial weight decay 0.0002; Scheduler: multi-step, milestones 156 and 221, with scheduler γ = 0.009; warmup epochs 5. 11. CIRCLE: reservoir internal dimension 384; head ensemble size 7; feature ensemble size 7; reservoir output dimension 1024; CNN-stem channels 16; CNN-stem kernels 3; leak 0.7; leaky-ReLU slope 0.01; maximum rotation 0; reservoir layers 2; patch sizes 2; sparsity 0.5; spectral radius 0.9. TinyImageNet, T = 20. 1. AdaGauss: batch size 256; α = 1; λ = 10; learning rate 0.1; adapter learning rate 0.01; backbone learning rate 0.01; epochs 110; s = 64; singular-value fraction 0.95; τ = 2; weight decay 0.0005. 2. EFC++: batch size 128; balanced epochs 100; balanced learning rate 0.001; damping 0.1; first-task epochs 100; later-task epochs 100; λ = 10; prototype update -0.2. 3. ADC: batch size 64; epochs 400; initial epochs 400; initial learning rate 0.005; λ = 10; learning rate 0.005; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 64; initial batch size 192; buffer size 8192; backbone ResNet18 CBAM; decay 0.05; γ = 0.18; initial epochs 180; initial learning rate 0.006; initial weight decay 0.0002; learning-rate milestones 108 and 153; scheduler MultiStep; warmup epochs 10. 5. FeCAM: batch size 128; α1 = 1; α2 = 1; β = 0.5; initial epochs 200; initial learning rate 0.5; initial weight decay 0.0005. 6. FeTRIL: batch size 16; epochs 50; initial epochs 300; initial learning rate 0.1; initial weight decay 0.0002; learning rate 0.0005; temperature 2; weight decay 0.001. 7. IL2A: batch size 64; epochs 100; γ = 0.3; feature-distillation weight 1.5; prototype weight 0.5; learning rate 0.0005; augmentation ratio 1; step size 45; temperature 0.8; softmax temperature 0.8; weight decay 0.0002. 8. PASS: batch size 32; epochs 100; γ = 0.1; feature-distillation weight 15; prototype weight 0.5; learning rate 0.001; step size 45; temperature 0.05; softmax temperature 0.05; weight decay 0.0002. 9. LwF: batch size 128; epochs 440; initial epochs 440; initial learning rate 0.02; initial learning-rate decay 0.5; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.5; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 160; initial batch size 128; buffer size 8192; compensation ratio 0.75; γ = 0.18; compensation γ = 0.13; initial epochs 180; initial learning rate 0.009; initial weight decay 0.0002; Scheduler: multi-step, milestones 108 and 153, with scheduler γ = 0.05; warmup epochs 10. 11. CIRCLE: reservoir internal dimension 384; head ensemble size 7; feature ensemble size 7; reservoir output dimension 1024; CNN-stem channels 16; CNN-stem kernels 3; leak 0.7; leaky-ReLU slope 0.01; maximum rotation 0; reservoir layers 2; patch sizes 2; sparsity 0.5; spectral radius 0.9. TinyImageNet, T = 50. 1. AdaGauss: batch size 192; α = 1.2; λ = 5; learning rate 0.08; adapter learning rate 0.01; backbone learning rate 0.01; epochs 220; s = 64; singular-value fraction 0.95; τ = 1.6; weight decay 0.0005. 2. EFC++: batch size 64; balanced epochs 12; balanced learning rate 0.001; damping 0.1; first-task epochs 100; later-task epochs 100; λ = 10; prototype update -0.2. 3. ADC: batch size 128; epochs 100; initial epochs 100; initial learning rate 0.06; λ = 60; learning rate 0.06; temperature 4; weight decay 0.0005. 24

4. ACIL: incremental batch size 96; initial batch size 384; buffer size 8192; backbone ResNet18 CBAM; decay 0.18; γ = 0.18; initial epochs 220; initial learning rate 0.015; initial weight decay 0.0002; learning-rate milestones 132 and 187; scheduler MultiStep; warmup epochs 10. 5. FeCAM: batch size 32; α1 = 1; α2 = 0.7; β = 0.7; initial epochs 190; initial learning rate 0.027; initial weight decay 0.0005. 6. FeTRIL: batch size 32; epochs 200; initial epochs 200; initial learning rate 0.005; initial weight decay 0.0002; learning rate 0.005; temperature 2; weight decay 0.001. 7. IL2A: batch size 96; epochs 112; γ = 0.05; learning rate 0.0005; augmentation ratio 1.5; step size 45; temperature 0.3; softmax temperature 0.3. 8. PASS: batch size 48; epochs 56; γ = 0.125; feature-distillation weight 15; prototype weight 7.5; learning rate 0.0009; step size 45; temperature 0.14; softmax temperature 0.14; weight decay 0.0002. 9. LwF: batch size 192; epochs 320; initial epochs 320; initial learning rate 0.01; initial learning-rate decay 0.5; initial weight decay 0.0005; λ = 4.2; learning rate 0.01; learningrate decay 0.75; temperature 3.2; weight decay 0.0005. 10. DS-AL: incremental batch size 256; initial batch size 256; buffer size 8192; compensation ratio 0.85; γ = 0.252; compensation γ = 0.1; initial epochs 139; initial learning rate 0.0918; initial weight decay 0.0005; Scheduler: multi-step, milestones 45 and 90, with scheduler γ = 0.294; warmup epochs 10. 11. CIRCLE: reservoir internal dimension 384; head ensemble size 7; feature ensemble size 7; reservoir output dimension 1024; CNN-stem channels 16; CNN-stem kernels 3; leak 0.7; leaky-ReLU slope 0.01; maximum rotation 0; reservoir layers 2; patch sizes 2; sparsity 0.5; spectral radius 0.9. TinyImageNet, T = 100. 1. AdaGauss: batch size 192; α = 0.72; λ = 1.5; learning rate 0.1; adapter learning rate 0.01; backbone learning rate 0.01; epochs 220; s = 64; singular-value fraction 0.95; τ = 3.2; weight decay 0.0005. 2. EFC++: batch size 48; balanced epochs 10; balanced learning rate 0.00125; damping 0.1; first-task epochs 90; later-task epochs 115; λ = 3; prototype update -0.16. 3. ADC: batch size 32; epochs 360; initial epochs 360; initial learning rate 0.005; λ = 22; learning rate 0.005; temperature 1; weight decay 0.0005. 4. ACIL: incremental batch size 384; initial batch size 384; buffer size 8192; backbone ResNet18 CBAM; decay 0.392; γ = 0.504; initial epochs 154; initial learning rate 0.054; initial weight decay 0.0005; learning-rate milestones 69 and 108; scheduler MultiStep; warmup epochs 5. 5. FeCAM: batch size 32; α1 = 1; α2 = 0.7; β = 0.7; initial epochs 190; initial learning rate 0.027; initial weight decay 0.0005. 6. FeTRIL: batch size 32; epochs 200; initial epochs 200; initial learning rate 0.01; initial weight decay 0.0002; learning rate 0.0075; temperature 2.5; weight decay 0.001. 7. IL2A: batch size 96; epochs 112; γ = 0.05; learning rate 0.0005; augmentation ratio 1.5; step size 45; temperature 0.3; softmax temperature 0.3; weight decay 0.0002. 8. PASS: batch size 32; epochs 100; γ = 0.2; feature-distillation weight 60; prototype weight 2; learning rate 0.0005; step size 45; temperature 0.1; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 64; epochs 308; initial epochs 308; initial learning rate 0.02; initial learningrate decay 0.5; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.25; temperature 1; weight decay 0.0005. 10. DS-AL: incremental batch size 512; initial batch size 192; buffer size 8192; compensation ratio 1; γ = 0.63; compensation γ = 0.1; initial epochs 169; initial learning rate 0.054; initial weight decay 0.0005; Scheduler: multi-step, milestones 101 and 144, with scheduler γ = 0.588; warmup epochs 0. 25

11. CIRCLE: reservoir internal dimension 384; head ensemble size 7; feature ensemble size 7; reservoir output dimension 1024; CNN-stem channels 16; CNN-stem kernels 3; leak 0.7; leaky-ReLU slope 0.01; maximum rotation 0; reservoir layers 2; patch sizes 2; sparsity 0.5; spectral radius 0.9. ImageNet-Subset, T = 10. 1. AdaGauss: batch size 128; α = 1; λ = 10; learning rate 0.1; adapter learning rate 0.05; backbone learning rate 0.05; epochs 110; s = 64; singular-value fraction 0.95; τ = 2; 224-pixel inputs enabled; weight decay 0.0005. 2. EFC++: batch size 64; balanced epochs 50; balanced learning rate 0.001; damping 0.1; first-task epochs 100; later-task epochs 100; λ = 10; prototype update -0.2. 3. ADC: batch size 256; epochs 200; initial epochs 200; initial learning rate 0.25; λ = 20; learning rate 0.25; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 256; initial batch size 128; buffer size 16384; decay 0.28; γ = 0.28; initial epochs 145; initial learning rate 0.0408; initial weight decay 0.0005. 5. FeCAM: batch size 256; α1 = 1; α2 = 0.6; β = 0.3; initial epochs 200; initial learning rate 0.5; initial weight decay 0.0005. 6. FeTRIL: batch size 256; epochs 400; initial epochs 400; initial learning rate 0.02; initial weight decay 0.0005; learning rate 0.02; temperature 2. 7. IL2A: batch size 64; epochs 100; γ = 0.1; learning rate 0.001; augmentation ratio 2.5; step size 45; temperature 2; softmax temperature 0.1. 8. PASS: batch size 32; epochs 100; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.0002; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 256; epochs 110; initial epochs 110; initial learning rate 0.02; initial learning-rate decay 0.02; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.02; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 256; initial batch size 128; buffer size 16384; compensation ratio 1.5; γ = 0.28; compensation γ = 0.1; initial epochs 145; initial learning rate 0.03468; initial weight decay 0.0005; Scheduler: multi-step, milestones 30 and 60, with scheduler γ = 0.28; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 460; head ensemble size 9; feature ensemble size 8; reservoir output dimension 900; CNN-stem activation leaky ReLU; CNN-stem channels 4, 6, and 6; CNN-stem kernels 3, 3, and 3; internal state width 344; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 7, 8, and 16; sparsity 0.9; spectral radius 0.9. ImageNet-Subset, T = 20. 1. AdaGauss: batch size 128; α = 1; λ = 10; learning rate 0.1; adapter learning rate 0.01; backbone learning rate 0.01; epochs 220; s = 64; singular-value fraction 0.95; τ = 2; 224-pixel inputs enabled; weight decay 0.0005. 2. EFC++: batch size 64; balanced epochs 50; balanced learning rate 0.001; damping 0.1; first-task epochs 100; later-task epochs 100; λ = 10; prototype update -0.2. 3. ADC: batch size 128; epochs 400; initial epochs 400; initial learning rate 0.05; λ = 20; learning rate 0.05; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 256; initial batch size 256; buffer size 16384; decay 0.1; γ = 0.1; initial epochs 180; initial learning rate 0.02; initial weight decay 0.0005. 5. FeCAM: batch size 64; α1 = 1; α2 = 1; β = 0.5; initial epochs 400; initial learning rate 0.1; initial weight decay 0.0005. 6. FeTRIL: batch size 256; epochs 200; initial epochs 200; initial learning rate 0.02; initial weight decay 0.0005; learning rate 0.02; temperature 2. 7. IL2A: batch size 128; epochs 200; γ = 0.1; learning rate 0.001; augmentation ratio 2.5; step size 45; temperature 2; softmax temperature 0.1. 26

8. PASS: batch size 32; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.0002; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 256; epochs 110; initial epochs 110; initial learning rate 0.02; initial learning-rate decay 0.02; initial weight decay 0.0005; λ = 3; learning rate 0.02; learning-rate decay 0.02; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 256; initial batch size 256; buffer size 16384; compensation ratio 1.5; γ = 0.12; compensation γ = 0.1; initial epochs 180; initial learning rate 0.02; initial weight decay 0.0005; Scheduler: multi-step, milestones 30 and 60, with scheduler γ = 0.1; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 460; head ensemble size 9; feature ensemble size 8; reservoir output dimension 900; CNN-stem activation leaky ReLU; CNN-stem channels 4, 6, and 6; CNN-stem kernels 3, 3, and 3; internal state width 344; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 7, 8, and 16; sparsity 0.9; spectral radius 0.9. ImageNet-Subset, T = 50. 1. AdaGauss: batch size 128; α = 1; λ = 10; learning rate 0.075; adapter learning rate 0.0075; backbone learning rate 0.0075; epochs 55; s = 64; singular-value fraction 0.95; τ = 2; 224-pixel inputs enabled; weight decay 0.0005. 2. EFC++: batch size 64; balanced epochs 12; balanced learning rate 0.001; damping 0.1; first-task epochs 100; later-task epochs 100; λ = 10; prototype update -0.2. 3. ADC: batch size 96; epochs 100; initial epochs 100; initial learning rate 0.0375; λ = 20; learning rate 0.0375; temperature 2; weight decay 0.0005. 4. ACIL: incremental batch size 384; initial batch size 128; buffer size 16384; decay 0.28; γ = 0.28; initial epochs 122; initial learning rate 0.0432; initial weight decay 0.0005. 5. FeCAM: batch size 384; α1 = 1; α2 = 1; β = 0.5; initial epochs 204; initial learning rate 0.3; initial weight decay 0.0005. 6. FeTRIL: batch size 128; epochs 150; initial epochs 150; initial learning rate 0.015; initial weight decay 0.0005; learning rate 0.015; temperature 2. 7. IL2A: batch size 128; epochs 50; γ = 0.1; learning rate 0.00025; augmentation ratio 2.5; step size 45; temperature 2; softmax temperature 0.1. 8. PASS: batch size 16; epochs 50; γ = 0.1; feature-distillation weight 10; prototype weight 10; learning rate 0.0002; step size 45; temperature 2; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 128; epochs 82; initial epochs 82; initial learning rate 0.005; initial learningrate decay 0.02; initial weight decay 0.0005; λ = 3; learning rate 0.005; learning-rate decay 0.02; temperature 2; weight decay 0.0005. 10. DS-AL: incremental batch size 384; initial batch size 128; buffer size 16384; compensation ratio 1.5; γ = 0.28; compensation γ = 0.1; initial epochs 122; initial learning rate 0.03672; initial weight decay 0.0005; Scheduler: multi-step, milestones 30 and 60, with scheduler γ = 0.28; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 460; head ensemble size 9; feature ensemble size 8; reservoir output dimension 900; CNN-stem activation leaky ReLU; CNN-stem channels 4, 6, and 6; CNN-stem kernels 3, 3, and 3; internal state width 344; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 7, 8, and 16; sparsity 0.9; spectral radius 0.9. ImageNet-Subset, T = 100. 1. AdaGauss: batch size 128; α = 1; λ = 10; learning rate 0.075; adapter learning rate 0.01; backbone learning rate 0.01; epochs 55; s = 64; singular-value fraction 0.95; τ = 2; 224-pixel inputs enabled; weight decay 0.0005. 2. EFC++: batch size 96; balanced epochs 40; balanced learning rate 0.00125; damping 0.16; first-task epochs 75; later-task epochs 75; λ = 8; prototype update -0.25. 27

3. ADC: batch size 128; epochs 115; initial epochs 115; initial learning rate 0.028125; λ = 30; learning rate 0.028125; temperature 1; weight decay 0.0005. 4. ACIL: incremental batch size 256; initial batch size 64; buffer size 16384; decay 0.28; γ = 0.392; initial epochs 110; initial learning rate 0.03024; initial weight decay 0.0005. 5. FeCAM: batch size 64; α1 = 1.5; α2 = 1.2; β = 0.6; initial epochs 440; initial learning rate 0.2; initial weight decay 0.0005. 6. FeTRIL: batch size 128; epochs 100; initial epochs 230; initial learning rate 0.017; initial weight decay 0.0005; learning rate 0.03; temperature 1. 7. IL2A: batch size 128; epochs 180; γ = 0.1; learning rate 0.0005; augmentation ratio 2.5; step size 45; temperature 0.3; softmax temperature 0.3. 8. PASS: batch size 48; γ = 0.16; feature-distillation weight 15; prototype weight 1.5; learning rate 0.0003; step size 45; temperature 0.1; softmax temperature 0.1; weight decay 0.0002. 9. LwF: batch size 96; epochs 70; initial epochs 70; initial learning rate 0.006; initial learningrate decay 0.01; initial weight decay 0.0005; λ = 6; learning rate 0.006; learning-rate decay 0.02; temperature 4; weight decay 0.0005. 10. DS-AL: incremental batch size 256; initial batch size 64; buffer size 16384; compensation ratio 1.25; γ = 0.392; compensation γ = 0.1; initial epochs 110; initial learning rate 0.03024; initial weight decay 0.0005; Scheduler: multi-step, milestones 30 and 60, with scheduler γ = 0.28; warmup epochs 0. 11. CIRCLE: reservoir internal dimension 460; head ensemble size 9; feature ensemble size 8; reservoir output dimension 900; CNN-stem activation leaky ReLU; CNN-stem channels 4, 6, and 6; CNN-stem kernels 3, 3, and 3; internal state width 344; leak 0.8; leaky-ReLU slope 0.01; reservoir layers 1; patch sizes 7, 8, and 16; sparsity 0.9; spectral radius 0.9.

28