Forgetting, plasticity, and co-observation: a third facet of continual learning

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

F ORGETTING , PLASTICITY, AND CO - OBSERVATION : A THIRD FACET OF CONTINUAL LEARNING Timm Hess ESAT † KU Leuven, Belgium

Abhishek Jha VIB.AI † KU Leuven, Belgium

Gido M. van de Ven Bernoulli Institute ‡ University of Groningen, Netherlands

Tinne Tuytelaars ESAT † KU Leuven, Belgium

arXiv:2608.18803v1 [cs.LG] 19 Aug 2026

A BSTRACT Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental “chunking” scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner’s generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

1

I NTRODUCTION

Deep neural networks are predominantly trained under a joint training paradigm, where the model is optimized on independent and identically distributed (i.i.d.) batches drawn uniformly from a static dataset. In many real-world settings, however, it would be transformative if this paradigm need not be maintained. When new data arrives sequentially over time, retraining on all previously observed data becomes expensive or sometimes infeasible due to, e.g. computational, privacy, or storage constraints (Verwimp et al., 2024). The model could instead be sequentially trained on separate partitions of data as the data arrives. This process of incrementally updating a model from a non-stationary data stream is referred to as continual learning (CL) or lifelong learning (Chen & Liu, 2018; Parisi et al., 2019). A fundamental challenge in this regime stems from what is known as the stability-plasticity dilemma (Grossberg, 1980; Mermillod et al., 2013). On the one hand, an effective continual learning system requires sufficient plasticity to adapt its representation based on new data. On the other hand, updating the model exclusively on new observations disrupts existing parameter configurations thereby severely impairing its stability, a.k.a. catastrophic forgetting (McCloskey & Cohen, 1989; Parisi et al., 2019). Consequently, a large part of the continual learning literature focuses on algorithmic interventions designed to retain prior knowledge while remaining capable of incorporating new information (De Lange et al., 2021; Wang et al., 2024). Progress on these two desiderata is typically assessed relative to a pair of canonical reference points: a lower target, established by naive sequential training, typically interpreted as exposing the severity of forgetting in the absence of any intervention; and an upper target, established by (offline) joint training, interpreted as reflecting the maximum achievable performance on the full dataset. This evaluation framework forwards the implicit assumption that perfectly managing stability and plasticity during sequential training would close the gap to joint training. In our work, we question this assumption. We hypothesize that, under a sequential training regime, even a model with perfect stability and plasticity would still underperform a jointly trained model in terms of its generalization properties. This intuition can be linked to multi-task learning theory (Caruana, 1997), which posits that simultaneous training provides a mutually beneficial inductive bias. Our work empirically investigates this dynamic in continual learning, and we demonstrate a benefit of simultaneous training even in identically distributed, data-incremental setups. To put this another way, †

{timmfelix.hess, abhishek.jha, tinne.tuytelaars}@kuleuven.be; ‡ g.m.van.de.ven@rug.nl

1

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

Figure 1: Intuitive illustration of the generalization properties of separate training in comparison to joint training. Finding a common structure across the presented information, here depicted as puzzle pieces, is easier when observing the information together, as in the joint training case (right). In contrast, we can think of separate training as placing information piece-by-piece while focusing on one piece of information at a time (left). Here, local fits emerge that struggle to map relationships across multiple pieces of information. we argue that the challenge of sequential training can be split into two parts: the temporal aspect, which underlies forgetting and plasticity, and the challenge of separate training, whereby cross-partition dependencies remain hidden. We refer to the influence of this phenomenon as the co-observation effect. An analogy, illustrated in Figure 1, provides a simple intuition for the challenge of separate training: assembling a puzzle is easier when all pieces are visible simultaneously, because overarching patterns are more readily recognized when the pieces that jointly encode them are co-observed. To empirically support our hypothesis, we report our results on data-incremental “chunking” scenarios (Lee & Storkey, 2023), which avoid macroscopic distribution shifts, arguably making it one of the hardest settings to show this effect. Furthermore, we show that this co-observation effect persists across both supervised and self-supervised learning paradigms. We design a diagnostic methodology based on continual pre-training (Hu et al., 2022; Zhou et al., 2025) and model ensembles (Vogelstein et al., 2025) to isolate the effect of co-observation as much as possible. Inspired by Hess et al. (2024), we evaluate continual learning at representation level via linear probing (Alain & Bengio, 2017), which decouples the model’s generalization from the confounding dynamics of a continually updated classifier head. Moreover, we employ an ensemble to control for forgetting. The ensemble preserves the full representational state of the model after each sequential training phase, thereby avoiding forgetting by design. Our results show that in the “chunking” setting, without distribution shift and class imbalance as additional confounds, the ensemble-baseline is consistently surpassed by incremental joint training. Because the impact of forgetting is controlled for, we attribute this residual difference in results to the absence of co-observation during optimization. Ultimately, our findings point towards an important realization for the conceptualization of continual learning mechanisms. Maintaining stability and plasticity is the fundamental requirement in continual learning. To approach the generalization capabilities of joint training, continual learning methods must also acknowledge and compensate for the representational deficit caused by the loss of co-observation in separate training. We summarize our contributions as follows: • Conceptualization of the co-observation effect: We distinguish the impact of separate training from stability-plasticity, demonstrating co-observation as a distinct mechanism when it comes to explaining the gap between continual learning and joint learning. • Diagnostic methodology for separating forgetting from co-observation: We design an experimental setup based on continual pre-training, linear probing, and a full-retention ensemble baseline. This setup effectively disentangles the performance degradation caused by catastrophic forgetting from the representational deficit caused by the missing co-observation of the data during training. • Empirical evidence in the vision domain: We provide rigorous empirical evidence of the co-observation effect across both supervised and self-supervised continual pre-training paradigms, on standard CL vision benchmarks in the “chunking” setting, demonstrating that it is a fundamental limitation of separate training. • Contextualization of memory replay and knowledge distillation: We discuss the broader implications of our findings, marking that the empirical success of memory rehearsal methods stems not only from preventing forgetting but also from actively restoring the conditions necessary for data co-observation. And vice versa, knowledge distillation at a representation level alone can compensate forgetting, but not co-observation. Code: https://github.com/TimmHess/Co-ob.

2

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

2

I SOLATING THE CO - OBSERVATION EFFECT

To understand the impact of data co-observation in continual learning, it must be carefully detached from the impact of forgetting and loss of plasticity. In this section, we first make a conceptual argument for the co-observation effect, detailing how the lack of simultaneous data access implies a fundamental representational deficit. Subsequently, we introduce our methodology to demonstrate this phenomenon empirically. Because standard end-to-end evaluations conflate the effect from co-observation with effects from stability-plasticity, and come with additional challenges regarding confounding variables such as task-recency bias and class imbalance (Wu et al., 2019; Davari et al., 2022), we design a diagnostic evaluation framework focused on continual representation learning to decouple these effects as cleanly as possible. 2.1

A CONCEPTUAL EXPLANATION OF THE CO - OBSERVATION EFFECT

Assume a global data distribution is partitioned into sequential observations, D1 and D2 . And, assume there exists a set of generalizable features, Zshared , which improve prediction of the target variable of interest across the joint distribution. Following multi-task learning theory (Baxter, 2000), discovering Zshared is facilitated by minimizing risk across multiple data variations simultaneously. When optimizing solely on D1 , neural networks naturally favor simpler functions (simplicity bias) (Shah et al., 2020). Consequently, the model relies on task-specific shortcuts Zlocal instead of learning Zshared . Because Zshared was never necessary to minimize the loss in D1 , it becomes unidentifiable from the frozen parameters θ1 . Consider an ideal continual learner transitioning to D2 with perfect stability (zero forgetting) and plasticity. While it perfectly retains θ1 , these parameters do not encode the discarded Zshared , and when optimizing exclusively on D2 , the model will again favor local shortcuts over discovering Zshared . We argue that discovering the overarching synergies of Zshared fundamentally requires updating features based on their simultaneous performance across both distributions. Without concurrent access to both D1 and D2 , the model lacks the mutual constraints necessary to favor shared features. Thus, even a model that perfectly retains all past parameter states cannot necessarily synthesize cross-distribution dependencies. 2.2

C ONTINUAL LEARNING AND LINEAR PROBING

We consider a standard separate training setting where a model is trained sequentially on a series of data subsets D1 , D2 , . . . , DT . At each training stage t, by default, the model has access only to Dt and cannot observe data from previous or future stages. Access to previous data is permitted only when a CL mechanism maintains a memory buffer. To evaluate the fundamental quality of the knowledge acquired during this process, we decouple the model into a feature extractor (backbone) and a classifier (head), focusing our analysis entirely on the feature extractor. Following standard practices in representation learning, we evaluate all approaches using linear probing (LP; Alain & Bengio, 2017). After completing a stage in the sequential training process, we freeze the learned feature extractor and train ST a linear classifier on top of the frozen representations using all available training data t=1 Dt . By evaluating the representation globally, we ensure that any differences in performance reflect its generalizability, rather than artifacts of a continually shifting decision boundary. 2.3

F OUR BASELINES

To isolate the influence of co-observation, we compare the sequentially trained representation against an upper target while controlling for catastrophic forgetting. We achieve this by comparing four distinct baselines (c.f. Figure 2): ST • Joint (offline): A single model is trained with simultaneous access to all data i=1 Di . This represents the standard offline machine learning paradigm, i.e. the common choice of upper target for representation quality. • IncrementalSjoint: At each stage t, the model is trained with simultaneous access to all data encountered t so far, i.e. i=1 Di . Because the model trains on all available past and present data simultaneously, it circumvents both fragmented data access and catastrophic forgetting. Because it is subject to loss of plasticity from repeated optimization, as other continual learning approaches, we utilize it for fair comparison. • Ensemble (perfect forgetting mitigation): To eliminate forgetting of previously learned knowledge without explicitly breaking the separate observability constraint of sequential training, we employ an ensemble approach. After each sequential training stage t, a copy of the feature extractor is stored. At evaluation time, the representations from all T stored checkpoints are concatenated, and a single linear probe is trained on this combined representation. 3

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

• Naive sequential (lower target): A single model is trained on each data subset incrementally, with each stage initialized from the previous stage’s final checkpoint. This naive continual learning approach is susceptible to both catastrophic forgetting and the lack of co-observation. Because the ensemble preserves every historical state of the model, features learned at any stage remain available to the linear probe and its forgetting is mitigated by definition. Conversely, incremental joint training establishes an “upper target” that accounts for the loss of plasticity from the repeated optimization trajectory that the other sequential learning approaches are subject to. We included joint (offline) training to visualize that degradation. To the best of our knowledge, the ensemble serves as the most controlled approximation for a perfect-retention baseline to isolate the co-observation effect as cleanly as possible. Also, we do not aim to position the ensemble-baseline as a practical motivated choice of a continual learning approach, but use it as an analytical device.

2.4

C ONTROLLING FOR REPRESENTATIONAL CAPACITY

(a)

(b)

Co-observation

Plasticity

Joint Incr. Joint Ensemble Naive

Forgetting

A methodological challenge arises when comparing the ensemble to single-model baselines. First, the concatenated ensemble yields an expanded representation of dimension T × d. To ensure a rigorous comparison of feature quality rather than raw capacity, we apply PCA where appropriate, retaining the d directions of greatest variance. This projection does not perfectly simulate a single model that has mitigated forgetting, but corrects for differences due to architectural bloat. Second, while our ensemble of frozen checkpoints prevents the loss of previously acquired information, i.e. forgetting, it inherently differs from a hypothetical “ideal” single model. Specifically, it does not fully account for the seamless consolidation of knowledge via forward and backward transfer. To gauge the influence of this potential confound regarding our experimental setup (Section 4), we empirically ablate it in Appendix C and find it to be minor.

✓ ✓ ✓ ✗

✓ ✓ ✗ ✗

✓ ✗ ✗ ✗

(c)

Figure 2: Overview of training and evaluation for joint and separate training and expected empirical decomposition of the co-observation effect. (a) In the (incremental) joint training settings (top), the backbone θBt is trained with simultaneous access to all data (accumulated up to stage t). The incremental joint model is continually updated from the previous stage’s initialization at each step, whereas the offline joint model is trained from scratch and depicted only at the final stage. In the separate training setting (bottom), the model observes only the current data chunk. To demonstrate the effect of co-observation, we eliminate forgetting without breaking the sequential visibility constraint using the ensemble baseline. The ensemble stores a frozen copy of the backbone after each task. During evaluation, representations from all t checkpoints are concatenated, optionally projected back to the original dimension d to control for representational capacity (dim. reduce), and evaluated via linear probing (LP). (b) Hypothesis of the expected results across four chunks. The performance difference between the naive sequential model and the ensemble quantifies forgetting, while the gap between the incremental joint model and the (offline) joint target highlights the loss of plasticity. The focus of this work lies on the remaining difference between the ensemble and the incremental joint model, which we attribute to the co-observation effect. (c) Summary of how the four baselines inherently account for the presence (✓) or absence (✗) of forgetting mitigation, data co-observation, and maintained plasticity.

4

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

3

P ROOF OF PRINCIPLE

To provide a proof-of-principle for the co-observation effect, we design an experiment where we can explicitly control the available information at each training stage. This allows us to trace how separate observation, in comparison to joint observation, affects the learned representation. We construct a sequence of two binary classification tasks from MNIST digits (LeCun et al., 1998). Task 1 (D1 ) is odd vs. even classification on clean images. Task 2 (D2 ) is small (digits 0–4) vs. large (digits 5–9) classification on images corrupted with Gaussian noise (severity level 5, applied via the imgaug library (Jung et al., 2020)). We select the evaluation task to cover odd vs. even classification on noisy images (Figure 3). Success on this evaluation task requires combining elements from both training tasks, namely the decision boundary from Task 1 and the ability to handle noisy inputs as encountered in Task 2. The question is whether the features, learned in the context of Task 1, remain available and transferable when learned sequentially versus jointly. For the experiments in this section, we utilize a 3-layer MLP backbone with 400 hidden units per layer. The models are trained for 10 epochs per task using a batch size of 64. Optimization is performed using Stochastic Gradient Descent (SGD) with a peak learning rate of 0.01, momentum of 0.9, and gradient clipping at 1.0, regulated by a cosine annealing schedule (5% warm-up phase). We report the average performance across three independent runs with varying random seeds. The results, displayed in Figure 3, show that the ensemble achieves higher LP-accuracy than naive sequential training, presenting the influence of forgetting. However, incremental joint training further improves accuracy over the ensemble, pointing to another source of knowledge gain that is not explained by mitigating forgetting, as the ensemble preserves all past knowledge, but by co-observation that distinguishes joint training and separate training. To elaborate, in separate training, the model learns features that are sufficient for the current task’s data, but in a form that generalizes primarily within that training context. As such, Task 2 does expose the model to noise and the resulting representation can handle noisy inputs for the small/large classification, but this capability does not transfer to the odd/even task when trained sequentially. In contrast, joint training, i.e. simultaneous observation, allows this transfer of knowledge. When data of both tasks are observed together during training, features that serve both objectives are realized.

Figure 3: Proof-of-principle MNIST experiment, demonstrating the co-observation effect. Four approaches are compared: naive sequential training, ensemble, incremental joint, and (offline) joint training. Task 1 is binary classification of odd vs. even digits on clean images. Task 2 is binary classification of small (≤ 4) vs. large (≥ 5) digits on images augmented with Gaussian noise. The evaluation task is odd vs. even classification on images that exhibit similar Gaussian noise as Task 2. Both joint training variants achieve higher LP-accuracy, demonstrating that noise features transfer to the “odd vs. even”-evaluation task when noisy data is being observed simultaneously, even when associated with a different task. This is contrasted by the ensemble, which preserves all learned representations but still achieves lower LP-accuracy. The gap between ensemble and joint training we attribute to the co-observation effect.

5

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

4

C O - OBSERVATION IN CONTINUAL LEARNING

While the controlled experiment in the previous section showcases the effect of co-observation in a controlled but artificially designed scenario, we now continue to demonstrate its influence as a generic property to be taken into account in sequential learning. Continual learning benchmarks typically induce a severe distribution shift between stages (e.g., class-incremental learning). To emphasize the generality of our findings beyond artifacts of such amplified distribution shift or class imbalance, we evaluate the co-observation gap in an environment of uniformly partitioned incremental data. Following Lee & Storkey (2023), we employ a “chunking” protocol (data-incremental learning) where the global training dataset is randomly and uniformly partitioned into four identically distributed chunks (D1 , . . . , D4 ). In this setting, even without macroscopic distribution shifts between the sequential stages, we are able to observe representation deficits that can be directly attributed to the lack of co-observation. We first assess the influence of co-observation across both supervised and self-supervised paradigms by comparing our diagnostic baselines. Subsequently, we evaluate exemplar replay and knowledge distillation in light of the coobservation effect. To ensure reproducibility, full details on the experimental setup are provided in Appendix A. Further implementation specifics regarding the diagnostic ensemble and linear evaluation are detailed in Appendix B, while the configurations for the continual learning methods are outlined in Appendix D. Finally, comprehensive tables containing the numerical results for all experiments are available in Appendix F. 4.1

S UPERVISED AND SELF - SUPERVISED CONTINUAL LEARNING

To establish the co-observation gap as a generic property of sequential optimization, we first compare our four baselines: naive sequential training, the ensemble, incremental joint, and (offline) joint. As explained in Section 2.4, to control for the increased representational capacity of the ensemble, we also report the ensemble baseline with original backbone dimension d reduced by PCA. We evaluate these baselines across both supervised and self-supervised paradigms using the 4-chunk partitioned setup. In particular, we evaluate on CIFAR-100 (Krizhevsky et al., 2009) and ImageNet-100 (Russakovsky et al., 2015) (IN-100) using a ResNet-18 (He et al., 2016) backbone for supervised learning and Barlow Twins (Zbontar et al., 2021), and a ViT-B/16 (Dosovitskiy et al., 2021) for I-JEPA (Assran et al., 2023). All models are trained from scratch. For fair comparison, all training is constrained to an identical, fixed iteration budget per chunk. This fixed number of optimization steps is calibrated to match the (offline) joint training budget of 100 epochs on CIFAR-100 and 90 epochs on IN-100. Similarly, for the self-supervised methods (Barlow Twins and I-JEPA), the fixed iteration budget is calibrated to match 400 epochs, aligned with Fini et al. (2022).

(a)

(b)

Figure 4: The co-observation effect across supervised and self-supervised learning paradigms. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds (6 for I-JEPA). (a) Supervised learning (Cross-Entropy) on CIFAR-100 and ImageNet-100 (IN-100). (b) Self-supervised learning using Barlow Twins and I-JEPA on IN-100. In all settings, the beneficial effect of co-observation is clearly visible. The ensemble baseline prevents the forgetting seen in the Naive baseline, and consistently falls short of the incremental joint training. This residual difference between ensemble (separate training) and incremental joint highlights the impact of co-observation, demonstrating that simultaneously observing data is important for representation learning, regardless of the objective function. Supervised continual learning: As illustrated in Figure 4a, and further extended in Appendix E, naive sequential training exhibits the lowest linear probing accuracy, affirming the impact of catastrophic forgetting on the representation even in the absence of explicit class-incremental distribution shifts. The ensemble baseline significantly improves upon naive sequential training and is not affected when controlling for representational capacity by PCA reduction, indicated by both lines overlapping. A clear benefit is visible from co-observation as the incremental joint training 6

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

consistently improves over the ensemble. Because the ensemble retains all past representations, this residual difference highlights the co-observation effect, i.e. the representational benefit derived from simultaneously observing data across different chunks, is a property that is fundamentally lost in separate training. Self-Supervised continual learning: To investigate whether the co-observation effect is specific to supervised crossentropy optimization, we extend our evaluation to self-supervised learning (SSL). Unlike supervised learning, which leverages explicit class boundaries, SSL methods rely on inferring diverse features within the training batches to form robust, invariant representations, which are thought to be more robust to forgetting (Fini et al., 2022). We evaluate two distinct families of SSL architectures on IN-100: Barlow Twins (Zbontar et al., 2021), which relies on redundancy reduction, and I-JEPA (Assran et al., 2023), a recent joint-embedding predictive framework. The results (Figure 4b) demonstrate that the co-observation gap is also pronounced in the self-supervised regime. Interestingly, applying PCA reduction to the ensemble yields varying effects depending on the SSL objective. For Barlow Twins it can occasionally improve the representation quality, while degrading it for I-JEPA. Yet, across both SSL algorithms and regardless of the capacity control, our ensemble baselines consistently evaluate lower than the representation quality obtained from the incremental joint model. This result indicates that the co-observation gap is a consequence of separate representation learning: synergistic features that span across data partitions cannot be fully realized without simultaneous observation, regardless of the underlying optimization objective. While it is difficult to perfectly isolate the co-observation effect from all sequential optimization dynamics, we account for the primary alternative explanation of a deficit in knowledge transfer in Appendix C. Supported by our controls, and this ablation, the persistent representational deficit observed across both supervised and distinct self-supervised paradigms strongly points to the structural lack of simultaneous data observation as the primary driver. 4.2

R ELATION TO CONTINUAL LEARNING MECHANISMS

Having confirmed the co-observation gap in generic continual learning scenarios, we now examine how replay and knowledge distillation compare to our established baselines, to decouple their ability to incrementally improve generalization in the model’s representation relative to the mitigation of forgetting alone. Experience Replay: Replaying stored data is intuitively the most direct approach to bridging separate learning stages. Mechanistically, it creates a minibatch-level approximation of the joint data distribution by presenting previous and new data together. While replay is predominantly renowned as a forgetting-mitigation tool, we evaluate it here to explicitly contextualize this minibatch approximation as mechanism that enables data co-observation. We evaluate replay on IN-100 with buffers allowing the storage of 1250(∼ 3%), 6250(∼ 15%), and 12500(∼ 30%) exemplars of the data from each previously observed chunk, where new data and all data from the replay buffers are combined and sampled i.i.d. into mini-batches during optimization (Rebuffi et al., 2017; Wu et al., 2019; Cho et al., 2026). In terms of raw values, as expected, the results presented in Figure 5 show that larger replay buffers yield higher accuracy. However, our interest is in the comparison to our baselines which allows to reason about a more nuanced

(a)

(b)

Figure 5: Interaction of continual learning mechanisms with the co-observation gap. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds for (a) Supervised Learning and (b) Self-Supervised Learning (Barlow Twins and I-JEPA). Comparing experience replay and knowledge distillation (LwF for SL; CaSSLe for Barlow) against the naive-, joint-incremental-, and ensemble-baselines, empirically demonstrates the relationship of both methods to mitigation of forgetting and obtained generalization from the representation learning perspective. 7

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

dynamic. While sparse rehearsal (e.g., 3%) has been shown to yield improvements when evaluated via end-to-end accuracy (Chaudhry et al., 2019), our linear-probing evaluation reveals a different dynamic at representation level. Here, replay with tiny buffers struggles to consistently match the diagnostic ensemble. Storing approximately 15% of the data appears sufficient for the replay mechanism to effectively match the performance of the ensemble. When the buffer size is increased to 30% of the observed exemplars per stage, the linear probing accuracy pushes past the ensemble baseline, closing half the gap to incremental joint training. These results hold for supervised and selfsupervised training with Barlow Twins and I-JEPA. The replay mechanism inherently conflates the mitigation of forgetting with the benefits of co-observation. However, our comparison against our ensemble baseline yields two crucial takeaways. First, our findings suggest that replay’s effectiveness is not bound merely by mitigating forgetting, but by actively reintroducing data co-observation. Because we strictly control the training budget across all setups, the gains from larger replay buffers are not an artifact of extended optimization on more data. Rather, the improved approximation of the joint distribution at the mini-batch level – instantiating the mechanism underlying co-observation – drives altered optimization trajectories that allow the model to synthesize cross-distribution features that perfect retention alone cannot recover. Second, there is a significant amount of stored exemplars necessary to improve beyond results achievable by mitigation of forgetting, although the specific number of samples required will inherently vary based on the domain and dataset diversity. Distillation: As a popular memory-free alternative to rehearsal, we evaluate knowledge distillation. We apply Learning without Forgetting (LwF) (Li & Hoiem, 2016) for the supervised setup, and CaSSLe (Fini et al., 2022) for the self-supervised Barlow Twins setup. As shown in Figure 5, distillation achieves linear probing accuracies that closely mirror those of our ensemble and the 15% replay buffer. Knowledge distillation has previously been shown to be an effective retention mechanism, successfully preventing representation-level forgetting (Ni et al., 2023). We deem it important to emphasize that knowledge distillation here is not failing to prevent forgetting, as indicated by its parity with the ensemble. Rather, distillation from the “incomplete” representation of the previous model does not facilitate the co-observation of information that would be required to recover the representational effectiveness found in large-scale replay.

5

R ELATED W ORK

5.1

F ORGETTING , PLASTICITY, AND A THIRD FACTOR

The continual learning literature has predominantly framed the challenges of sequential optimization along the stability-plasticity axis (Mermillod et al., 2013), typically addressed through mechanisms such as regularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2016), replay (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Buzzega et al., 2020; Rolnick et al., 2019), or architecture expansion (Rusu et al., 2016; Yan et al., 2021). Progress is measured against a joint-training upper target, implicitly assuming that a model with perfect stability (zero forgetting) and perfect plasticity (ability to incorporate new information) would naturally recover joint-training performance (Wang et al., 2024; Momeni & Liu, 2025). While recent work by Dohare et al. (2024) and others (Verwimp et al., 2025; Cho et al., 2026) has established loss of plasticity as a distinct degradation independent of forgetting, we identify the co-observation effect as a third factor. Several recent studies have empirically encountered symptoms of the co-observation effect, though these observations have typically been isolated within specific continual learning setups: In class-incremental scenarios, Ni et al. (2023) observed “inter-phase confusion” persisting after addressing forgetting, and Soutif-Cormerais et al. (2021) noted a deficit in cross-task features. While conceptually related, these observations have been confined to classincremental setups; we demonstrate that they are part of a general deficit in continual representation learning, extending well beyond class-incremental considerations. Moving beyond class-incremental boundaries, Busch et al. (2025) demonstrated that “continual confounders” prevent sequential learners from discovering generalizable rules. To clearly isolate this effect, the authors systematically introduce explicitly constructed, adversarial spurious correlations within a synthetic dataset. Our work highlights the broader implications of this result for separate training, demonstrating that such confounding effects can be considered to arise naturally simply from the sequence of the data itself. To do so, we build on the “chunking” scenario previously utilized by Lee & Storkey (2023) to relate the impact of separate training to forgetting. We shift the focus to data co-observation, demonstrating that the co-observation effect emerges generically from random partitioning and affects generalization alongside forgetting, class imbalance, or adversarial structures. Finally, our work establishes that this effect on continual representation learning is not merely an artifact of supervised training dynamics or the narrow decision boundaries induced by cross-entropy. Self-supervised learning (SSL) meth8

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

ods are broadly considered to learn more diverse, task-agnostic features (Fini et al., 2022; Madaan et al., 2022). Despite these advantages, we show that the lack of co-observation affects learning similarly across the distinct SSL frameworks of Barlow Twins (Zbontar et al., 2021) and I-JEPA (Assran et al., 2023). This establishes the co-observation effect as a fundamental limitation of fragmented optimization, affecting both supervised and self-supervised sequential learning. 5.2

R EPRESENTATION EVALUATION AND THE ENSEMBLE BASELINE

End-to-end accuracy in continual learning often conflates actual representation quality with the dynamics of a continually updated classifier head (Wu et al., 2019). To address this, Davari et al. (2022) used linear probing (LP) (Alain & Bengio, 2017) as an evaluation tool for CL, demonstrating that representation-level forgetting under naive finetuning is substantially milder than end-to-end metrics often suggest. We also adopt LP to evaluate the knowledge accumulated within our model at representation level. Thereby, our approach is more closely related to Hess et al. (2024), who combined LP evaluation with an ensemble baseline to demonstrate that even seemingly “mild” representation-level forgetting in absolute numbers still exerts a detrimental effect on the effectiveness of continual representation learning. Instead of utilizing probing to highlight the penalty of forgetting, in the current work, we leverage the ensemble as a baseline to control for forgetting, enabling our novel analysis regarding the isolation of the co-observation effect. 5.3

M ULTI - TASK LEARNING AND JOINT OPTIMIZATION

Historically, continual learning shares conceptual roots with multi-task learning (MTL). Both paradigms strive to accumulate and leverage knowledge across multiple objectives (Chen & Liu, 2018) and early continual learning works were differentiated from MTL by training the tasks incrementally instead of jointly. Regarding MTL, Caruana (1997) showed that learning tasks in parallel can improve generalization because related tasks provide a mutually beneficial inductive bias, helping the model discover features it would not find when trained on either task in isolation. Ruder (2017) articulated several mechanisms driving this benefit, including implicit data augmentation, attention focusing, and regularization through multi-objective optimization. Baxter (2000) provided a theoretical foundation for this, showing that the risk of overfitting shared parameters decreases with the number of co-observed tasks. Furthermore, as demonstrated by Bengio et al. (2009), the specific composition of training data significantly influences optimization trajectories in non-convex landscapes. These works underscore that which data is observed simultaneously, fundamentally shapes the quality of the learned solutions. To the best of our knowledge the classical MTL desideratum of improving representation quality was largely left aside in continual learning, presumably because of the observed overwhelming impact of catastrophic forgetting. In this sense, our work can be understood as demonstrating that even within a single task (rather than multiple different tasks), fragmenting data into separate portions leads to reduced generalization of the model. Consequently, a limit is placed on the representational quality that can be achieved from strict separate learning, regardless of how well the aspect of forgetting is controlled.

6

D ISCUSSION

6.1

L IMITING FACTORS OTHER THAN FORGETTING

Continual learning algorithms are expected to improve over naive sequential training. However, one could argue that there are cases where reaching the joint training upper target is a rather theoretical construct because of strict limitations in compute (Prabhu et al., 2023; Verwimp et al., 2024), memory (De Lange et al., 2021; Chaudhry et al., 2019), privacy boundaries (Farquhar & Gal, 2019), the need to operate in an open world (Mundt et al., 2023), etc. Nevertheless, achieving best possible results is the goal and a prevailing narrative heavily focuses on catastrophic forgetting as the primary culprit for the performance drop observed when training continually, with some recent pushes to also consider effects from plasticity losses (Cho et al., 2026; Verwimp et al., 2025). Our work points out yet another, fundamental, effect where previous knowledge is perfectly retained or protected, but the model’s generalization falls short of the upper target. That is, without requiring any particularity of the distribution shift underlying the separate observations, apart from the samples being disjoint. Different instances of this effect have been studied, predominantly with respect to class-incremental learning, where intuitively observing instances of different classes together is required to construct discriminating features (Wu et al., 2019; Soutif-Cormerais et al., 2021; Ni et al., 2023). But also regarding the potential of confounding factors in the separately observed training data, which inflict deficiencies on the continually training model that are not easy to recover (Busch et al., 2025). We demonstrate data co-observation as an inherent and distinct limiting factor in separate training, without additional prerequisites as studied in prior works. 9

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

6.2

C ONNECTION TO LARGE PRE - TRAINED MODELS AND LLM S

Recent continual learning literature increasingly leverages pre-trained models. Approaches that leave the backbone frozen reduce continual learning to incremental prototype memorization (Janson et al., 2022). In such settings, the discovery of cross-distribution synergies is typically of minor importance and the co-observation effect is bypassed. Methods employing conservative fine-tuning, e.g. applying slow learning rates (Zhang et al., 2023) or utilizing adapters (Aghasanli et al., 2025), aim to make only slight adjustments to the pre-trained features. Arguably, the magnitude of new learning is inherently small relative to the initial pre-training phase, and optimization is heavily regularized to protect existing weights. Consequently, the dynamics of catastrophic forgetting and its mitigation largely overpower any representational gains that would theoretically emerge from co-observation. Continual pre-training to broadly update and improve the generalization of the backbone in a more foundational manner (Hu et al., 2022) is where we hypothesize co-observation effects to be more important. While our empirical analysis is conducted on standard vision benchmarks, evidence from the large language model (LLM) literature suggests that the interplay between data co-occurrence and representation quality extends to larger scales. Ye et al. (2024) demonstrate that the composition of training batches during LLM pre-training, specifically, cross-domain interaction effects, quantitatively predicts model performance. Since their analysis concerns standard i.i.d. pre-training rather than sequential learning, this provides evidence that which data is observed together shapes learned representations independently of any forgetting dynamics. In the continual pre-training setting, Ibrahim et al. (2024) show that replaying previous data into new training batches is necessary to match the performance of retraining from scratch, a gap that learning rate adjustments alone cannot close. While this deficit is conventionally attributed to forgetting, our framework raises the complementary hypothesis that replay also restores co-observation conditions that enable the model to discover cross-distribution patterns. Disentangling these two roles of replay, forgetting prevention versus co-observation restoration, at LLM scale constitutes a natural direction of investigation. The diagnostic framework introduced in this work, comparing forgetting-free ensembles against joint training via linear probing, offers a principled starting point for this investigation.

7

C ONCLUSION

We empirically separated the effect of data co-observation from stability-plasticity considerations for the representation-level generalization of continually trained deep neural networks, and demonstrated the generic emergence of that effect across supervised and self-supervised training in data-incremental scenarios. While the representational synergies of joint training are well-established in the multi-task learning literature, we demonstrate such a performance gap persists generically in standard vision benchmarks even in “chunking” scenarios. We argue that making the influence of co-observation explicit and positioning it distinct from forgetting and plasticity, as a third facet of continual learning, is important for the community: assuming plasticity and successful forgetting mitigation whilst learning without co-observation appears to hit a performance ceiling that can be significantly lower than the upper target of a jointly trained model. We emphasize that we do not claim the performance penalty of co-observation universally rivals the sheer magnitude of catastrophic forgetting. As discussed in the context of pre-trained models, forgetting often remains the most striking source of degradation in many applications. Nevertheless, we deem explicitly isolating and understanding the co-observation effect to be conceptually vital for mapping the full dynamics and limits of continual learning. Improvement beyond requires continual learning algorithms to actively synthesize cross-data synergies. How to best facilitate such synergies, especially in compute-bounded settings, needs to be the subject of further research.

8

L IMITATIONS

We see three limitations to this work. The first regards our diagnostic methodology, which intends to isolate the co-observation gap using linear probing and an ensemble baseline. While this effectively removes confounders like classifier deviation and forgetting, strictly controlling the ensemble’s capacity requires post-hoc dimensionality reduction (in our case using PCA). Although an ideal baseline would natively match the original model’s parameter count, we deem our capacity-controlled ensemble to serve as the most viable analytical proxy. Secondly, we employed uniform optimization schedules to ensure fair comparability between joint and separate training, rather than to maximize absolute performance. Other continual optimizers might yield higher accuracies, mitigate part of the forgetting or better control plasticity, but we expect the underlying trends due to data co-observation to remain robust. Finally, while we demonstrate the effect from co-observation across supervised and self-supervised paradigms on moderate-scale vision tasks, representational dynamics can shift at massive scales. Future work must investigate whether billion-parameter over-parameterization naturally mitigates or exacerbates the need for simultaneous data observation. 10

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

ACKNOWLEDGMENTS This paper is part of a project that has received funding from the European Union under the Horizon 2020 research and innovation program (ERC project KeepOnLearning, grant agreement No. 101021347), and KU Leuven C1 project (Reference: C14/23/100). Part of this work was conducted while Abhishek was affiliated with ESAT-PSI, KU Leuven, Belgium. The computational resources and services used in this work were partly provided by the VSC (Flemish Supercomputer Center).

R EFERENCES Agil Aghasanli, Yi Li, and Plamen Angelov. Prototype-based continual learning with label-free replay buffer and cluster preservation loss. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6545– 6554, 2025. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference on Learning Representations (ICLR) workshop, 2017. Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15619–15629, 2023. Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149–198, 2000. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML), pp. 41–48. ACM, 2009. Florian Peter Busch, Roshni Ramanna Kamath, Rupert Mitchell, Wolfgang Stammer, Kristian Kersting, and Martin Mundt. Where is the truth? The risk of getting confounded in a continual world. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 6046–6076. PMLR, 2025. Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems, volume 33, pp. 15920–15930, 2020. Antonio Carta, Lorenzo Pellegrini, Andrea Cossu, Hamed Hemati, and Vincenzo Lomonaco. Avalanche: A pytorch library for deep continual learning. Journal of Machine Learning Research, 24(363):1–6, 2023. URL http: //jmlr.org/papers/v24/23-0130.html. Rich Caruana. Multitask learning. Machine Learning, 28(1):41–75, 1997. Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019. Zhiyuan Chen and Bing Liu. Lifelong Machine Learning, volume 12 of Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, second edition, 2018. doi: 10.2200/S00866ED2V01Y201807AIM039. URL https://doi.org/10.2200/ S00866ED2V01Y201807AIM039. Dongkyu Cho, Taesup Moon, Rumi Chunara, Kyunghyun Cho, and Sungmin Cha. Forget forgetting: Continual learning in a world of abundant memory. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=fvL8IIEPxG. MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16712–16721, 2022. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021. 11

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632:768–774, 2024. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy. Sebastian Farquhar and Yarin Gal. Differentially private continual learning. arXiv preprint arXiv:1902.06497, 2019. Enrico Fini, Victor G. Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Selfsupervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9621–9630, 2022. Stephen Grossberg. How does a brain build a cognitive code? Psychological review, 87(1):1, 1980. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016. Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumulation in continually learned representations and the issue of feature forgetting. Transactions on Machine Learning Research, 2024. Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, Lanqing Hong, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. How well does self-supervised pre-training perform with streaming data? In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=EwqEx5ipbOu. Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763, 2024. Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428, 2022. doi: 10.48550/arXiv.2210.04428. Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, Christoph Reinders, Sarthak Yadav, Joy Banerjee, Gábor Vecsei, Adam Kraft, Zheng Rui, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian Pfeiffer, Ben Cook, Ismael Fernández, François-Michel De Rainville, Chi-Hung Weng, Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et al. imgaug. https://github.com/aleju/imgaug, 2020. Online; accessed 01-Feb-2020. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. Thomas L Lee and Amos Storkey. Chunking: Continual learning is not just about distribution shift. CoLLAs, 2023. Zhizhong Li and Derek Hoiem. Learning without forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), volume 2, pp. 4, 2016. David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, pp. 6467–6476, 2017. Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International Conference on Learning Representations (ICLR), 2022. Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109–165. Elsevier, 1989. 12

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, Volume 4 - 2013, 2013. ISSN 1664-1078. doi: 10.3389/fpsyg.2013.00504. Saleh Momeni and Bing Liu. Achieving upper bound accuracy of joint training in continual learning. arXiv preprint arXiv:2502.12388, 2025. Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural Networks, 160: 306–336, 2023. Zixuan Ni, Haizhou Shi, Longhui Wei, Yueting Zhuang, Qi Tian, and Siliang Tang. On representation-level forgetting in class incremental learning: What’s the bottleneck? SSRN, 2023. doi: 10.2139/ssrn.4383369. German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019. Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3698–3707, 2023. Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001–2010, 2017. David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016. Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573–9585, 2020. Albin Soutif-Cormerais, Marc Masana, Joost Van De Weijer, and Bartłomiej Twardowski. On the importance of crosstask features for class-incremental learning. In ICML Workshop on Theory and Foundations of Continual Learning, 2021. URL https://arxiv.org/abs/2106.11930. Eli Verwimp, Rahaf Aljundi, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L. Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H. Lampert, Martin Mundt, Razvan Pascanu, Adrian Popescu, Andreas S. Tolias, Joost van de Weijer, Bing Liu, Vincenzo Lomonaco, Tinne Tuytelaars, and Gido M van de Ven. Continual learning: Applications and the road forward. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=axBIMcGZn9. Eli Verwimp, Guy Hacohen, and Tinne Tuytelaars. Same accuracy, twice as fast: continuous training surpasses retraining from scratch. arXiv preprint arXiv:2502.21147, 2025. Joshua T. Vogelstein, Jayanta Dey, Hayden S. Helm, Will LeVine, Ronak D. Mehta, Tyler M. Tomita, Haoyin Xu, Ali Geisa, Qingyang Wang, Gido M. van de Ven, Chenyu Gao, Weiwei Yang, Bryan Tower, Jonathan Larson, Christopher M. White, and Carey E. Priebe. Simple lifelong learning machines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(11):10033–10046, 2025. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362–5383, 2024. doi: 10.1109/TPAMI.2024.3367329. 13

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 374–382, 2019. Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3014–3023, 2021. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310–12320. PMLR, 2021. Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), pp. 3987–3995. PMLR, 2017. Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19148–19158, 2023. Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 133 (3):1012–1032, 2025.

14

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

A PPENDIX The following appendix provides supplementary details regarding our experimental setups, further details on our evaluation methodology, and quantitative results. The contents are structured as follows: • Appendix A details the optimization hyperparameters, dataset preprocessing, and augmentation pipelines for both the supervised and self-supervised pre-training regimes. • Appendix B expands upon our evaluation framework, describing the construction of the perfect-retention ensemble, the capacity-controlling dimensionality reduction via PCA, and the linear probing protocol, in full detail. • Appendix C provides empirical verification of the diagnostic ensemble, demonstrating its zero-forgetting properties on previously observed training data and ablating the potential confounding influence of forward and backward transfer using distillation-enhanced checkpoints. • Appendix D formally defines the continual learning mechanisms evaluated in this work, specifically our adaptations of Experience Replay and Knowledge Distillation (LwF and CaSSLe). • Appendix E presents additional results extending the supervised learning evaluation on ImageNet-100 to longer continual sequences of 8 and 12 chunks, demonstrating the robustness of the co-observation gap under increased data fragmentation. • Appendix F provides the comprehensive numerical results, corresponding to the empirical evaluations and figures presented in the main text Section 4.

A

D ETAILED E XPERIMENTAL S ETUP

Datasets and Augmentations. For CIFAR-100, training images (original resolution 32 × 32) are augmented using random cropping (padding of 4, reflect mode) and random horizontal flipping (p = 0.5). Images are normalized with µ = (0.485, 0.456, 0.406) and σ = (0.229, 0.224, 0.225). At evaluation, images are strictly evaluated at 32 × 32. For ImageNet-100 (IN-100), images are stored at 256 pixels on their longer side. During training, we apply a RandomResizedCrop to 224 × 224, a random horizontal flip, and RandAugment (2 operations, magnitude 9). At evaluation, images are center cropped to 224 × 224, normalized with µ = (0.4914, 0.4822, 0.4465) and σ = (0.2023, 0.1994, 0.2010). Supervised Learning Configurations. For supervised training, we utilize a ResNet-18 backbone. For CIFAR-100, the backbone is adjusted to accommodate the smaller input size, as proposed by He et al. (2016) (i.e., replacing the initial 7 × 7 convolution with a 3 × 3 convolution and removing the first max-pooling layer). The model is optimized using SGD with a momentum of 0.9, a weight decay of 5 × 10−4 , and gradient clipping set to 1.0. The learning rate follows a warmup phase constituting 5% of the total iterations, followed by a cosine annealing schedule with a peak learning rate of 0.1, and a final learning rate of 10−5 . To maintain consistent training dynamics across datasets, the total number of iterations is scaled to be equivalent to approximately 100 epochs for CIFAR-100 and 90 epochs for IN-100 on the respective full datasets. Specifically, training on CIFAR-100 proceeds for 9, 750 iterations per chunk with a mini-batch size of 128, while training on IN-100 proceeds for 11, 070 iterations per chunk with a mini-batch size of 256. The linear probe is trained mirroring the linear evaluation protocol of Fini et al. (2022). We use a fixed number of 100 epochs on the (optionally reduced) frozen representations, using SGD with a momentum of 0.9, an initial learning rate of 0.1, and no weight decay. The learning rate is decayed by a factor of 10 at 60 and 80 epochs. Self-Supervised Learning Configurations. For the self-supervised continual learning experiments, we evaluate two distinct frameworks with their respective standard configurations. Barlow Twins. The ResNet-18 encoder is paired with a 3-layer MLP projector (hidden and output dimensions of 2048) using BatchNorm and ReLU activations. The objective uses an off-diagonal scaling factor λ = 5 × 10−3 and an overall loss scale of 0.1. Two views are generated per image using random resized cropping (scale [0.08, 1.0]), horizontal flipping, color jitter, Gaussian blur, and random solarization. The model is optimized using LARS (learning rate 0.4, weight decay 10−4 , momentum 0.9) with a warmup-cosine schedule for 400 epochs per chunk and a batch size of 256. I-JEPA. The ViT-B/16 encoder (patch size 16, 224 × 224 input) is trained alongside an Exponential Moving Average (EMA) target encoder, with the momentum coefficient annealed from 0.996 to 1.0. The predictor is a depth-6 ViT with 15

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

an embedding dimension of 384. A block-masking strategy samples one context mask (85–100% of patches) and four target masks (15–20% of patches). The model minimizes the smooth-ℓ1 (Huber) loss and is optimized using Adam (learning rate 10−3 , weight decay 0.04) with a warmup-cosine schedule for 400 epochs per chunk at a batch size of 512.

B

M ETHOD DETAILS : E NSEMBLE

B.1

E NSEMBLE C ONSTRUCTION

The ensemble-baseline introduced in Section 2 is designed to provide a perfect-retention baseline by storing the exact state of the continually training model after each training phase. After completing training on experience t, a frozen copy of the feature extractor (backbone) fθt is appended to a persistent buffer B = {fθ1 , fθ2 , . . . , fθt }. Thereafter, these backbone weights are never updated. During evaluation, an input x is passed through all |B| frozen backbones, as illustrated in Figure 2. The resulting features are concatenated along the channel dimension to form a single combined representation z ∈ R|B|·d , where d is the feature dimension of a single backbone. This concatenated representation is then used for downstream linear probing. B.2

D IMENSIONALITY R EDUCTION

Because the ensemble concatenates representations from |B| checkpoints, its output dimension |B| · d scales linearly with the number of training phases. To ensure a fair comparison against single-model baselines and isolate feature quality from increased dimensional capacity, we apply an optional dimensionality reduction step to compress z back to a target dimension d′ = d before linear probing. For the reduction a PCA is fitted exclusively on the concatenated representations of the downstream training set, retaining the d′ components with the largest variance. The fitted projection matrix is then fixed and applied to the evaluation set to prevent information leakage. This reduction step is only applied when evaluating the ensemble. B.3

L INEAR -P ROBE E VALUATION

At each evaluation point, a newly initialized linear classification head W ∈ RC×din is instantiated and trained on the downstream training set, where C is the number of downstream classes and din is the appropriate feature dimension. To optimize training efficiency, all training-set inputs are passed through the frozen backbone(s) once, and the resulting representations are cached. The linear probe is then trained on these cached features using a standard cross-entropy loss. Similarly to Fini et al. (2022) the optimizer is an SGD (with momentum and either a multi-step or cosine annealing learning rate schedule). After training, the probe is evaluated on the downstream test set via a single forward pass, reporting top-1 accuracy.

C

E MPIRICAL VERIFICATION OF KNOWLEDGE RETENTION IN THE ENSEMBLE

Our diagnostic ensemble serves as analytical device to isolate the co-observation effect from the standard stabilityplasticity considerations as well as possible. To ensure the validity of this isolation, this section provides empirical verification that the ensemble reliably controls for catastrophic forgetting (perfect retention), and ablates potential confounding deficits regarding knowledge transfer. C.1

E MPIRICAL VERIFICATION OF PERFECT RETENTION

In Section 2.3, we introduce an ensemble baseline designed as an analytical device to achieve perfect retention (zero forgetting). To empirically corroborate that our ensemble baseline exhibits this stability, we mirror the diagnostic methodology utilized by Lee & Storkey (2023) and conduct an evaluation directly on the training data. Because data-incremental “chunking” scenarios do not inherently possess a separate validation set per chunk, evaluating performance on the previously seen training data serves as a direct proxy for information retention. As demonstrated in Figure 6, once the ensemble trains on a given chunk, its accuracy on that chunk’s training data never declines in any subsequent stage. Instead, we observe an increase in accuracy for chunk 0 and forward transfer w.r.t. later chunks. This occurs because the representations of past chunks are perfectly preserved, and concatenating 16

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

Figure 6: Empirical verification of zero forgetting in the diagnostic ensemble baseline. The linear probe accuracy is evaluated on the training sets of individual chunks (T0 through T3 ) across 4 sequential training stages. After a chunk is actively trained on, its train-set accuracy does not decrease in subsequent stages.

newly frozen backbones from later stages only provides the global linear probe with strictly more features to combine, effectively enabling backward transfer at the readout level. This confirms the ensemble operates as a strict perfectretention baseline. C.2

A BLATION ON KNOWLEDGE TRANSFER CAPABILITIES

While the ensemble perfectly mitigates the loss of previously acquired information post-hoc, it is constructed from a naive sequential continual learner. Consequently, the individual checkpoints inherently suffer from forgetting during the sequential training process itself, which can impair forward transfer (the ability of past learned features to facilitate future learning). This means the ensemble does not fully account for the seamless consolidation of knowledge via ”ideal” forward and backward transfer that a perfect, single continually learning model would possess. To gauge this potential confounding influence, we conduct an ablation where we construct the ensemble using checkpoints from models equipped with stronger transfer and retention properties, rather than naive sequential checkpoints. Specifically, we utilize knowledge distillation methods, i.e. LwF (Li & Hoiem, 2016) for the supervised setting, and CaSSLe (Fini et al., 2022) for the self-supervised setting. We choose distillation because it does not reintroduce data co-observation, preserving the validity of the ablation.

Figure 7: Ablation of forward/backward transfer using distillation-enhanced checkpoints. Comparing the standard diagnostic ensemble (built from naive sequential checkpoints) against an ensemble built from distillation-enhanced checkpoints (LwF/CaSSLe) across sequential chunks. Even with enhanced transfer properties, the ensemble at best barely surpasses the standard baseline and fails to close the gap to joint-incremental training. As shown in Figure 7, substituting the naive checkpoints with distillation-enhanced checkpoints yields marginal (in case of CaSSLe) to no (in case of LwF) improvements. The performance of this enhanced ensemble tightly tracks the standard ensemble baseline and remains significantly below incremental joint training. Combined with our conceptual arguments in Section 2.1, this supports that the residual performance gap between the ensemble and joint training can be attributed to the representational deficit caused by the lack of simultaneous data co-observation, rather than being solely an artifact of poor transfer interaction between isolated checkpoints. 17

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

D

M ETHOD DETAILS : C ONTINUAL L EARNING M ECHANISMS

Implementation of the experiments builds on the Avalanche CL library (Carta et al., 2023), the CaSSLe (Fini et al., 2022) framework for self-supervised continual learning, and the official codebase of I-JEPA (Assran et al., 2023). D.1

E XPERIENCE R EPLAY

Our implementation of experience replay maintains a persistent memory buffer M of past exemplars, with a buffer policy that stores m samples per observed task. Similar to a number of works that utilize exemplar replay in the literature (Rebuffi et al., 2017; Wu et al., 2019; Cho et al., 2026), during optimization on experience t, the buffer M is concatenated directly onto the current-task dataset Dt to form a single combined dataset Dt+ = Dt ∪ M. Mini-batches of size B are sampled uniformly from Dt+ . Consequently, the expected fraction of buffer samples in any given training batch is given by |M|/(|Dt | + |M|). This approach naturally mirrors the data balance observed during cumulative joint training and automatically adapts as the buffer fills over successive experiences. D.2

K NOWLEDGE D ISTILLATION (S UPERVISED : LW F)

For knowledge distillation in the supervised continual learning case, we utilize Learning without Forgetting (LwF) Li & Hoiem (2016). After training on experience t, a frozen copy of the model θt−1 is saved as the teacher. When training on experience t + 1, the teacher’s soft predictions over previously seen classes serve as targets for the current student model. Let z and zprev denote the logits of the student and teacher models, respectively. The distillation loss for a previous task τ is the Kullback-Leibler (KL) divergence between the softened distributions:

LτKD = KL σ zprev (1) Aτ /T ∥ σ(zAτ /T ) , where σ(·) is the softmax function, T = 2 is the temperature scaling as proposed by the original authors, and Aτ is the set of active class indices for task τ . The total loss is a weighted sum of the cross-entropy loss on the current experience and the distillation loss over all previously encountered tasks. To ensure stability, distillation is applied strictly over the output units Aτ corresponding to classes the model has actively trained on. D.3

K NOWLEDGE D ISTILLATION (S ELF -S UPERVISED : C A SSL E )

To adapt distillation to the self-supervised regime, we employ the CaSSLe framework Fini et al. (2022) using the Barlow Twins Zbontar et al. (2021) objective. Barlow Twins learns representations by driving the empirical crosscorrelation matrix C of two augmented views toward the identity matrix, utilizing an invariance term (diagonal) and a redundancy-reduction term (off-diagonal). In the CaSSLe decorrelative variant, at the end of experience t, the encoder fθ̃ and projector hψ̃ are frozen to act as the teacher. During experience t + 1, a lightweight, randomly initialized MLP predictor gϕ is added to the student model. The student processes the input through its encoder, projector, and the new predictor to yield predictions P. The teacher processes the same input to yield target projections Z̃. The distillation loss applies the Barlow Twins objective (LBT ) to align the student’s predictions with the teacher’s frozen projections: i 1h (2) Ldistill = LBT (PA , Z̃A ) + LBT (PB , Z̃B ) 2 The predictor gϕ is applied exclusively to the student network. This asymmetry serves two functions: it prevents representation collapse, and it decouples the student’s current-task projector from the rigid distillation constraint, allowing the student to adapt to new data while the predictor bridges the representation spaces. The total training objective is the sum of the standard Barlow Twins loss on the student’s current projections and the distillation loss Ldistill .

https://github.com/ContinualAI/avalanche https://github.com/DonkeyShot21/cassle https://github.com/facebookresearch/ijepa

18

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

E

E XTENDED CHUNKING EVALUATION

In the main text Section 4, our empirical evaluation utilizes a 4-chunk data-incremental setup. To demonstrate that the co-observation effect is not an artifact of this specific sequence length, but rather a robust property of the fragmented optimization, we extend our evaluation to longer sequences. We replicate the supervised learning setup on ImageNet100, but partition the global training dataset uniformly into 8 and 12 identically distributed chunks, respectively. The fixed iteration budget is scaled proportionally to ensure the total optimization steps across the entire sequence remain consistent with the 4-chunk and joint-training baselines.

Figure 8: The co-observation effect across extended sequences. Linear probing accuracy, reported as the mean and standard error over 3 seeds, for supervised learning on ImageNet-100 partitioned into 8 chunks (left) and 12 chunks (right). The performance gap between the diagnostic ensemble (perfect retention) and incremental joint (simultaneous observation) persists consistently across longer sequences. As illustrated in Figure 8, increasing the granularity of the data partitions does not alleviate the representational deficit. Across both the 8-chunk and 12-chunk sequences, the perfect-retention ensemble baseline successfully mitigates catastrophic forgetting relative to the naive sequential model, yet consistently falls short of incremental joint training. This sustained gap confirms that the co-observation effect remains a fundamental limiting factor regardless of the specific length or fragmentation of the continual learning sequence.

19

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

F

N UMERICAL RESULTS

Table 1: SL-CIFAR100 - LP-accuracy[%] CIFAR100. Runs reported as mean ± standard error over 3 runs. Method

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Naive Ensemble Ensemble PCA Incremental Joint Joint (offline)

60.80±0.31 60.78±0.30 60.83±0.37 61.15±0.16 –

63.54±0.17 65.93±0.22 65.81±0.12 68.22±0.03 –

63.92±0.20 68.19±0.37 68.12±0.31 72.45±0.03 –

64.64±0.16 70.39±0.04 70.25±0.06 74.29±0.31 75.18±0.55

Table 2: SL-IN-100 - LP-accuracy[%] IN-100 Runs reported as mean ± standard error over 3 runs. Method

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Naive Ensemble Ensemble PCA Incremental Joint Joint (offline)

73.24±0.48 73.22±0.42 73.25±0.43 73.24±0.48 –

76.66±0.16 77.48±0.16 77.47±0.21 79.58±0.12 –

77.98±0.29 79.20±0.24 79.15±0.20 81.85±0.10 –

78.27±0.21 80.55±0.10 80.57±0.11 82.75±0.15 83.68±0.11

Replay (mt = 1250) Replay (mt = 6250) Replay (mt = 12500)

73.07±0.22 73.06±0.22 73.25±0.07

76.99±0.14 77.67±0.12 78.05±0.07

78.35±0.63 79.54±0.13 80.68±0.15

79.41±0.23 80.55±0.23 81.27±0.24

LwF

73.35±0.23

77.41±0.15

79.44±0.22

79.93±0.25

20

Published at 5th Conference on Lifelong Learning Agents (CoLLAs), 2026

Table 3: Barlow-IN-100 - LP-accuracy[%] IN-100 Runs reported as mean ± standard error over 3 runs. Method

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Naive Ensemble Ensemble PCA Incremental Joint Joint (offline)

67.47±0.38 67.56±0.29 67.57±0.37 67.19±0.45 –

69.04±0.46 69.81±0.29 70.20±0.30 71.61±0.36 –

69.64±0.14 70.37±0.43 71.43±0.30 73.28±0.34 –

69.25±0.06 71.25±0.19 71.53±0.31 73.50±0.41 76.42±0.40

Replay (mt = 1250) Replay (mt = 6250) Replay (mt = 12500)

67.63±0.42 67.47±0.29 67.78±0.50

69.25±0.24 69.69±0.12 70.93±0.04

69.83±0.14 70.93±0.17 71.99±0.28

69.68±0.16 71.63±0.15 72.59±0.09

CaSSLe

67.62±0.25

69.96±0.39

71.66±0.28

72.07±0.23

Table 4: IJEPA-IN-100 - LP-accuracy[%] IN-100 Runs reported as mean ± standard error over 6 runs for Naive, Ensemble (PCA), and (Incremental) Joint. And over 3 runs for Replay. Method

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Naive Ensemble Ensemble PCA Incremental Joint Joint (offline)

55.87±0.27 55.88±0.26 55.91±0.26 55.80±0.28 –

55.00±0.38 58.33±0.41 57.08±0.55 59.59±0.32 –

54.63±0.63 59.11±0.51 58.12±0.67 61.83±0.62 –

53.83±0.60 59.82±0.46 58.83±0.69 62.64±0.55 69.35±0.38

Replay (mt = 1250) Replay (mt = 6250) Replay (mt = 12500)

55.93±0.69 55.85±0.31 56.56±0.10

56.18±0.34 57.07±0.71 58.72±0.06

56.28±0.88 57.29±0.37 59.41±0.67

54.79±0.99 56.59±0.65 60.21±0.99

21

Related documents

• The Epistemology of Forgetting#753498
• Intentional Forgetting of Autobiographical Memories: Current Neuroimaging Evidence and Moving Towards an Integrationist EEG-fMRI Approach#1108189
• Intentional Forgetting of Autobiographical Memories: Current Neuroimaging Evidence and Moving Towards an Integrationist EEG-fMRI Approach#1108137
• Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss#1118332
• Learning Is Rewiring : How brains actually change through practice / James R. Hartwell#487563