1
Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems
arXiv:2604.20745v1 [cs.LG] 22 Apr 2026
Beining Wu, Member, IEEE, and Jun Huang, Senior Member, IEEE
Abstract—Federated continual learning (FCL) allows distributed autonomous fleets to adapt collaboratively to evolving terrain types across extended mission lifecycles. However, current approaches face several key challenges: 1) they use uniform protection strategies that do not account for the varying sensitivities to forgetting on different network layers; 2) they focus primarily on preventing forgetting during training, without addressing the long-term effects of cumulative drift; and 3) they often depend on idealized simulations that fail to capture the realworld heterogeneity present in distributed fleets. In this paper, we propose a lifecycle-aware dual-timescale FCL framework that incorporates training-time (pre-forgetting) prevention and (postforgetting) recovery. Under this framework, we design a layerselective rehearsal strategy that mitigates immediate forgetting during local training, and a rapid knowledge recovery strategy that restores degraded models after long-term cumulative drift. We present a theoretical analysis that characterizes heterogeneous forgetting dynamics and establishes the inevitability of long-term degradation. Our experimental results show that this framework achieves up to 8.3% mIoU improvement over the strongest federated baseline and up to 31.7% over conventional fine-tuning. We also deploy the FCL framework on a real-world rover testbed to assess system-level robustness under realistic constraints; the testing results further confirm the effectiveness of our FCL design. Index Terms—Federated Learning, Continual Learning, Planetary Exploration, Lifecycle Knowledge Management
I. I NTRODUCTION
M
OBILE autonomous fleets are deployed across a range of domains where environments evolve throughout the mission lifecycle, from polar research stations and precision agriculture to planetary exploration [1]–[3]. Among these, planetary exploration missions, especially those on Mars, represent the most demanding instance: they have grown from single probes into coordinated fleets of autonomous rovers that work together to operate across geologically diverse regions [4], [5]. In these missions, semantic terrain segmentation plays a vital role in guiding rovers safely across unfamiliar regions and helping them recognize sites of scientific interest [6]. Because each rover acquires imagery in a different region, the data observed across the fleet is inherently heterogeneous, and local onboard decision-making requires models that remain up to date with each rover’s own operating conditions. At the same time, the bandwidth and delay limits of the Interplanetary Internet [7], [8] make it impractical to return large volumes of raw data to Earth for centralized training. Federated Learning Beining Wu and Jun Huang are with the Department of Electrical Engineering and Computer Science, South Dakota State University, Brookings, 57006 SD. Email: [email protected]., [email protected]. Manuscript received [Date]; revised [Date].
(FL) has therefore become a natural architectural choice [9]– [12], allowing rover fleets to train shared segmentation models collaboratively while exchanging only model parameters. Yet the reality of long journeys through space reveals a gap in current methods: models must evolve and adapt throughout a long-term mission lifecycle, rather than remain unchanged after their first deployment. As rovers travel through diverse regions, they encounter terrain types that were absent in earlier exploration zones, such as shifts from loose soil to exposed bedrock. In such settings, the fleet must learn new knowledge step by step while retaining to what it has already learned [13]. However, continual adaptation on resource-limited edge devices often leads to catastrophic forgetting. As new knowledge is absorbed, changes to model parameters can cause earlier knowledge to fade and be lost [14], [15]. In FL settings, such a degradation typically occurs at two timescales. At the training-time scale, rovers face immediate conflicts among gradients during local updates, as they seek to learn new tasks without weakening their performance on prior ones [16]. At the long-term scale, repeated federated aggregation of NonIID models throughout the mission introduces cumulative drift, which slowly erodes the model’s ability to recognize classes learned in earlier stages [17], [18]. These challenges call for federated continual learning (FCL), which enables distributed rover fleets to learn over time without forgetting what was learned before. We term a framework lifecycle-aware when it explicitly coordinates defenses across both temporal phases, coupling training-time forgetting mitigation with long-term degradation recovery through shared operational state. While FCL has been recently investigated in the AI communities [14]–[17], [19]–[27], existing research presents significant limitations in the following aspects. First, prior rehearsalbased strategies (replay while learning new) ignore the heterogeneous forgetting sensitivities of deep network layers [14], [15]. Although recent studies have begun to examine layerwise sensitivity [23]–[25], they still apply uniform protection strategies to all layers. Such a “one-size-fits-all” approach neglects the fact that different layers have distinct plasticity and stability requirements. Backbone layers benefit from continued adaptation as representations evolve, whereas classifier layers require strong stabilization to preserve decision boundaries [26]. As a result, existing rehearsal-based frameworks with uniform or static layer-wise protection cause harmful effects: over-protected layers that should remain adaptable, while under-protected layers that are more prone to forgetting [20]. Second, existing recovery mechanisms, including LowRank Adaptation [24], [25], Knowledge Distillation [19], and
2
Variational Inference [22], [27], depend heavily on iterative gradient-based optimization. Such methods often require multiple training epochs to converge, making them computationally expensive and vulnerable to ongoing multiplicative drift in federated learning [17]. Consequently, prior recovery approaches lack fast mechanisms capable of restoring degraded models with low computational and communication overhead. Third, although several theoretical FCL frameworks have been proposed [16], [21], most current studies are based on simulations that do not account for the strict system heterogeneity in real-world rover fleets. For example, these simulations often overlook differences in onboard storage capacity and the impact of limited communication resources. Therefore, solutions developed through simulation alone cannot guarantee reliable mission deployment in practice. In a nutshell, prior studies on FCL present the following significant research gaps: 1) Current rehearsal-based approaches that employ uniform or static layer-wise protection often result in overprotected layers requiring adaptability, while underprotected layers are more prone to forgetting. 2) Previous recovery methods exhibit slow convergence and are unable to restore degraded models while maintaining low computational and communication overhead. 3) Most existing studies are based on simulations that fail to account for the system heterogeneity in real-world rover fleets. Solutions developed solely through simulation cannot ensure reliable mission deployment in practice. To close the above research gaps, this work aims to design a lifecycle-aware FCL framework that leverages stratified rehearsal for short-term learning and meta-learned recovery for long-term learning to combat catastrophic forgetting. Specifically, we make the following contributions. • We propose a dual-timescale FCL framework that incorporates training-time (pre-forgetting) prevention and (post-forgetting) recovery. Under this framework, we design a Layer-Selective Rehearsal (LSR) strategy that mitigates immediate forgetting during local training, and a Rapid Knowledge Recovery (RKR) strategy that actively restores degraded models after long-term cumulative drift. • We present a theoretical analysis that characterizes heterogeneous forgetting dynamics over deep network layers and demonstrates the inevitability of long-term degradation. We also provide a formal justification for the LSR protection design and derive sample-complexity bounds for the performance of RKR compared with retraining. • We evaluate our FCL framework on three representative Mars terrain datasets (MarsScapes, S5Mars, AI4MARS), achieving up to 8.3% mIoU improvement over the strongest federated baseline and up to 31.7% over conventional fine-tuning. We then deploy the FCL framework on a real-world rover testbed to assess system-level robustness under realistic constraints, including extreme NonIID conditions and resource heterogeneity. The testing results further confirm the effectiveness of our FCL design. The remainder of this paper is organized as follows. Sec-
tion II briefly summarizes the related studies. Section III describes the system model and formulates the federated class-incremental learning problem. In Section IV, we present our proposed lifecycle-aware defense framework. Section V provides the theoretical analysis of forgetting dynamics and recovery bounds. We discuss the experimental results from both simulations and the physical testbed in Section VI, and draw the conclusion in Section VII. II. R ELATED W ORK A. Federated Learning in MAS For collaborative learning in mobile autonomous systems (MAS), federated learning (FL) trains a decentralized global model by aggregating network parameters from distributed autonomous agents. McMahan et al. [9] propose a communication-efficient aggregation strategy that enables collaborative training across edge devices without exchanging raw data. Li et al. [10] develop FedProx to address data heterogeneity across federated agents via introducing proximal terms into the local optimization objective. Recent works [1]–[3], [28]– [31] apply FL to autonomous systems, including vehicular platooning, connected autonomous vehicles, UAV trajectory planning, dual-level UAV–vehicle collaborative learning, precisionagriculture path planning, and communication-efficient federated unlearning for smart agriculture, where distributed agents must collaboratively learn models under communication constraints and extreme data heterogeneity. A complementary line of work on edge collaborative perception further studies distributed online learning for edge video analytics [32], taskoriented collaborative perception [33], and edge–aerial taskoriented communication for low-altitude navigation [34], while FL on heterogeneous IoT edges under missing modalities has also been investigated [12]. At the wireless and coordination substrate, parallel studies on AoI-aware DRL-based resource allocation [35], SWIPT-assisted D2D energy harvesting [36], fault-tolerant nano-communication switching [37], model-free cooperative optimal output regulation for multi-agent systems [38], and multi-agent reinforcement learning for Stackelberg security games in MEC [39] underpin the networking and control layer on which such federated autonomous systems operate. Similar challenges arise in planetary exploration missions where rover fleets must collaboratively learn terrain segmentation models across geologically diverse regions while preserving data locality. To enable learning new knowledge continuously in federated settings, He et al. [40] propose federated continual learning via server-side generative replay, though the method assumes identifiable task transitions and focuses on bounding immediate forgetting within individual task stages. Dong et al. [21] address federated incremental semantic segmentation by storing old global models for immediate distillation. The above FL methods [9], [10], [21], [40] assume static deployment and are evaluated at task completion, whereas missions require models to adapt throughout the operational lifecycle across multiple temporal scales. A parallel line of work studies learned terrain analysis for Mars rovers. SPOC [41] introduced CNN-based Mars terrain
3
classification and was later integrated into the rover-operations analytics pipeline MAARS [42]. MLNav [43] studies onboard learned navigation under flight-software constraints. Several terrain datasets have been released by the community, including AI4MARS [44], MarsScapes [45], and S5Mars [46], which we use in our evaluation. These works share an offline, singlerover, non-incremental training assumption: data is collected in situ, returned to Earth for labeling, and used to train a static model that is subsequently uploaded back to the rover. The federated, class-incremental regime that arises over a multiyear mission lifecycle is left open, which is the gap our work targets.
B. Continual Learning Continual learning (a.k.a. incremental learning or lifelong learning) focuses on identifying new categories continuously in dynamic real-world scenarios [16], [47]. A major challenge is catastrophic forgetting, where models learn new categories sequentially and have limited memory for storing old training data. Existing methods fall into regularizationbased approaches, generative replay, and exemplar memory construction [14]. Rebuffi et al. [14] propose iCaRL that maintains a fixed-size exemplar buffer to rehearse old classes during new task training. Buzzega et al. [48] design dark experience replay to store and replay past model predictions. Beyond sample-level rehearsal, recent work on shared spatial memory through predictive coding explores representation-level consolidation mechanisms that complement exemplar-based replay [49]. Recent advances [15], [50] extend incremental learning to semantic segmentation tasks, addressing challenges such as background shift and spatial consistency in dense prediction scenarios. Additional works [51]–[54] further develop adaptive strategies for conflict mitigation and instance replay mechanisms. Recent theoretical analyses [17], [55] reveal that catastrophic forgetting exhibits complex dynamics beyond immediate performance degradation, with model myopia and task confusion accumulating inevitably across extended task sequences. Empirical studies [20] further show that different network modules exhibit heterogeneous forgetting patterns, with only a few task-specific parameters sensitive to distribution shifts, while others can be shared across tasks. Layerselective protection in CL has been investigated along two directions. The first applies uniform or pre-specified structural protection, including binary freeze/adapt masks [20] and fixed low-rank adaptation modules at chosen layers [23]. The perlayer allocation in these methods is fixed at the outset and does not change with task dynamics. The second applies posthoc sensitivity analysis: a one-shot importance score identifies sensitive parameters after a task is completed, and rehearsal is then applied over the selected subset [20]. These methods [20], [23] target centralized settings. Prior incremental learning approaches [14], [15] evaluate forgetting at task completion and do not account for the cumulative degradation induced by repeated federated aggregation under extreme Non-IID conditions.
C. Lifecycle-Aware Knowledge Management In long-term scenarios, knowledge degradation across extended learning sequences remains a challenge despite training-time defense mechanisms. Recent theoretical work [17] reveals that catastrophic forgetting accumulates inevitably as incremental errors compound across sequential tasks. Memory buffer coverage limitations exacerbate this degradation as the number of learned classes grows [55]. A trivial solution to recover from such degradation is to periodically retrain the entire model on memory buffers, but this approach incurs prohibitive computational overhead and communication costs in resource-constrained autonomous systems. To enable efficient knowledge recovery, meta-learning-based approaches [56] learn transferable adaptation patterns that generalize across tasks. Wang et al. [19] employ episodic replay distillation that simulates task transitions during meta-training, enabling rapid few-shot adaptation through learned correction patterns. Recent works [22], [24] further improve recovery efficiency through parameter-efficient strategies such as lowrank adaptation and variational distillation. These methods [19], [22], [24] assume centralized settings with full model access and do not address the multiplicative degradation induced by repeated federated aggregation under Non-IID conditions. They also treat prevention and recovery as isolated operations rather than coordinating them across temporal scales. III. S YSTEM M ODEL AND P ROBLEM F ORMULATION A. System Model In a long-term Mars exploration mission, a fleet of K autonomous rovers, denoted as {Ck }K k=1 , is deployed to conduct scientific investigations under the wireless federated learning framework. Each rover is responsible for collecting data and performing local learning, while a central server {S} coordinates collaborative model training through periodic synchronization. Each rover is constrained by its onboard storage capacity Ck and, as such, can retain only a limited volume of historical data for training. The federated architecture adopted here is a consequence of the communication budget rather than a privacy requirement. Even if the aggregation server were placed on Mars rather than on Earth, the relay and surface links between rovers share the same order of bandwidth as the Mars–Earth relay path [57], so raw imagery from multiple rovers cannot be collected at a single node; only model parameters, which constitute a far smaller per-round payload, are compatible with the available link budget. As the mission progresses and rovers operate in new territories, they encounter previously unseen phenomena and knowledge not present in earlier regions. We model this T process as a sequence of learning tasks T t t=1 , where each task t T corresponds to data collected in a newly explored region. Since future tasks may revisit environments where earlier knowledge remains relevant, the learning system must preserve acquired knowledge while incorporating new information, avoiding knowledge override and catastrophic forgetting. The data observed by each rover is inherently heterogeneous due to spatially varying geological and environmental factors.
4
Specifically, each rover Ck samples data from its local distribution Pk , and the set of distributions Pk is non-independent and non-identically distributed (Non-IID) across the fleet. These characteristics give rise to learning challenges at two different time scales. At the training-time scale, rovers face immediate optimization challenges in each task as they balance learning new knowledge with preserving existing knowledge through local gradient updates. At the long-term scale, the cumulative effects over sequential tasks lead to progressive performance degradation on early-learned knowledge. B. Problem Formulation We consider a sequence of tasks, denoted as T = {T t }Tt=1 , where T is the total number of tasks. For each task t, the t dataset is given by T t = {(xti , yit )}N i=1 , where each sample t consists of an input RGB image xi ∈ RH×W ×3 and its corresponding pixel-wise label map yit ∈ Y H×W . Each pixel in the mask (label map) is assigned a semantic class from the label space Y t , and N t denotes the number of image-mask pairs in task t. The label space Y t contains C t new classes that are unique to the current task. For example, in Mars terrain segmentation, Task 0 provides N 0 training samples, each consisting of an RGB image x0i ∈ RH×W ×3 (a 3channel color image with height H and width W pixels) and its ground-truth label map yi0 ∈ Y H×W (a same-sized matrix where each spatial location (h, w) stores the terrain category of that pixel). Task 0 learns C 0 terrain classes Y 0 = {Soil, Sand, Bedrock}, meaning each pixel is classified as one of these three types. Task 1 then introduces N 1 images with C 1 classes Y 1 = {Gravel, Rocks}. Although Task 1 provides annotations only for its new classes, the model must be able to segment each pixel into one of the C 0 + C 1 classes learned so far. In the context of federated continual learning (FCL), a key constraint is that the label spaces for different tasks are strictly j non-overlapping, that is, Y t ∩(∪t−1 j=1 Y ) = ∅. This formulation reflects a practical reality of the mission lifecycle: as rovers progressively enter new geological regions, they encounter terrain types that were absent from earlier mission stages, and the complete set of terrain categories cannot be predetermined at deployment time. As a result, each new task introduces Pt−1 C t classes that are entirely different from the C o = i=1 C i classes in previous tasks. This requires the model to incrementally expand its segmentation capability to accommodate new classes, while maintaining good performance on previously learned classes. Each rover Ck receives a local subset Tkt ⊂ T t consisting of Nkt samples drawn from its regional distribution Pk . The local label space Ykt ⊆ Y t contains Ckt classes, and the global t label space is formed through the union: Y t = ∪K k=1 Yk . K The data distributions {Pk }k=1 are Non-IID due to regional heterogeneity. To mitigate catastrophic forgetting, each rover maintains an exemplar memory buffer Mk that stores representative image-mask pairs from previously learned classes. The memory capacity |Mk | is fixed due to onboard storage constraints. As the number of learned classes C o grows across tasks, the per-class sample allocation |Mk |/C o diminishes.
The learning process operates over iterative communication rounds. Within each task T t , the training proceeds for R communication rounds indexed by r = 1, . . . , R. At round r, the server S distributes the current global model Θr,t to a selected subset of rovers Cˆr ⊆ {C1 , . . . , CK }. Each selected rover Ck ∈ Cˆr then performs local optimization on the combined dataset Tkt ∪ Mk , seeking to minimize: min E(x,y)∼Tkt ℓ(fΘk (x), y) r,t Θk (1) + λE(x,y)∼Mk ℓ(fΘk (x), y) , where fΘk denotes the segmentation model parameterized by Θr,t k , ℓ is the pixel-wise cross-entropy loss, and λ balances learning on new task data against rehearsal on memorized samples. Upon receiving the locally updated models {Θr,t k }k∈Ĉ r from all selected rovers, the server performs federated aggregation to obtain the global model for the next round: P Θr+1,t = |Ĉ1r | k∈Ĉ r Θr,t k . The main goal of our FCL framework is to maximize segmentation performance in the entire mission lifecycle. To measure this, we use the average mean Intersection over Union (mIoU) calculated over all classes after the final task T is completed, since mIoU gives how well the model maintains segmentation accuracy for all classes seen during the mission. The changes in mIoU over time show two main types of degradation, each linked to a different phase of the mission lifecycle. During training within a task, the rover faces immediate conflicts between learning from new task data Tkt and keeping the knowledge already stored in the local model Mk . Each local update must balance learning new classes (plasticity) with keeping previous knowledge (stability). As a result, sudden drops in mIoU for earlier tasks right after training on T t can be observed. Over the long term, as the model progresses through the full sequence of tasks, a slower but ongoing degradation appears. Even if strategies during training help reduce immediate forgetting, repeated rounds of local updates and global aggregation, especially under strong Non-IID conditions, cause the representations of early-learned classes to drift. This problem becomes more serious as the memory buffer Mk covers less of each class when more classes are learned. Because of this, we observe a steady drop in mIoU for tasks learned earlier, which is due to the repeated accumulation of aggregation errors over the course of the mission. IV. P ROPOSED M ETHOD A. Initial Test and Observation To understand catastrophic forgetting in FCL, we conduct a controlled layer replacement experiment to isolate the impact of each network layer on performance degradation. The experimental procedure is designed as follows. First, we train a baseline model Θ0 on Task 0. Next, we train a model Θt on Task t using standard FCL. We then replace each layer of Θ0 individually with the corresponding layer from Θt , keeping all other layers fixed. The change in mIoU on Task 0 after each replacement quantifies the specific contribution of that layer to forgetting.
5
Output
Segmentation Mask
Segmentation Fc (t=0) Replace Fc (t=T) Head Deep Layers
Layer 3 (t=0) Layer 4 (t=0)
Replace
Replace
Layer 3 (t=T) Layer 4 (t=T)
Layer
mIoU↑ (∆) Contr.%
Shallow Layers
6.5
L1 After Task 1 After Task 2 After Task 4
44.7 (-0.3) 44.0 (-1.0) 43.6 (-1.4)
7.9 8.5 7.6 7.5
L2 After Task 1 After Task 2 After Task 4
44.8 (-0.2) 44.3 (-0.7) 43.9 (-1.1)
5.0 4.9 5.3 4.9
Deep Layers
Layer 2 (t=0)
Replace
Layer 2 (t=T)
Shallow Layers Layer 1 (t=0) Replace Layer 1 (t=T)
Input
-6.9
L3 After Task 1 After Task 2 After Task 4
45.4 (+0.4) 46.3 (+1.3) 46.5 (+1.5)
-8.9 -9.8 -9.3 -7.8
L4 After Task 1 After Task 2 After Task 4
45.2 (+0.2) 45.6 (+0.6) 45.9 (+0.9)
-4.8 -4.9 -4.6 -4.9
Segmentation Head
Task 0
Task t-1
Task t
(a) Layer replacement
Head After Task 1 After Task 2 After Task 4
87.1 41.8 (-3.2) 33.2 (-11.8) 21.4 (-23.6)
87.1 86.6 87.1 87.5
(b) Forgetting analysis
Fig. 1: Layer-wise sensitivity analysis. (a) Controlled replacement experiment where baseline layers from Θ0 are individually replaced with trained layers from Θt . (b) Quantitative forgetting contribution across different layer groups. We base this study on MarsScapes [45], the first panoramic Mars terrain dataset collected by the Curiosity rover at Gale Crater, under extreme Non-IID conditions, using a Dirichlet distribution with β = 0.1 across K = 30 rovers. Task 0 includes classes 0–4 (Soil, Sand, Bedrock, Gravel, Rocks) and achieves an initial mIoU of 45.0%. Task t introduces new terrain classes, while requiring the retention of Task 0 knowledge through the exemplar memory Mk . The results in Fig. 1 indicate significant variation in the forgetting performance over different network layers. Each layer group’s contribution is reported as its signed ∆mIoU P normalized by l |∆mIoUl | (unsigned shares sum to ≈ 100% 1 up to rounding), which is an empirical counterpart of the κl in Lemma 1 rather than a numerical identity. Specifically, the segmentation head contributes 87.1% of the total forgetting magnitude, with mIoU decreasing from 45.0% to 21.4%. In contrast, replacing deeper backbone layers yields negative forgetting of −6.9%, improving Task 0 performance from 45.0% to 46.5%. Shallow layers contribute minimally (6.5%), causing a 1.4 pp decrease in mIoU. Note that existing rehearsal-based methods [14], [40], which mitigate forgetting by maintaining memory buffers Mk and replaying samples from earlier tasks during new task training, apply the same level of protection to all layers. However, our test results demonstrate that this approach is not optimal, because applying uniform rehearsal constraints presents a dilemma: strong constraints limit beneficial adaptation in the backbone, while weak constraints allow significant forgetting in the segmentation head. This motivates us to design rehearsal mechanisms that enforce stability where necessary and maintain plasticity where it is beneficial. B. The Proposed Framework The proposed framework is shown in Fig. 2. The framework consists of two components for two different time scales:
Layer-Selective Rehearsal (LSR) operates during local training by applying stratified corrections via learned generators for shallow, deep, and segmentation head layers, while Rapid Knowledge Recovery (RKR) provides server-side meta-learned recovery when long-term degradation is detected. At the training-time scale, LSR applies heterogeneous protection strategies through learned generators that decouple plasticity and stability requirements across layer groups. At the longterm scale, RKR provides efficient repair mechanisms that restore degraded segmentation head states through learned correction patterns. LSR contributes a stratified, online-learned rehearsal strategy for the training-time scale, together with a closed-form suboptimality bound for any uniform-α rehearsal (Theorem 1). RKR contributes a single-pass recovery function meta-trained once on Task 0 and applied without further retraining at subsequent tasks, together with a sample-complexity statement (Corollary 1) that compares its fine-tuning cost against retraining. The design departs from prior layer-selective approaches: protection strengths are continuously stratified (αs < αd < αc ) rather than binary, and the corrections are produced by learned generators conditioned on the current gradient and memory features, so the per-layer adjustment adapts as training proceeds rather than being fixed in advance. The two components are combined because Theorem 2 indicates that training-time prevention alone cannot bound the longterm degradation under Non-IID federated aggregation, so a recovery mechanism is required in addition. 1) Layer-Selective Rehearsal (LSR): LSR applies layeraware protection strategies during continual learning. Unlike existing rehearsal methods [48] that impose uniform constraints on all model parameters, LSR separates the plasticity–stability trade-off by allowing the backbone to evolve where beneficial while explicitly stabilizing the segmentation head. To realize this design, LSR employs three learned generator functions, ϕs , ϕd , ϕc , which provide adaptive update corrections for shallow layers, deep layers, and the segmentation head, respectively. Each generator ϕl , where l ∈ {s, d, c}, takes as input compact statistics of the current layer parameters and gradient together with memory-derived features hm , and outputs a correction term ∆θl that modulates the standard gradient update. The resulting corrections differ in magnitude to reflect layer-specific sensitivity: shallow layers receive minimal intervention weighted by αs , deep layers receive moderate guidance weighted by αd , and the segmentation head receives strong stabilization weighted by αc , with αs < αd < αc . These generators are trained online alongside the model parameters, enabling LSR to adapt continuously as the semantic class space expands over sequential tasks. Each generator ϕl operates on compact per-channel summaries of the current layer rather than on its full parameter vector. Let zl denote the concatenation of the per-channel mean and variance of θl , and let gl denote the corresponding per-channel summary of the gradient ∇l . The generator input is [zl ; gl ; hm ], where hm is obtained by globally averagepooling the backbone feature maps over the exemplars in Mk . The generator output is a per-channel modulation factor that is broadcast to all parameters within the same channel to form
6
Task t
Task t+1
Deep
Class-Imbalanced Distributions
So Sa Be Pe Ro Bo Ou D F il nd dr bb ck ul tc ust ine de ro oc le s p k r
Shallow
Server: Recovery Module Moderate Guidance
Deep Generator
Lightweight Correction
Shallow Generator
Step 1: Meta-Training on Initial Task
Initial Task
Input
Samples
Task t
Task t+1
Deep
Class-Imbalanced Distributions
Shallow So
il
Sa
B P R B O D F nd edr ebb ock oul utc ust ine de ro oc le s p k r
Aggressive Stabilization Classifier Generator
Meta-Learner
Moderate Guidance
Deep Generator
Lightweight Correction
Shallow Generator
Performance Drift
Rapid Recovery
mIoU
The K-th Local Rover
Task t-1
Meta-Training
Step 2: Rapid Recovery Deployment
Output
Segmentation Head
Meta-Learner
mIoU
Samples
Task t-1
Aggressive Stabilization Classifier Generator
...
The l-th Local Rover
Output
Segmentation Head
Task Index
Meta-Learner
Training Epochs
Input
Fig. 2: Overview of the proposed dual-timescale framework. Left: Layer-Selective Rehearsal (LSR) operates during local training on rovers, applying stratified corrections through learned generators (ϕs , ϕd , ϕc ) that produce adaptive updates for shallow, deep, and segmentation head layers with heterogeneous protection strengths (αs < αd < αc ). Right: Rapid Knowledge Recovery (RKR) is meta-trained on the initial task and deployed when long-term degradation is detected, efficiently restoring segmentation head performance through learned recovery patterns (ψ). ∆θl . This design keeps the generator size independent of the layer width and enables ϕl to be instantiated as compact MLPs. The shallow layer generator ϕs makes lightweight correction with minimal architectural complexity. Given the perchannel summaries zs , gs ∈ R2Cs of the shallow-layer parameters and gradient (with Cs the shallow-layer channel count) and the memory encoding hm ∈ Rdm , the generator is instantiated as a compact multi-layer perceptron that produces correction: ∆θs = ϕs ([zs ; gs ; hm ]; ωs ),
(2)
where [·; ·; ·] denotes concatenation and ωs represents learnable parameters. The lightweight design reflects the minimal forgetting contribution (6.5%) while preserving plasticity for learning new task features. The deep layer generator ϕd is designed to guide the beneficial evolution observed in sensitivity analysis (-6.9%). It produces corrections that enhance rather than constrain adaptation: ∆θd = ϕd ([zd ; gd ; hm ]; ωd ),
(3)
where zd , gd ∈ R2Cd are the per-channel summaries of the deep-layer parameters and gradient. The moderately sized MLP learns to amplify update directions that are beneficial for both new task learning and old task preservation, allowing for natural feature evolution while memory-guided corrections prevent drift toward task-specific patterns. The segmentation head generator ϕc implements stabilization to address the dominant forgetting source (87.1%). Beyond parameters and gradients, it incorporates preserved o knowledge including class prototypes p ∈ RC ×d , the memory-pooled feature hm (backbone-pooled features of the exemplars in Mk , i.e., the same representation used in
Eqs. (2)–(3)), and geometric features a capturing inter-class angular relationships: ∆θc = ϕc ([zc ; gc ; p; hm ; a]; ωc ),
(4)
where zc , gc ∈ R2Cc are the per-channel summaries of the segmentation-head parameters and gradient. The architecture employs multiple specialized encoders for different input modalities, followed by a fusion layer that produces correction ∆θc . Operating on hm rather than on raw image-mask pairs keeps the input dimensionality tractable for a compact MLPbased generator. The richer input context enables precise boundary preservation while accommodating new class integration. The parameter update for each layer group l integrates standard gradient descent with learned correction weighted by αl : θl ← θl − η∇l + αl ∆θl ,
(5)
where η is the learning rate. The stratified weighting αs < αd < αc ensures minimal intervention in shallow layers, moderate guidance in deep layers, and aggressive stabilization in the segmentation head, directly reflecting their respective forgetting contributions. The generators must be trained online rather than pre-trained due to incremental evaluation characteristics of the FCL. After completing task t, models are evaluated on the cumulative test set spanning tasks 0 through t, requiring generators to adapt continuously to the expanding label space. Pre-trained generators with fixed capacity cannot accommodate the growing number of classes and evolving class relationships across T sequential tasks. During local training on rover Ck at round r of task t, we jointly optimize model parameters Θr,t = k {θs , θd , θc } and generator parameters Ω = {ωs , ωd , ωc }. The
7
selective rehearsal loss combines task learning with memory rehearsal: Ls = E(x,y)∼Tkt ℓ(fΘ (x), y) (6) + λE(x,y)∼Mk ℓ(fΘ (x), y) , where ℓ denotes pixel-wise cross-entropy loss and λ balances new task learning against old task preservation. The generators are updated to minimize forgetting while enabling model parameters to learn effectively. Note in each communication round that, selected rovers receive global model Θr,t from server S and perform local training with LSR. The generators {ϕs , ϕd , ϕc } operate locally during gradient computation, giving corrections that guide parameter updates according to the stratified rule. After local training, rovers transmit updated model P parameters to the server for aggregation: Θr+1,t = |Ĉ1r | k∈Ĉ r Θr,t k . Generator parameters Ω remain local and are not communicated, ensuring minimal overhead beyond standard federated learning. 2) Rapid Knowledge Recovery (RKR): Although LSR mitigates forgetting at the training-time scale, long-term degradation over sequential tasks remains challenging. This is because, on the one hand, prior work [17], [55] has established that catastrophic forgetting in continual learning is inevitable due to the accumulation of incremental errors over extended task sequences, which are further augmented by repeated aggregation under Non-IID conditions. On the other hand, while fine-tuning on memory buffers can partially restore performance, the communication overhead and computational cost make frequent complete retraining impossible in resourceconstrained rovers. To address these issues, we introduce RKR that learns to efficiently restore degraded segmentation head states by leveraging preserved knowledge for targeted corrections. The recovery function ψ operates exclusively on the segmentation head to address the dominant forgetting source. Given a degraded segmentation head state θc∗ ∈ Rdc after task t, the function generates correction ∆θc by integrating multiple sources of preserved knowledge. The architecture employs specialized encoders for degraded parameters (Es ), class prototypes (Ep ), memory exemplars (Em ), and geometric features (Eg ), which are fused through attention-weighted aggregation. The complete forward pass is formulated as: ∆θc = ψ(θc∗ , p, Mk , a; ξ) ⊤ X exp(q E (·)) i P =F · Ei (·) , ⊤ j exp(q Ej (·))
(7)
i∈{s,p,m,g}
where Es (θc∗ ), Ep (p), Em (Mk ), and Eg (a) produce encoded representations, q is a learnable query vector computing attention weights via softmax, and F denotes the fusion decoder network parameterized by ξ. The recovered segmentation head is obtained as θc+ = θc∗ + ∆θc , followed by brief fine-tuning on Mk to stabilize restoration. The recovery function is trained on task 0 using episodic procedures inspired by meta-learning principles [19]. Each episode simulates degradation by partitioning task 0’s label space Y 0 into training subset Y − and held-out subset Y + = Y 0 \ Y − . A degraded segmentation head θc∗ is constructed by
optimizing only on Y − , inducing artificial forgetting on Y + . The recovery parameters ξ are then optimized through a bilevel objective: X 1 ℓ(fθc∗ +ψ(·;ξ) (x), y) , (8) min EY − ,Y + ξ |DY + | (x,y)∈DY +
P where θc∗ = arg minθc |D 1 − | (x,y)∈D − ℓ(fθc (x), y), DY − Y Y and DY + denote data distributions over respective label subsets, and ℓ is the pixel-wise cross-entropy loss. The above episodic training learns general recovery patterns rather than class-specific corrections, which enables direct transfer to tasks t > 0 without retraining ψ. During deployment phase, each rover monitors cumulative mIoU Ikc after completing task t. When performance on early tasks drops below threshold τ , the rover invokes recovery by generating ∆θc using current prototypes p, memory buffer Mk , and geometric features a, then applies the correction θc+ = θc∗ +∆θc followed by brief fine-tuning. The recovery function parameters ξ remain constant in all tasks that requires no additional training or communication beyond the initial Task 0 meta-learning phase, maintaining the minimal overhead. C. Training Pipeline The training pipeline of our framework is presented in Algorithm 1. Task 0 serves as the initialization phase where all rovers train the initial model on their local data and construct memory buffers Mk following iCaRL [14]. Note that the recovery function ψ is trained during Task 0 via the episodic meta-learning procedure described in Section IV-B2, learning transferable recovery patterns that will be applied to all subsequent tasks without retraining. The learned generators {ϕs , ϕd , ϕc } are initialized and will be updated online throughout the training. For each task t ≥ 1, the training proceeds through R communication rounds. At round r, the server S distributes the global model Θr,t to selected rovers Cˆr (blue lines 6-16 of Algorithm 1). Each selected rover Ck performs local training by sampling batches from Tkt ∪ Mk and applying LSR. For each batch, the rover computes task gradients and generates layer-wise corrections via the three generators following Equations (2)-(4), then updates parameters according to the stratified rule in Equation (5). The generators Ω = {ωs , ωd , ωc } are jointly optimized with model parameters to minimize the selective rehearsal loss in Eq. (6). After local training, rovers upload their updated models to thePserver, which performs federated aggregation Θr+1,t = |Ĉ1r | k∈Ĉ r Θr,t k (orange lines 17-18 of Algorithm 1). Generator parameters remain local and are not communicated. After completing all R rounds for task t, each rover evaluates its cumulative mIoU on tasks {0, . . . , t} (see green lines 20-28 of Algorithm 1). When performance on early tasks drops below threshold τ , RKR is invoked. The rover generates correction ∆θc using the pre-trained recovery function ψ via Equation (7), applies the correction θc+ = θc∗ + ∆θc , and performs brief fine-tuning on memory buffer Mk . Since the recovery function parameters ξ remain fixed after Task 0
8
Algorithm 1 Training Lifecycle in FCL ( LSR Training , RKR Triggering , and Aggregation )
based bounds, gradient-variance decomposition) use standard tools and serve as building blocks for these three results.
Input: K rovers {Ck }K k=1 , T tasks, R rounds per task, threshold τ 1: Task 0 Initialization: 2: Initialize Θ0,0 , memory {Mk }, generators {ϕs , ϕd , ϕc } 3: Train recovery function ψ via episodic meta-learning 4: for t = 1 to T do 5: for r = 1 to R do 6: ▷ Rovers perform local training with LSR: 7: for each selected rover Ck ∈ Cˆr do r,t 8: Receive Θr,t ; Initialize Θr,t k =Θ ; t 9: for each batch (x, y) ∼ Tk ∪ Mk do 10: Compute ∇s , ∇d , ∇c from Ls (Eq. (6)); 11: Generate corrections (Eqs. (2)-(4)); 12: Update layers (Eq. (5)); 13: Update generators Ω; 14: end for 15: Upload Θr,t k to server; 16: end for 17: ▷ Server performs P aggregation: 18: Θr+1,t ← |Ĉ1r | k∈Ĉ r Θr,t k ; Distribute to rovers;
Definition 1 (Task-Level Forgetting). Let P t denote the global test distribution for task t. For task t′ < t, the forgetting of task t′ after training on task t is defined as: ′ ∆t,t := E(x,y)∼P t′ ℓ(fΘt (x), y) (9) − E(x,y)∼P t′ ℓ(fΘt′ (x), y) .
end for ▷ Long-term recovery triggering: for each rover Ck do Evaluate cumulative mIoU Ikc ; if Ikc < τ then 24: Generate ∆θc via ψ (Eq. (7)); 25: Apply θc+ ← θc∗ + ∆θc ; 26: Fine-tune on Mk for few epochs; 27: end if 28: end for 29: end for 30: return ΘR,T
19: 20: 21: 22: 23:
The cumulative forgetting at task t is: t−1
∆t :=
1 X t,t′ ∆ . t ′
(10)
t =0
We now present the assumptions required for our theoretical results. Assumption 1 (Layer-wise Smoothness). For each layer l ∈ {s, d, c}, the loss function is Ll -smooth with respect to layer parameters, i.e., for any θl , θ̃l : ∥∇θl ℓ(fΘ (x), y) − ∇θ̃l ℓ(fΘ̃ (x), y)∥ ≤ Ll ∥θl − θ̃l ∥,
where Ls , Ld , Lc are layer-specific smoothness parameters. Assumption 2 (Bounded Gradient Variance). For any model parameters Θ and rover k, the stochastic gradient has bounded variance: h E(x,y)∼Pk ∥∇Θ ℓ(fΘ (x), y) i (12) − ∇Θ E[ℓ]∥2 ≤ σ 2 . Assumption 3 (Distribution Heterogeneity). The Non-IID degree is characterized by heterogeneity parameter γ 2 : K
1 X ∇Θ E(x,y)∼Pk [ℓ] K
(13)
k=1
2
training, the process incurs minimal computational overhead and requires no additional communication with the server. V. T HEORETICAL A NALYSIS In this section, we establish theoretical foundations that support our dual-timescale framework. We first formalize the forgetting metric and state the necessary assumptions. We then characterize training-time forgetting through layerwise decomposition and prove the suboptimality of uniform rehearsal strategies. Finally, we analyze long-term degradation inevitability and derive sample complexity bounds for recovery mechanisms. The theoretical analysis in this section is intended to provide qualitative design guidance rather than tight performance bounds. The three results that go beyond standard machinery are: Lemma 1, a layer-wise decomposition of forgetting that provides the structural basis for stratified protection; Theorem 1, a closed-form suboptimality bound for any uniformα rehearsal; and Theorem 2, a long-term degradation lower bound that couples federated aggregation drift with memorybuffer-coverage dilution. The remaining steps (smoothness-
(11)
− ∇Θ E(x,y)∼P̄ [ℓ] 1 where P̄ = K
≤ γ2,
PK
k=1 Pk is the average distribution.
Assumption 4 (Memory Buffer Constraint). Each rover maintains a fixed-capacity memory buffer |Mk | = M . As tasks accumulate, the per-class coverage diminishes: M |Mk | = Pt−1 →0 i Co i=0 C
as t → ∞.
(14)
We acknowledge that the layer-wise smoothness assumption and the bounded gradient variance assumption are idealized conditions when applied to deep segmentation networks trained under strong heterogeneity. The theoretical results in this section should therefore be read as directional insight about the design: they indicate which failure modes are structural (layer-wise heterogeneous forgetting, long-term degradation under Non-IID aggregation) rather than artifacts of a particular training run. The experiments in Section VI are intended to provide empirical validation of these qualitative predictions.
9
A. Training-Time Forgetting Characterization We now characterize how catastrophic forgetting manifests heterogeneously across network layers and establish the theoretical foundation for layer-selective rehearsal. ′
Lemma 1 (Layer-wise Forgetting Decomposition). Let Θt = ′ ′ ′ {θst , θdt , θct } and Θt = {θst , θdt , θct } be model states after tasks t′ and t > t′ , respectively. Under Assumption 1, there exist layer-wise contribution coefficients κs , κd , κc ≥ 0 with κs + κd + κc = 1 such that: h X ′ ∆t,t = κl EP t′ ℓ fΘt′ ,l←t (x), y l∈{s,d,c} (15) i ′ − ℓ fΘt′ (x), y + O ∥Θt − Θt ∥2 , ′
′
′
where Θt ,l←t := Θt \ {θlt } ∪ {θlt }. ′
Proof. By Taylor expansion of the loss function around Θt : X ′ ℓ(fΘt ) − ℓ(fΘt′ ) ≈ ⟨∇θl ℓ, θlt − θlt ⟩ l∈{s,d,c}
+
′ ′ 1X t (θl − θlt )⊤ ∇2θl ℓ(θlt − θlt ). 2
(16)
l
Define the layer-wise contribution coefficient as: ′
∥∇θl ℓ∥ · ∥θlt − θlt ∥ . (17) κl := P t t′ l′ ∈{s,d,c} ∥∇θl′ ℓ∥ · ∥θl′ − θl′ ∥ P By construction, l κl = 1. The first-order term captures the impact of replacing individual layers, which corresponds to the controlled layer replacement experiment. The second-order terms are bounded by Assumption 1: for each layer l, we have ′ ′ ′ ∥(θlt −θlt )⊤ ∇2θl ℓ(θlt −θlt )∥ ≤ Ll ∥θlt −θlt ∥2 , yielding the stated ′ O(∥Θt − Θt ∥2 ) residual. Lemma 2 (Gradient Conflict Characterization). During training on task t, the gradient conflict between new task data and old task memory exhibits layer-wise heterogeneity. For classifier parameters: h i E ∥∇nc − ∇oc ∥2 L2 C t 2 (18) ≥ c · · Ex∼P o [ϕ(x)] 4 Co 2
− Ex∼P n [ϕ(x)]
rows corresponding to new classes. Let ei denote the i-th standard basis vector. The new task gradient can be approximated as: C o +C t 1 X ei ⊗ Ex∼P n [ϕ(x)], (20) ∇nc ≈ t C o i=C +1
while the old task gradient maintains existing class boundaries: Co
∇oc ≈
1 X ei ⊗ Ex∼P o [ϕ(x)]. C o i=1
Since {ei } are orthogonal and the two approximations populate disjoint row sets (old-class vs. new-class), the squared-norm difference decomposes into two component squared norms. Applying Lc -Lipschitz continuity with the row-normalization factors 1/C o and 1/C t yields the stated bound in terms of the feature-mean shift ∥Ex∼P o [ϕ(x)] − Ex∼P n [ϕ(x)]∥ directly. For deep layers, feature evolution is continuous across tasks. The gradient difference is bounded by: 1 (22) ∥∇nd − ∇od ∥ ≤ Ld · √ · ∥E[xn ] − E[xo ]∥. dd √ The 1/ dd factor arises from averaging over dd feature dimensions. Taking expectations and applying Cauchy-Schwarz inequality yields the stated upper bound. Theorem 1 (Suboptimality of Uniform Rehearsal). Let Rα apply a uniform weight α to all layers, and let RL apply stratified weights {αs , αd , αc } with αs ≤ αd ≤ αc . We compare the two at matched average regularization, writing αc = α(1 + δ) and αd = α(1 − δ ′ ) with κc δ ≈ κd δ ′ (LSR redistributes protection across layers without imposing a globally stronger total). Under Assumptions 1 and 2, if κc > κd , then for any task sequence of length T : (κc − κd )(αc − αd ) E[∆TL ] ≤ 1 − (23) 4α · E[∆Tu ], where ∆Tu and ∆TL denote cumulative forgetting under uniform and LSR strategies. Moreover, when κc − κd = Ω(1) and αc − αd = Ω(α), LSR achieves a constant factor improvement over uniform rehearsal. Proof. Let ∆tl denote the instantaneous forgetting contribution of layer l at task t. For uniform rehearsal with weight α applied to all layers, the cumulative forgetting decomposes as:
,
where ∇nc and ∇oc denote gradients on new and old task data, n o
P and P denote new and old data distributions. For deep layer parameters: h i L2 h i E ∥∇nd − ∇od ∥2 ≤ d · E ∥xn − xo ∥2 , dd
(19)
(21)
∆Tu =
X
κl ·
T X
(1 − α)T −t ∆tl ,
(24)
t=1
l∈{s,d,c}
where ∆tl denotes the instantaneous forgetting contribution of layer l at task t. For LSR with stratified weights, we have: ∆TL = κc
T X (1 − αc )T −t ∆tc t=1
where ϕ(x) denotes backbone features, dd is the deep layer dimension, and xn , xo are samples from new and old distributions.
+ κd
Proof. For the classifier with weight matrix Wc ∈ o t R(C +C )×d , the gradient on new task data primarily updates
+ κs
T X (1 − αd )T −t ∆td t=1 T X t=1
(1 − αs )T −t ∆ts .
(25)
10
Since αc > α > αd , we have (1 − αc )T −t < (1 − α)T −t < (1 − αd )T −t . The dominant term corresponds to the classifier due to κc ≫ κd (by Lemma 2). To quantify the improvement, consider setting αc = α(1+δ) and αd = α(1 − δ ′ ) where δ, δ ′ > 0. To maintain approximately equal total regularization, we require κc δ ≈ κd δ ′ . The difference in cumulative forgetting becomes: ∆Tu − ∆TL ≥ κc
T h X
i (1 − α)T −t − (1 − αc )T −t ∆tc
t=1
− κd
T h X
(26)
i
(1 − αd )T −t − (1 − α)T −t ∆td .
t=1
Using the approximation (1 − α)T −t − (1 − αc )T −t ≈ (T − t)δα(1 − α)T −t−1 for small δ, and noting that κc ∆tc ≫ κd ∆td by Lemma 2, the dominant term yields: ∆Tu − ∆TL ≥
(κc − κd )δαT (T − 1) ¯ ∆, 2
(27)
¯ is the average instantaneous forgetting. Dividing by where ∆ T ¯ and choosing δ = (αc − αd )/(2α) yields the stated ∆u ≈ T ∆ bound. Remark 1. Theorem 1 indicates that uniform rehearsal allocates regularization strength in proportion to layer count rather than in proportion to per-layer forgetting contribution. The closed-form factor is derived under the smoothness and bounded-variance assumptions and should be read as a directional statement about when stratification helps; the empirical ablation in Section VI provides the corresponding quantitative check. B. Long-Term Degradation Analysis We now analyze the inevitable accumulation of degradation over extended task sequences and establish theoretical foundations for recovery mechanisms. Lemma 3 (Aggregation-Induced Drift). Under Assumption P 3, for each federated aggregation round Θr+1,t = r,t 1 k∈Ĉ r Θk , the drift relative to the ideal global optimum |Ĉ r | Θ̂ satisfies: h i E ∥Θr+1,t − Θ̂∥2 i 2η 2 γ 2 η h r,t ≤ 1− E ∥Θ − Θ̂∥2 + , 2L |Cˆr |
(28)
(29)
After aggregation:
k∈Ĉ r
¯ Θr,t − Θ̂⟩ (31) ≤ ∥Θr,t − Θ̂∥2 − 2η⟨∇, 2 ¯ 2 + η ∥∇∥ , P 1 ¯ = where ∇ k∈Ĉ r ∇Θk E[ℓ] is the aggregated gradient. |Ĉ r | ¯ Θr,t − Θ̂⟩ ≥ ∥Θr,t − By the smoothness assumption, ⟨∇, Θ̂∥2 /L when Θr,t is sufficiently close to Θ̂. Furthermore, by Assumption 3: h i 2 ¯ − ∇ E[ℓ]∥2 ≤ γ . (32) E ∥∇ Θ̂ |Cˆr | Taking expectations and noting that ∇Θ̂ E[ℓ] = 0 at the optimum, we obtain: h i E ∥Θr+1,t − Θ̂∥2 i η2 γ 2 (33) η h r,t E ∥Θ − Θ̂∥2 + ≤ 1− . L |Cˆr | Applying the contraction factor (1 − η/L) ≤ (1 − η/(2L)) when η ≤ 1/L completes the proof. Theorem 2 (Inevitable Long-Term Degradation). Consider a FCL setting with T tasks, R rounds per task, and K rovers under Non-IID conditions characterized by γ ≥ γ0 for some constant γ0 > 0. Assume the existence of a perfect trainingtime defense mechanism such that instantaneous forgetting ∆ti = 0 for all t ∈ [1, T ]. Nevertheless, the long-term cumulative degradation satisfies the lower bound: C o 1/2 E[∆T ] ≥ Ω γ 2 · T · R · (34) M 2 3/2 ≥ Ω γ0 · T ·R , where M = |Mk | is the memory buffer size and ∆ti denotes instantaneous forgetting at task t. The second inequality uses C o = Ω(T ). Proof. Even with perfect instantaneous forgetting prevention (∆ti = 0), each aggregation round introduces drift according to Lemma 3. Let δ r,t denote the drift at round r of task t: h i η2 γ 2 δ r,t := E ∥Θr+1,t − Θ̂r,t ∥2 ≥ , (35) K where Θ̂r,t is the ideal model at that round. Accumulating over T tasks and R rounds per task:
t=1 r=1
Proof. Each local update can be written as:
Θr+1,t − Θ̂ = Θr,t − Θ̂ 1 X −η· ∇Θk E[ℓ]. |Cˆr |
∥Θr+1,t − Θ̂∥2
T X R X
where η is the learning rate and L = maxl Ll .
r,t Θr,t − η∇Θk E(x,y)∼Pk [ℓ]. k =Θ
Taking norms and applying the smoothness condition from Assumption 1:
(30)
δ r,t ≥ T · R ·
η2 γ 2 . K
(36)
The memory-buffer coverage effect (Assumption 4) implies diminishing rehearsal effectiveness as C o grows. Under iCaRL exemplar selection [14], with only M/C o samples per previously learned class, the memory-based rehearsal gradient ∇m is a finite-sample class-mean p approximation whose magnitude scales on the order of M/C o relative to the full-data gradient ∇f : r M ∥∇m ∥ ≲ ∥∇f ∥. (37) Co
11
This coverage limitation amplifies the effective drift. The actual degradation on old tasks must account for both the parameter drift and the diminished rehearsal effectiveness: r T X R X C o (t) T r,t ∆ ≥Ω δ · M t=1 r=1 (38) r T η2 γ 2 T R 1 X C o (t) . ≥Ω · K T t=1 M Pt−1 i Since C o (t) = i=0 C ≥ c · t for some constant c > 0 (assuming each task introduces at least a constant number of classes), we have: T T 1 Xp o 1 X√ C (t) ≥ ct = Ω(T 1/2 ). T t=1 T t=1
(39)
Substituting back and using γ ≥ γ0 yields: 1 ∆T ≥ Ω γ02 · T 3/2 · R · √ . (40) M When M is a fixed constant (Assumption 4), this simplifies to the stated bound. Theorem 2 does not argue against the continual-learning formulation; rather, it indicates that training-time prevention alone is insufficient over an extended mission horizon, so that a complete framework must include a recovery mechanism that operates at the long-term scale. The alternatives, either retraining from scratch at each new region or freezing the model after initial deployment, are ruled out respectively by the uplink budget and by the need to remain safe on newly encountered terrain. This observation is the direct motivation for the Rapid Knowledge Recovery component in Section IV. Corollary 1 (Sample Complexity of Recovery). Let the classifier degrade to state θc∗ after task t. To recover to an ϵapproximation of the optimal classifier θc+ using recovery function ψ, the required memory buffer size and fine-tuning complexity satisfy: Co · d c |Mk | = Ω , ϵ2 (41) 1 Ef = O log , ϵ where Ef denotes the number of fine-tuning epochs. In contrast, complete retraining from scratch requires Er = Ω(C o ) epochs to achieve the same accuracy level. Proof. The recovery function ψ learns to generate corrections through episodic meta-learning: θc+ = θc∗ + ∆θc ,
∆θc = ψ(θc∗ , p, Mk , a; ξ).
(42)
By Rademacher-complexity-based uniform-convergence results for multi-output function learning [58, Theorem 6.8 and Chapter 26], to learn a mapping from the combined input space (degraded parameters, prototypes, memory samples, geometric features) to a dc -dimensional output space with C o output classes at error ϵ, the required sample complexity is: Co · d c |Mk | = Ω , (43) ϵ2
where the complexity scales with C o · dc as the recovery function must distinguish correction patterns for C o classes across dc dimensions. This is an informal order-of-magnitude sketch; the sharpest hypothesis-class-dependent constants are not pursued here. After applying the learned correction ∆θc , fine-tuning on Mk exhibits exponential convergence due to the warm-start initialization. By strong convexity of the fine-tuning objective in a neighborhood of θc+ : ∥θcE − θc+ ∥2 ≤ (1 − µ)E ∥∆θc ∥2 ,
(44)
where µ > 0 is the strong convexity parameter and E is the number of fine-tuning epochs. Setting (1 − µ)E ≤ ϵ gives E = O(log(1/ϵ)). For complete retraining, the model must relearn decision boundaries for all C o classes from scratch. Even with memory buffer Mk , standard training requires traversing the data multiple times to converge, yielding Er = Ω(C o ) complexity in the worst case. Remark 2. The logarithmic fine-tuning bound in Corollary 1 uses strong convexity of the fine-tuning objective in a neighborhood of θc+ . This condition is an approximation in the deep-network setting: the objective is only locally wellbehaved around a warm-started initialization. Corollary 1 should therefore be read as a directional efficiency statement, and the empirical recovery curves in Section VI provide the corresponding validation. VI. P ERFORMANCE E VALUATION A. Datasets and Evaluation Metrics To evaluate our FCL framework under realistic planetary exploration conditions, we conduct experiments on three publicly available Mars terrain segmentation datasets that represent different operational contexts and domain characteristics. MarsScapes [45] is the first panoramic dataset designed for Martian terrain understanding. It consists of 18,460 images at a resolution of 512 by 512, all collected by the Curiosity rover at Gale Crater. The dataset includes pixel-level annotations for nine semantic categories: Soil, Sand, Gravel, Bedrock, Rocks, Tracks, Shadows, Background, and Unknown. These categories are essential for assessing rover traversability, since loose materials can increase mobility risks, while consolidated surfaces are safer for navigation. S5Mars [46] contains 6,000 high-resolution images at 1200 by 1200, also from Curiosity’s Mastcam. It covers nine categories, including soil, sand, bedrock, rock, ridge, trace, hole, sky, and rover. Notably, S5Mars introduces navigation-critical classes such as hole, which represents negative obstacles, and ridge, which indicates terrain undulation. Together, MarsScapes and S5Mars support a thorough evaluation of continual learning within a single mission. To evaluate generalization across missions, we use AI4MARS [44], which is NASA’s largest Mars terrain dataset. AI4MARS contains about 35,000 images from the Spirit, Opportunity, and Curiosity rovers, spanning three different landing sites. It provides four categories that are critical for traversability: Soil, Bedrock, Sand, and Big Rock. The domain shift between geological regions and rover platforms
12
TABLE I: Comparisons of mIoU (%) on MarsScapes dataset [45] under the setting of 5-1. Class IDs: 0-Soil, 1-Sand, 2-Bedrock, 3-Gravel, 4-Rocks, 5-Tracks, 6-Shadows, 7-Background, 8-Unknown. Class ID
0
1
2
3
4
5
6
7
8
mIoU
Imp.
Centralized (upper bound)
57.2±.14
53.8±.19
41.6±.17
41.5±.21
28.4±.24
25.6±.13
14.8±.22
15.2±.16
15.3±.25
32.6
ref.
Finetuning + FL PLOP [15] + FL CUE [50] + FL CS2 K [51] + FL CoMBO [52] + FL ADAPT [53] + FL EIR [54] + FL FBL [21]
32.6±.26 43.2±.18 46.3±.21 48.5±.14 50.1±.23 51.8±.17 55.2±.12 53.6±.24
18.4±.19 33.4±.24 36.2±.15 38.8±.27 40.6±.16 42.2±.29 43.5±.21 45.2±.13
0.0±.00 26.6±.13 29.8±.28 32.2±.19 33.9±.21 35.5±.14 37.3±.26 39.5±.18
0.0±.00 19.5±.27 23.6±.17 26.4±.23 28.2±.14 30.1±.26 31.4±.15 33.2±.29
0.0±.00 14.2±.16 18.1±.24 20.6±.11 22.4±.28 26.2±.18 24.3±.23 25.1±.15
0.0±.00 5.3±.31 8.5±.12 11.1±.26 13.2±.17 14.6±.23 16.4±.19 18.5±.22
0.0±.00 1.8±.19 4.2±.29 6.3±.17 12.5±.25 8.8±.12 9.6±.28 11.1±.16
0.0±.00 0.0±.28 1.4±.16 3.1±.31 4.3±.19 5.8±.27 7.1±.14 8.4±.25
4.2±.23 4.1±.22 6.8±.25 8.5±.14 5.4±.32 10.2±.21 11.8±.17 13.6±.11
6.1 16.5 19.4 21.7 23.4 25.0 26.3 27.6
⇑ 24.3 ⇑ 13.9 ⇑ 11.0 ⇑ 8.7 ⇑ 7.0 ⇑ 5.4 ⇑ 4.1 ⇑ 2.8
Ours
55.0±.16
51.8±.22
39.2±.11
39.0±.19
26.0±.27
23.7±.14
12.4±.23
13.0±.18
13.4±.26
30.4
–
Imp.
TABLE II: Comparisons of mIoU (%) on MarsScapes dataset [45] under the setting of 3-2. Class ID
0
1
2
3
4
5
6
7
8
mIoU
Centralized (upper bound)
58.5±.15
45.8±.22
42.0±.13
32.5±.18
29.0±.25
19.4±.14
15.5±.20
11.5±.23
16.8±.16
30.1
ref.
Finetuning + FL PLOP [15] + FL CUE [50] + FL CS2 K [51] + FL CoMBO [52] + FL ADAPT [53] + FL EIR [54] + FL FBL [21]
46.8±.27 47.8±.23 49.6±.19 51.2±.27 46.8±.16 50.4±.24 51.8±.15 52.2±.21
16.2±.23 30.4±.16 33.2±.26 35.5±.14 37.6±.28 44.3±.17 40.8±.23 41.5±.12
0.0±.00 23.2±.28 26.5±.13 29.1±.21 30.8±.17 32.2±.29 33.6±.18 35.2±.25
0.0±.00 16.3±.14 20.1±.24 22.8±.18 24.5±.23 30.6±.13 27.4±.27 28.2±.16
0.0±.00 10.6±.25 14.4±.17 17.1±.31 19.2±.14 20.4±.22 23.1±.12 21.8±.29
8.5±.21 3.4±.19 6.2±.29 8.8±.16 0.0±.00 12.1±.15 17.7±.24 14.8±.18
0.0±.00 0.0±.00 2.6±.15 4.5±.24 7.1±.21 6.2±.27 9.3±.19 7.6±.13
3.8±.19 0.0±.00 0.0±.00 1.8±.12 9.2±.32 4.4±.19 5.6±.28 7.4±.24
5.2±.24 2.8±.27 5.1±.22 0.0±.00 3.7±.18 8.5±.31 9.8±.16 11.1±.21
8.9 14.9 17.5 19.0 19.9 23.2 24.3 24.4
⇑ 18.9 ⇑ 12.9 ⇑ 10.3 ⇑ 8.8 ⇑ 7.9 ⇑ 4.6 ⇑ 3.5 ⇑ 3.4
Ours
56.2±.18
43.6±.27
39.8±.14
30.2±.22
26.8±.17
17.0±.31
13.2±.23
9.0±.15
14.4±.19
27.8
–
TABLE III: Comparisons of mIoU (%) on S5Mars dataset [46] under the setting of 5-1. Class IDs: 0-soil, 1-sand, 2-bedrock, 3-rock, 4-ridge, 5-trace, 6-hole, 7-sky, 8-rover. Class ID
0
1
2
3
4
5
6
7
8
mIoU
Centralized (upper bound)
74.8±.11
66.8±.22
58.2±.16
43.5±.25
36.8±.19
39.6±.13
28.0±.26
25.6±.15
27.2±.21
44.5
ref.
Finetuning + FL PLOP [15] + FL CUE [50] + FL CS2 K [51] + FL CoMBO [52] + FL ADAPT [53] + FL EIR [54] + FL FBL [21]
51.2±.26 62.4±.19 65.8±.23 67.4±.16 64.2±.27 68.6±.14 70.8±.21 69.2±.18
22.4±.19 48.6±.26 52.2±.14 55.6±.29 57.4±.18 61.6±.23 59.2±.15 62.4±.27
8.6±.23 42.4±.14 45.8±.28 48.6±.21 52.6±.23 51.6±.17 54.2±.29 53.8±.14
0.0±.00 35.8±.31 43.4±.17 39.4±.24 40.2±.15 41.6±.28 42.8±.18 39.6±.23
0.0±.00 36.4±.22 33.2±.25 34.8±.13 32.6±.31 33.8±.19 35.6±.24 34.2±.16
5.8±.21 20.2±.18 0.0±.00 10.8±.18 28.4±.14 35.6±.26 32.4±.17 33.8±.32
0.0±.00 0.0±.00 0.0±.00 27.8±.17 23.8±.26 25.2±.22 25.6±.31 26.5±.19
0.0±.00 12.8±.15 18.4±.32 22.6±.26 26.8±.19 20.4±.13 24.2±.22 21.8±.25
6.2±.25 0.0±.00 17.6±.19 19.4±.14 0.0±.00 25.8±.31 22.4±.16 24.2±.21
10.5 28.7 30.7 36.3 36.2 40.5 40.8 40.6
⇑ 31.7 ⇑ 13.5 ⇑ 11.5 ⇑ 5.9 ⇑ 6.0 ⇑ 1.7 ⇑ 1.4 ⇑ 1.6
Ours
72.5±.13
64.5±.24
55.8±.18
41.2±.27
34.6±.21
37.4±.15
25.8±.28
23.6±.17
24.8±.23
42.2
–
in AI4MARS introduces significant data heterogeneity, which presents challenges for federated learning.
We consider two experimental settings on MarsScapes and S5Mars. The first, referred to as the 5-1 setting, involves learning 5 base classes, followed by 4 new tasks, each introducing 1 additional class (T = 5). The second, the 3-2 setting, starts with 3 base classes and then adds 3 new tasks, each with 2 classes (T = 4). For AI4MARS, we use a 2-1 setting, where 2 base classes are learned first, followed by 2 new tasks with 1 class each (T = 3). This setup is intended to simulate crossmission knowledge integration, where new rover deployments encounter terrain types not present in earlier missions. As the main evaluation metric, we use mean Intersection over Union (mIoU) calculated over all classes after the final task. We also report per-class IoU to examine how forgetting patterns vary in different terrain categories.
Imp.
B. Implementation Details For fair comparison with state-of-the-art continual learning methods under federated settings, we adapt six representative segmentation approaches [15], [50]–[54] to federated learning by incorporating local training and federated aggregation protocols, along with one federated segmentation method [21] and a standard fine-tuning baseline. All adapted baselines share the same FedAvg aggregation as our framework and keep their method-specific mechanisms strictly client-local: PLOP’s [15] KD teacher is the client’s own previous-task model; the exemplar memory in CoMBO [52], CS2 K [51], EIR [54], CUE [50], and ADAPT [53] is per-client and never exchanged; FBL [21] is used in its native federated form. All baselines therefore operate under identical communication budgets. Additionally, to evaluate long-term recovery efficiency, we compare against three recovery strategies: low-rank adaptation [24], meta-learning based episodic distillation [19], and variational knowledge distillation [22]. All methods employ the identical segmentation backbone DeepLabV3+ [59]
13
TABLE IV: Comparisons of mIoU (%) on S5Mars dataset [46] under the setting of 3-2. Class ID
0
1
2
3
4
5
6
7
8
mIoU
Imp.
Centralized (upper bound)
72.5±.14
62.5±.21
54.2±.17
33.8±.19
29.0±.24
41.8±.12
34.2±.26
17.8±.16
27.7±.22
41.5
ref.
Finetuning + FL PLOP [15] + FL CUE [50] + FL CS2 K [51] + FL CoMBO [52] + FL ADAPT [53] + FL EIR [54] + FL FBL [21]
48.4±.28 51.8±.22 55.2±.16 57.8±.24 53.6±.18 58.8±.27 60.6±.15 59.2±.21
18.6±.24 37.4±.17 41.6±.28 44.8±.15 46.6±.26 52.2±.13 48.4±.24 50.6±.19
0.0±.00 31.2±.28 34.8±.19 38.2±.27 44.4±.14 40.6±.22 42.2±.18 43.4±.26
0.0±.00 24.6±.14 32.8±.25 29.4±.18 28.8±.31 31.4±.16 32.2±.27 30.2±.14
0.0±.00 20.4±.26 22.6±.13 27.8±.21 21.8±.17 24.4±.29 23.8±.14 26.2±.23
7.2±.22 10.6±.31 0.0±.00 15.6±.16 0.0±.00 25.4±.18 21.6±.23 23.8±.17
0.0±.00 0.0±.00 0.0±.00 13.6±.32 18.8±.21 15.2±.24 17.4±.31 16.2±.28
4.5±.18 6.8±.24 10.2±.17 16.8±.23 11.6±.15 13.4±.31 15.6±.16 14.8±.22
5.8±.26 6.4±.21 8.2±.31 0.0±.00 8.8±.27 13.4±.19 14.8±.28 13.6±.15
9.4 21.0 22.8 27.1 26.0 30.5 30.7 30.9
⇑ 29.8 ⇑ 18.2 ⇑ 16.4 ⇑ 12.1 ⇑ 13.2 ⇑ 8.7 ⇑ 8.5 ⇑ 8.3
Ours
70.2±.19
60.2±.28
51.8±.15
31.6±.22
26.8±.27
39.6±.14
31.8±.23
15.8±.18
25.0±.26
39.2
–
TABLE V: Comparisons of mIoU (%) on AI4MARS dataset [44] under the setting of 2-1. Class IDs: 0-Soil, 1-Bedrock, 2-Sand, 3-Big Rock. Class ID
0
1
2
3
mIoU
Imp.
Centralized (upper bound)
60.2±.16
51.0±.22
15.0±.19
18.6±.14
36.2
ref.
Finetuning + FL PLOP [15] + FL CUE [50] + FL CS2 K [51] + FL CoMBO [52] + FL ADAPT [53] + FL EIR [54] + FL FBL [21]
35.4±.28 42.8±.19 46.2±.24 50.6±.17 47.8±.26 54.2±.21 59.6±.15 56.8±.23
1.8±.21 13.4±.27 19.8±.16 26.2±.31 38.4±.14 33.6±.18 32.4±.28 42.8±.19
0.0±.00 0.0±.00 0.0±.00 3.4±.22 4.2±.29 8.2±.25 7.6±.17 10.4±.31
0.0±.00 0.0±.00 0.0±.00 0.0±.00 0.0±.00 0.0±.00 0.0±.00 8.2±.26
9.3 14.1 16.5 20.1 22.6 24.0 24.9 29.6
⇑ 24.2 ⇑ 19.4 ⇑ 17.0 ⇑ 13.4 ⇑ 10.9 ⇑ 9.5 ⇑ 8.6 ⇑ 3.9
Ours
58.4±.18
48.2±.24
12.6±.21
14.6±.16
33.5
–