Proceedings of Machine Learning Research vol :1–42, 2026
39th Annual Conference on Learning Theory
Convergence of Continual Learning in Homogeneous Deep Networks Matan Schliserman
SCHLISERMAN @ MAIL . TAU . AC . IL
Blavatnik School of Computer Science and AI, Tel Aviv University
Gon Buzaglo
GON . BUZAGLO @ PRINCETON . EDU
Princeton University
arXiv:2606.30559v1 [cs.LG] 29 Jun 2026
Itay Evron
ITAY @ EVRON . ME
Daniel Soudry
DANIEL . SOUDRY @ GMAIL . COM
Department of Electrical and Computing Engineering, Technion
Editors: Steve Hanneke and Tor Lattimore
Abstract We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models. Keywords: Continual learning, Lifelong learning, Algorithmic bias, Projection algorithms.
1. Introduction Continual learning focuses on training models on a sequence of tasks to incrementally accumulate expertise. This paradigm has gained traction with large-scale foundation models, where full retraining is often prohibitive due to computational costs, privacy regulations, or data retention constraints. In these settings, models must be finetuned sequentially as data arrives, while avoiding catastrophic forgetting (Robins, 1995; van de Ven et al., 2025) and maintaining plasticity for future learning (Dohare et al., 2024). Alongside the development of practical algorithms (e.g., Qiao et al. 2024; Behrouz et al. 2025; see Yang et al. 2025 for a survey), deepening our theoretical understanding of the underlying dynamics remains essential. We address this by developing analytical tools specifically for continual learning in homogeneous deep neural networks (DNNs). Theoretical literature has largely centered on continual linear regression due to its tractability, shedding light on critical aspects of the field: convergence under various task orderings (Evron et al., 2022, 2026; Kong et al., 2023; Attia et al., 2025), task similarity (e.g., Asanuma et al., 2021; Lin et al., 2023; Li and Hiratani, 2025; Tsipory et al., 2025), regularization (e.g., Zhao et al., 2024; Levinstein et al., 2025; Karpel et al., 2026), overparameterization (Goldfarb and Hand, 2023; Goldfarb et al., 2024), and algorithmic effects (Doan et al., 2021; Peng and Risteski, 2022; Peng et al., 2023). Other notable works study continual linear classification, primarily deriving convergence guarantees under explicit or implicit regularization (Evron et al., 2023; Jung et al., 2025). While illuminating, linear analysis cannot fully account for the complex, nonlinear dynamics of modern DNNs. © 2026 M. Schliserman, G. Buzaglo, I. Evron & D. Soudry.
S CHLISERMAN B UZAGLO E VRON S OUDRY
Some recent works analyze continual learning in nonlinear models, largely focusing on simplified settings, such as two-layer models that enable precise analyses (Lee et al., 2021, 2022; Li et al., 2025; Taheri et al., 2025). Others leverage perturbative, mean-field, or scaling-limit techniques, including regimes that interpolate between lazy and feature-learning dynamics, but are typically restricted to shallow architectures or infinite-width limits (Shan et al., 2024; Graldi et al., 2025). As a result, depth influences forgetting primarily through static feature statistics or last-layer adaptation. In contrast, our work analyzes continual learning in homogeneous deep networks of arbitrary depth, with all layers trained and without relying on lazy-training or infinite-width assumptions, thereby capturing fully nonlinear feature evolution across layers. A popular approach to stabilizing continual models and preventing forgetting is the use of explicit regularization in the parameter space (e.g., Kirkpatrick et al., 2017). These methods aim to bias the optimization toward previous iterates via: (λ) (λ) 2 Θt ← argminΘ Lt (Θ) + λ Θ − Θt−1 Bt , (1) where Bt ⪰ 0 skews the regularization toward directions deemed “important” for previous tasks— e.g., based on their Fisher information (see Benzing, 2022). Interestingly, even isotropic regularization (Bt = I) has proven beneficial both practically (e.g., Lubana et al., 2021; Smith et al., 2023) and theoretically (Li et al., 2023; Levinstein et al., 2025; Karpel et al., 2026). Our work adopts a standard analytical framework with weak isotropic regularization in the limit as λ ↓ 0, as detailed next. The limit of weak regularization is an influential regime for analyzing algorithmic biases— extending beyond continual learning—due to its tractability and connections to minimum-norm and max-margin solutions (e.g., Hastie et al., 2022; Aubin et al., 2020). Specifically, in “traditional” stationary classification, Rosset et al. (2004) proved that weakly regularized linear models converge1 to their max-margin counterparts. That is, linear models trained with margin-based losses—e.g., the logistic loss—on separable data recover the Hard-Margin SVM solution as regularization vanishes: in direction (2) θ (λ) = argmin L(θ) + λ ∥θ∥2 −−−−−−→ argmax min yi θ̄ ⊤ xi = argmin ∥θ̄∥ . λ↓0
θ
i
∥θ̄∥≤1
yi θ̄ ⊤ xi ≥1, ∀i
A later work by Wei et al. (2019) generalized this to DNNs, showing that the normalized margin of a weakly-regularized homogeneous model f converges to the max margin: (λ)
Θ min yi f (xi ; ∥Θ −−−−−→ max min yi f (xi ; Θ) . (λ) ∥ ) − i
λ↓0
∥Θ∥≤1
(3)
i
In continual learning, Evron et al. (2023) used weak regularization to study jointly-separable linear classification. They showed an equivalence between the iterates of weakly regularized continual linear classification and those of a sequential margin-separating projection algorithm. That is, defining (t) (t) a convex “margin set” Ct ≜ θ̄ yi θ̄ ⊤ xi ≥ 1, ∀i}, they showed: in direction 2 (λ) (λ) θt = argmin Lt (θ) + λ∥θ − θt−1 ∥2 −−−−−−→ θ̄t = argmin θ̄ − θ̄t−1 . (4) θ
λ↓0
t
θ̄∈Ct
t
We complete the analysis landscape by proving that weakly-regularized continual learning with (t) (t) homogeneous DNNs projects onto nonconvex margin sets, Ct ≜ Θ̄ yi f (xi ; Θ̄) ≥ 1, ∀i}: in direction 2 (λ) (λ) Θt = argmin Lt (Θ) + λ∥Θ − Θt−1 ∥2 −−−−−−→ Θ̄t ∈ argmin Θ̄ − Θ̄t−1 . (5) Θ
t
2
λ↓0
Θ̄∈Ct
t
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Table 1: Landscape of weak-regularization analysis across learning regimes and model classes.
Stationary setting Continual learning
Linear models
Homogeneous DNNs
Rosset et al. (2004) Evron et al. (2023)
Wei et al. (2019) Our work (2026)
After establishing an equivalence between continual classification in homogeneous models and sequential projections, we first show that, unlike previous results for linear models, global convergence is generally not guaranteed in homogeneous models. Encouragingly, we prove local convergence guarantees for these models through tools from the literature on nonconvex projections (e.g., Lewis and Malick, 2008; Dao and Phan, 2019). Lastly, we demonstrate the generality of these tools by extending the analysis from classification to regression. Overall, these tools provide a principled framework for understanding continual learning in deep neural networks. Summary of contributions.
The main contributions of this paper are:
1. We prove that weakly-regularized continual learning with homogeneous DNNs implicitly performs sequential margin projections. 2. We show that, unlike in prior work on linear models, these projection sets are not necessarily convex, leading to qualitative differences between settings. 3. We demonstrate that global convergence is not guaranteed; consequently, forgetting can be catastrophic, even for a simple homogeneous model with only 4 parameters and 2 tasks. 4. Bridging to the nonconvex projection literature, we establish local convergence guarantees for random and cyclic task orderings, with rates depending on model Lipschitzness and smoothness. 5. Finally, we extend our results to continual regression in homogeneous models, providing a unified framework for both classification and regression.
2. Continual Classification in Homogeneous DNNs as Sequential Projections Notation. We denote [n] ≜ {1, 2, . . . , n}, [z]+ ≜ max{0, z}, and Bδ (u) ≜ {v | ∥v − u∥ ≤ δ}. We call a function f : Rp → R positively homogeneous of degree r ∈ R if f (cz) = cr f (z), ∀c > 0. Models. Throughout the paper we consider r-positively-homogeneous models parametrized by weights Θ ∈ Rp . For example, a fully connected neural network with L ≥ 1 layers, no bias terms, and a positively homogeneous activation function of degree 1 (such as ReLU or leaky ReLU) is L-positively-homogeneous in its parameters Θ = (W1 , . . . , WL ): f (x; Θ) = WL σ(WL−1 σ(· · · σ (W1 x)))
=⇒
f (x; cΘ) = cL f (·; Θ),
∀c > 0 .
(6)
In this section, we focus on binary classification to establish our fundamental result: weakly regularized homogeneous models in continual learning converge to a sequential projection algorithm. We extend these results to regression in Section 4. In both settings, the projection perspective bridges our analysis to existing literature on nonconvex projections, facilitating rigorous convergence results. (λ)
1. In this introduction, we say that θ (λ) converges in direction to θ̄ if limλ↓0 ∥θθ(λ) ∥ = ∥θ̄θ̄∥ .
3
S CHLISERMAN B UZAGLO E VRON S OUDRY
2.1. Setting: Continual Classification Learning setup. We consider a setup with M classification tasks X(1) , y(1) , . . . , X(M ) , y(M ) consisting of feature matrices X(m) ∈ Rnm ×d and binary labels y(m) ∈ {−1, 1}nm . The learner is exposed to tasks sequentially in k ∈ N+ iterations according to a task ordering τ : [k] → [M ], which are commonly assumed to be cyclic or random, to capture adversarial vs. nonadversarial behaviors (Evron et al., 2022, 2023; Kong et al., 2023; Levinstein et al., 2025). That is, for iteration t ∈ [k], i.i.d.
τcyc (t) = ((t − 1) mod M ) + 1,
τiid (t) ∼ Unif{1, . . . , M } .
(7)
Our analysis assumes the model can separate all M training sets. This assumption follows prior theoretical work on continual linear classification (Evron et al., 2023; Jung et al., 2025), where it is justified by standard teacher assumptions or high-dimensional randomness (e.g., Cover, 1965). For the deeper architectures considered here, joint separability is a significantly milder requirement; sufficiently expressive networks can interpolate finite datasets under general conditions, a phenomenon rooted in universal approximation results (Cybenko, 1989). Assumption A (Joint Separability) Define the individual feasible sets n o (m) (m) Cm ≜ Θ̄ yi f (xi ; Θ̄) ≥ 1, ∀i ∈ [nm ] , ∀m ∈ [M ] .
(8)
We assume the intersection, i.e., the joint feasible set, is nonempty: C ⋆ ≜ C1 ∩ · · · ∩ CM ̸= ∅. Metrics. We follow the projection analysis for continual classification in Evron et al. (2023) and derive results for three quantities of interest: d(Θ̄, C ⋆ ) = minΘ̄⋆ ∈C ⋆ ∥Θ̄ − Θ̄⋆ ∥.
1. Distance to joint feasible set:
∀m ∈ [M ] , d(Θ̄, Cm ) = minΘ̄′ ∈Cm ∥Θ̄ − Θ̄′ ∥.
2. Distance to individual feasible set:
3. Forgetting: At iteration t, we define the forgetting of a task previously seen at iteration t′ ≤ t as τ (t′ ) the maximal hinge loss over its training points: Fτ (t′ ) (Θ̄t ) = maxi 1 − yi f (xi ; Θ̄t ) + . As noted by Evron et al. (2023), the hinge loss possesses favorable properties for analyzing continual learning. First, we will show that immediately after learning task m, the iterate resides in the feasible set, Θ̄ ∈ Cm ; consequently, the forgetting of the most recent task is zero. More importantly, lower forgetting—quantified here by a lower hinge loss—implies improved training margins, which are often tied to better generalization. 2.2. Fundamental Result: Continual Classification to Sequential Margin Projections Learning algorithm. We consider a learner that minimizes the margin-based logistic loss,2 defined for each task m ∈ [M ] as Lm (Θ) ≜
nm 1 X (m) (m) log 1 + exp(− yi f (xi ; Θ)) . nm
(9)
i=1
2. While we measure forgetting using the hinge loss—which can reach zero once the margin is sufficiently large—the learner optimizes the logistic loss. This choice aligns our model with standard deep learning practices and is made for simplicity (i.e., our analysis can be extended to other common loss functions).
4
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
In Algorithm 1 we adopt a common strategy in continual learning by regularizing parameter shifts to prevent catastrophic forgetting. Such methods effectively trade off model plasticity for increased stability. While several prominent works utilize weighted L2 regularization (often guided by Fisher information) to protect task-specific parameters (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Benzing, 2022), we focus on isotropic L2 regularization. This variant offers a lower memory footprint and greater analytical tractability while remaining empirically competitive (Hsu et al., 2018; Lubana et al., 2021; Smith et al., 2023). Furthermore, isotropic regularization has become a focal point of recent theoretical inquiries into model consolidation (e.g., Li et al., 2023; Levinstein et al., 2025; Karpel et al., 2026). Formally, in Algorithm 1 we study the regularization at the limit as λ ↓ 0 and show that its iterates converge to the sequential projections of Algorithm 2. Algorithm 1 Regularized Continual Learning
Algorithm 2 Sequential Margin Projections
(λ)
Initialization: Θ̄0 = 0 For each iteration t = 1, . . . , k:
Initialization: Θ0 = 0 For each iteration t = 1, . . . , k: (λ) (λ) 2 Θt ← argmin Lτ (t) (Θ) + λ Θ − Θt−1
Θ̄t ← Πτ (t) (Θ̄t−1 ) ≜ argmin Θ̄ − Θ̄t−1
Θ
2
Θ̄∈Cτ (t)
(λ) Output: Θk
Output: Θ̄k
Theorem 1 (Weakly-Regularized CL → Sequential Margin Projections) Consider an r r-positively-homogeneous model f (·; Θ) : X → R, i.e., f (x; cΘ) = c f (x; Θ) for all c > 0. Assume individual separability, i.e., nonempty feasible sets C1 , . . . , CM (implied by Assumption A). Then, as λ ↓ 0, Algorithm 1 trained with the logistic loss aligns with Algorithm 2. That is, for every iteration t ∈ [k], any sequence λ ↓ 0 admits a subsequence (λj ) and a point 2 Θ̄t ∈ Πτ (t) (Θ̄t−1 ) ≜ argminΘ̄∈Cτ (t) Θ̄ − Θ̄t−1 such that (λ )
Θt j
(λ )
Θt j
j→∞
−−−−−−−→
Θ̄t
.
Θ̄t
The full proof is provided in Appendix A. Below, we describe our proof techniques, contrast them with prior work, and outline the core ideas behind the derivation. Comparison to prior work. As discussed in the introduction, the limit of weak regularization is a standard analytical tool. In stationary classification—equivalent to the first iteration of Algorithms 1 and 2—weakly-regularized solutions have been linked to max-margin solutions in both linear (Rosset et al., 2004) and homogeneous models (Wei et al., 2019). More recently, Evron et al. (2023) extended this framework to continual linear classification. Similar to our approach, they adjust the projection reference point to capture how expertise accumulates across tasks. Our work completes this landscape by analyzing weakly regularized continual learning in the broader class of nonlinear, homogeneous DNNs (see Table 1). Technically, our proof of Theorem 1 differs from previous approaches. For instance, while Evron et al. (2023) rely on convex geometry and KKT optimality, these tools do not readily extend to our setting because the feasible sets induced by DNNs are nonconvex. Furthermore, Wei et al. (2019) rely on the pointwise convergence of the loss to a fixed limit. This notion of convergence is insufficient 5
S CHLISERMAN B UZAGLO E VRON S OUDRY
for the continual setting, where the regularization term is dynamic and defined relative to the shifting limit of the previous task. To address these challenges, we employ the framework of Γ-convergence. This allows us to handle nonconvex geometry by analyzing the global minimizers of a sequence of evolving functionals. Definition 2 (Γ-convergence; Braides, 2006, Theorem 2.1) Let Gn : Rp → (−∞, +∞] and G : Rp → (−∞, +∞]. We say that Gn Γ-converges to G if: (a) For every Θ ∈ Rp and every sequence Θn → Θ, G(Θ) ≤ lim inf n→∞ Gn (Θn ). (b) For every Θ ∈ Rp there exists a sequence Θn → Θ such that G(Θ) ≥ lim supn→∞ Gn (Θn ). The key property of Γ-convergence is that the Γ-limit of a sequence of functionals ensures the convergence of their respective minimizers to a minimizer of the limit, as formalized next. Lemma 3 (Fundamental Theorem of Γ-convergence; Braides, 2006, Theorem 2.10) Let (Gn )n∈N be a family of functionals Gn : Rp → (−∞, +∞]. Assume that (Gn )n∈N is equicoercive, i.e., that for every C ∈ R, {Θ ∈ Rp : Gn (Θ) ≤ C} ⊂ KC for some compact set KC . Then, if Gn Γ-converges to a functional G : Rp → (−∞, +∞] as n → ∞, the following hold: (a) Convergence of minimal values: limn→∞ inf Θ∈Rp Gλ (Θ) = minΘ∈Rp G(Θ). (b) Convergence of minimizers: Every sequence (Θn ) such that Θn ∈ argmin Gn admits a convergent subsequence (Θnj ) such that limj→∞ Θnj = Θ⋆ , for Θ⋆ ∈ argmin G. If the minimizer Θ⋆ of G is unique, then the entire sequence (Θn ) converges to Θ⋆ . Having established these preliminaries, we now detail the key ideas of our proof. Proof sketch of our Theorem 1. Let (λn ) satisfy λn → 0. For λ > 0 and t ∈ [k], define (λ) (λ) cλ,r = (log λ1 )1/r and Θ̂t = Θt /cλ,r . To establish convergence in direction, we first show (λ)
convergence of the scaled iterates Θ̂t . For this, we need to prove that there exists a subsequence (λ ) (λℓ ) such that Θ̂t ℓ → Θ̄t for some Θ̄t ∈ argminΘ̄∈Cτ (t) ∥Θ̄ − Θ̄t−1 ∥2 . (λ )
j We proceed by induction on t. Assume that for some subsequence (λj ), Θ̂t−1 → Θ̄t−1 ∈ 2 argminΘ̄∈Cτ (t−1) ∥Θ̄ − Θ̄t−2 ∥ , we define
(λ)
Gt (Θ) ≜
Lτ (t) (cλ,r Θ) (λ) + ∥Θ − Θ̂t−1 ∥2 . 2 cλ,r λ
A change of variables yields (λ)
Θt
(λ)
∈ argmin{Lτ (t) (Θ) + λ∥Θ − Θt−1 ∥2 }
⇐⇒
(λ)
Θ̂t
(λ)
∈ argmin Gt (Θ).
(λ )
Then, we show that, along (λj ), the functionals Gt j are equicoercive and Γ-converge to Gt (Θ) = 1Cτ (t) (Θ) + ∥Θ − Θ̄t−1 ∥2 , where 1C (Θ) = 0 for Θ ∈ C and ∞ otherwise. By Lemma 3, (λ )
any sequence of minimizers Θ̂t j admits a convergent subsequence whose limit is a global minimizer of Gt , establishing the inductive step. Since this holds for any (λn ) with λn → 0, and since the (λ) (λ) directions of the scaled iterates Θ̂t and the original iterates Θt are identical, the theorem follows. 6
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.5 1.0
w2
1.0
w2
w2
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
0.5 1.5
1.0
0.5 0.0 w1
0.5
1.0
1.5
(a) Linear model. Consider two 2D data points x1 = [5, −1]⊤ , x2 = [2, 2]⊤ , y1 = y2 = +1, and a linear model f (x; w1 , w2 ) = w1 x1 +w2 x2 . As explained in Evron et al. (2023), the resulting feasible set is a closed and convex affine polyhedral cone of w1 1 the form [ 52 −1 2 ] [ w2 ] ≥ [ 1 ].
1.0
0.5 1.5
1.0
0.5 0.0 w1
0.5
1.0
1.5
(b) Simplified ReLU. Consider a single 1D data point x = 2, y = +1, and a 1-positively-homogeneous model f (x; w1 , w2 ) = [w1 x]+ + [w2 x]+ . Here, the induced feasible set f (2; w1 , w2 ) ≥ 1 is nonconvex. Consequently, a single point can have multiple valid projections.
1.0
1.5
1.0
0.5 0.0 w1
0.5
1.0
1.5
(c) Squared linear model. Consider a single 1D data point x = 2, y = +1, and a 2-positively-homogeneous model f (x; w1 , w2 ) = w12 x − w22 x as in Woodworth et al. (2020). While comprised of two convex regions, the resulting feasible set f (2; w1 , w2 ) ≥ 1 is nonconvex.
Figure 1: Feasible sets under homogeneous models are not necessarily convex. In the two-parameter spaces depicted, only the linear model yields convex feasible sets and, consequently, unique projections.
3. Convergence Analysis from a Projection Perspective The established projection perspective (Theorem 1) facilitates analyzing continual learning through the lens of projection theory. Indeed, prior work has used classical results from alternating projections and Projections Onto Convex Sets (POCS) to analyze the convergence of continual linear models (Evron et al., 2022, 2023). However, these tools are not directly applicable in our nonlinear setting as feasible sets are no longer convex. While this raises several challenges—detailed below—the projection viewpoint remains informative for structuring our analysis of the induced dynamics.
3.1. Challenges: Projections onto Nonconvex Sets Extending projection-based analysis beyond convexity breaks classical analytical frameworks. Nonunique projections. As illustrated in Figure 1, linear models induce convex feasible sets for which projections are uniquely defined—but general homogeneous models yield nonconvex feasible sets where a projection ΠC (u) ∈ argminv∈C ∥u − v∥2 may admit multiple minimizers. This nonuniqueness is critical: Algorithm 2 may select an arbitrary minimizer at each step, potentially inducing qualitatively distinct dynamics and branching into trajectories with diverging behaviors. Such branching complicates any uniform analysis that must account for all admissible selection rules. Breakdown of projection properties. Another distinction is that convex projections monotonically approach the joint feasible set, i.e., d(ΠCm (θ̄), C ⋆ ) ≤ d(θ̄, C ⋆ ), due to operator nonexpansiveness, i.e., ∥ΠC (u) − ΠC (v)∥ ≤ ∥u − v∥ (Evron et al., 2023, Lemma 4.5 and Corollary D.1). While crucial for convergence, these properties fail in the nonconvex regime; for instance, nonuniqueness in Figure 1b implies that it is even possible that, roughly, ∥ΠC (u + ε) − ΠC (u)∥ ≫ 0 while ∥u + ε − u∥ = ε → 0. Overall, this removes primary analytical mechanisms used in prior work. In Section 3.2, we show how this property breakdown precludes global convergence, while Section 3.3 establishes local convergence for models initialized near the intersection. 7
S CHLISERMAN B UZAGLO E VRON S OUDRY
3.2. Lower Bound: Forgetting Can Be Catastrophic Evron et al. (2022) proposed that forgetting is truly “catastrophic” only when it fails to converge to zero in the infinite-task-sequence limit. In their continual linear regression setting, they demonstrated that cyclic or random task orderings eliminate forgetting as the number of iterations k → ∞. Similar convergence was later established for continual linear classification (Evron et al., 2023). While significant, these results are expected from a projection perspective. Both works prove an equivalence to sequential projections onto convex sets—specifically affine subspaces in regression and polyhedral cones in classification (see Figure 1a). Such processes are well-known to converge to the intersection under cyclic or random orderings (Deutsch and Hundal, 2006; Nedić, 2010). We now ask: Can forgetting be catastrophic in homogeneous models whose projections are nonconvex? We answer in the affirmative. We show that the sequential margin projections of Algorithm 2 may fail to converge even as k → ∞. This means that continual learning on homogeneous models can forget catastrophically, i.e., never reaching a configuration that satisfies all tasks simultaneously. Technically, we employ a standard 2-positively-homogeneous model that is linear in the data but nonlinear in its parameters (e.g., as in Woodworth et al., 2020), as illustrated in Figure 1c. Definition 4 (Squared Linear Model) For inputs x ∈ Rd , we define a 2-positively-homogeneous model parameterized by Θ̄ = (u, v) where u, v ∈ Rd : f x; u, v
=
2
2
u ,x − v ,x
=
d X
u[i]2 − v[i]2 x[i] .
i=1
We construct a specific case exhibiting catastrophic forgetting with as few as M = 2 tasks and p = 4 learnable parameters. This highlights a major qualitative gap between convex and nonconvex projection interpretations of continual learning: whereas for linear models, catastrophic forgetting arises only in the limit of M → ∞ tasks (Evron et al., 2022, 2023), our nonconvex setting may fail with minimal complexity. Moreover, unlike the convex case—where task repetition drives iterates toward joint feasibility—alternating between M = 2 tasks here does not lead to convergence. Theorem 5 (Catastrophic Forgetting in Squared Linear Models) Let f (x; Θ̄) be a squared linear model in d = 2 parameterized by Θ̄ = (u, v) where u, v ∈ R2 . Fix ε ∈ (0, 0.01) and define two tasks 1 1 X(1) = x⊤ X(2) = x⊤ y(1) = y(2) = (1) . 1 = 2 = 9 +ε, 1−ε , 9 −ε, −1−ε , (a) The two tasks are jointly separable, i.e., C ⋆ ≜ C1 ∩ C2 ̸= ∅ where the feasible sets are C1 = {Θ̄ ∈ R4 | f (x1 ; Θ̄) ≥ 1}, C2 = {Θ̄ ∈ R4 | f (x2 ; Θ̄) ≥ 1}. (b) Algorithm 2 forgets catastrophically: d Θ̄t , C ⋆ ≥ 2, maxm∈{1,2} Fm (Θ̄t ) ≥ 1.5, ∀t ≥ 0. The proof, provided in Appendix B, relies on the fact that for the parameters Θ̄ = (u, v) to lie in the intersection C1 ∩ C2 , it must hold that u[1] ≥ 2. However, we show that the sequential projection dynamics, when initialized at the origin Θ̄0 = 0, are restricted to the subspace defined by u[1] = 0 and v[1] = 0. Namely, under the specific geometry of our construction, each projection step modifies only u[2] and v[2], leaving the other components at zero. Consequently, the iterates Θ̄t are effectively trapped in a subspace bounded away from the joint feasible set, i.e., C1 ∩ C2 . Importantly, the failure in Theorem 5 is not an artifact of a “degenerate” dataset or an adversarial task ordering: it persists over a nonzero measure of datasets under any ordering. Thus, global convergence guarantees require further assumptions in this nonconvex setting, as introduced next. 8
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Moreover, the lower-bound construction in Theorem 5 extends beyond the case of zero initialization. In particular, a similar phenomenon can arise for arbitrary (and in particular random) initializations within a bounded ball. By appropriately adjusting the ratio between the coordinates in the construction, one can ensure that the distance to the jointly feasible set C ⋆ is arbitrarily larger than the initialization radius. In this regime, the sequential projection dynamics behave similarly to the zero-initialization case: only a subset of the coordinates is updated, even though reaching a global solution requires movement in other directions. 3.3. Upper Bounds via Local Convergence Analysis As we established, global convergence is generally unattainable in the nonconvex setting, even when individual tasks are solved to optimality. Thus, we focus on local convergence analysis, as is common in nonconvex optimization, where global guarantees are elusive (e.g., Bottou et al., 2016). A practical motivation is the sequential finetuning of large pretrained foundation models. These models often exhibit small zero-shot loss, implying an initialization near a joint solution. Our analysis shows that local convergence can be guaranteed in this regime: if the initialization is sufficiently close to the joint feasible set C ⋆ , the sequence of projections converges to C ⋆ at a linear rate. We establish this result when f is β-smooth3 and G-Lipschitz4 for every data point x. In particular, both assumptions hold for DNNs with smooth activations (e.g., squared ReLU), where the Lipschitz constant may depend on architectural properties such as the network depth and homogeneity degree, as well as the magnitude of the data. We prove the following local convergence theorem. Theorem 6 (Convergence Rate for Lipschitz, Smooth Homogeneous Models) Consider a G-Lipschitz, β-smooth, r-positively-homogeneous model f (·; Θ̄) : X → R. Assume joint separar2 bility, i.e., nonempty C ⋆ = ∩m Cm ̸= ∅ (Assumption A). Let ε hold (1−ε)1M −1 = 1 + 2(M −1)G 2 ∥Θ⋆ ∥2 ⋆ ⋆ ⋆ ⋆ and δ = εr/β∥Θ ∥. Then, there exists Θ ∈ C such that if Θ̄0 ∈ Bδ (Θ ), the sequential projections of Algorithm 2 converge linearly with a rate depending on the task ordering, i.e., ∀k ∈ N+ , 1 ⋆ ⋆ r2 k • Random: E max Fm (Θ̄k ) ≤ E d(Θ̄k , C ) ≤ exp − M E d(Θ̄0 , C ). 2 ⋆ 2 4G ∥Θ ∥ G τiid m∈[M ] τiid τiid 1 ⋆ r2 k max Fm (Θ̄k ) ≤ d(Θ̄k , C ) ≤ exp − M 2 4G2 ∥Θ⋆ ∥2 d(Θ̄0 , C ⋆ ). • Cyclic (M | k): G m∈[M ] The proof is provided in Appendix C. Here, we make a few remarks and then outline the proof idea. Remark 7 (Comparison to Linear Model) For a linear model, the local convergence rates above recover the global rates of Evron et al. (2023) for both cyclic and random task orderings. Remark 8 (Nonzero Initialization) Local convergence requires the initialization of Algorithm 2 to lie within a neighborhood Bδ (Θ⋆ ). While we prove Theorem 1 for the case of zero initialization, this does not necessitate that Θ⋆ is near the origin. Indeed, by using a technical modification, it is possible to extend the analysis to other fixed initializations where the initialization of Algorithm 2 scales according to the weak regularization’s λ. This could be the case, for instance, when Algorithm 2 is initialized with a model pretrained using weight decay with the same order of λ, a change that does not alter the underlying geometric intuition. 3. f (x; ·) is β-smooth if ∀u, w, f (x; u) − f (x; w) − ⟨∇f (x; w), u − w⟩ ≤ β2 u − w 4. f (x; ·) is G-Lipschitz if ∀u, w , f (x; u) − f (x; w) ≤ G u − w .
9
2
.
S CHLISERMAN B UZAGLO E VRON S OUDRY
Remark 9 (Order of Limits) The number of iterations k may be arbitrarily large to guarantee convergence, potentially requiring task repetitions. Following (Evron et al., 2023, Remark 4.6), we clarify that our analysis takes the limit λ ↓ 0 after fixing k. Proof sketch. As in the convex case where proofs are based on properties of POCS (Evron et al., 2023), our proof of Theorem 6 leverages properties of projections on nonconvex sets. However, as discussed, nonconvex projections can be ill-behaved; they are often nonunique and tiny perturbations can cause large “jumps”. As a result, nonconvex projections are typically analyzed under regularity conditions that rule out pathological geometry and ensure the projection is locally stable, which is essential for any convergence guarantees (e.g., Lewis and Malick, 2008; Lewis et al., 2009; Dao and Phan, 2019). The resulting proof consists of two steps. 1. First step: Smoothness and Lipschitz continuity imply regularity. We show that a smooth and G-Lipschitz homogeneous model satisfies the regularity properties required for local convergence. We rely on tools from variational analysis and set regularity (e.g., Rockafellar and Wets, 1998) to assert that the feasible sets induced by homogeneous DNNs satisfy the needed local regularity conditions. In particular, we establish the following two conditions. Definition 10 ((ε, δ)-Regularity) Let C be a nonemptysubset of Rp , w ∈ Rp , ε ≥ 0 and δ > 0. Let ΠC (w) project w onto C and define NCprox (w) ≜ λ(z − w) z ∈ Π−1 C (w), λ ≥ 0 . We say that C is (ε, δ)-regular at w if x, y ∈ C ∩ B(w; δ), u ∈ NCprox (x)
=⇒
⟨u, x − y⟩ ≤ ε∥u∥ · ∥x − y∥.
Definition 11 (κ-Linear Regularity of Set Collection) A system {Ci }i∈I is κ-linearly regular on a subset U ⊆ X if , for all x ∈ U , d(x, ∩i∈I Ci ) ≤ κ maxi∈I d(x, Ci ) . In Lemmas C.10 and C.11 in Appendix C we show that, in the smooth and Lipschitz case, homogeneous models satisfy the conditions given in Definitions 10 and 11. 2. Second step: Invoke convergence under regularity. We invoke a local linear convergence result for nonconvex projections. In particular, the following proposition shows that under the conditions of Definitions 10 and 11, the iterates of Algorithm 2 exhibit local convergence to the intersection C ⋆ both in random and cyclic orderings. For random ordering, the proof appears in Appendix C, while the cyclic case follows directly from Dao and Phan (2019, Corollary 5.10). Proposition 12 (Stepwise decrease under random and cyclic TM projections) Consider⋆ closed p ⋆ sets C1 , . . . , CM ⊂ R with nonempty intersection C ≜ m=1 Cm ̸= ∅, and let Θ ∈ C ⋆ . Assume that there exist constants κ > 0, ε ∈ [0, 1), and δ > 0 such that: (a) Each set Cm is (ε, δ)-regular at Θ⋆ . ⋆ (b) The collection {Cm }M m=1 is κ-linearly regular on the ball Bδ/2 (Θ ). Then, we get one-step decrease depending on the task ordering, 1 (a) Random ordering. Let τ = τiid , t ≥ 1. If Θ̄t−1 ∈ Bδ/2 (Θ⋆ ) and ρiid ≜ 1−ε − M1κ2 < 1, 2 ⋆ 2 ⋆ it holds that Eτ (t) d (Θ̄t , C ) Θ̄t−1 ≤ ρiid d (Θ̄t−1 , C ). (b) Cyclic ordering. Let τ = τcyc and t such that M | t. If, Θ̄t ∈ Bδ/2 (Θ⋆ ) and 1 2(M −1) 1 ρc ≜ (1−ε)1M −1 − (M −1)κ < 1, then, d(Θ̄t+M , C) ≤ ρcyc d(Θ̄t , C). 2
10
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
4. Extension: Continual Regression in Homogeneous DNNs as Sequential Projections We now show that the techniques developed above are general enough to extend naturally to continual regression. This extension requires only minor adjustments to the definitions. Notational adjustments. For a task m ∈ [M ], the label vector is no longer binary, but rather y(m) ∈ Rnm . Consequently, the feasible sets are now defined as, n o (m) (m) Cm ≜ Θ ∈ Rp f (xi ; Θ) = yi , ∀i ∈ [nm ] , ∀m ∈ [M ] . (10) Accordingly, the joint separability assumption (Assumption A) is now a joint realizability assumption, still requiring C ⋆ ≜ C1 ∩ · · · ∩ CM ̸= ∅. Furthermore, instead of the logistic loss, we now optimize the mean squared loss, i.e., nm 1 X (m) (m) 2 , ∀m ∈ [M ] . (11) f (xi ; Θ) − yi Lm (Θ) ≜ nm i=1
We still consider d(·, Cm ) and d(·, C ⋆ ) (as in Section 2.1), but redefine the forgetting with respect to the squared loss instead of the hinge loss, i.e., τ (t′ ) τ (t′ ) 2 Fτ (t′ ) (Θ̄t ) = max f (xi ; Θ̄t ) − yi , ∀t ∈ [k] , ∀t′ ≤ t . (12) i
Result. Analogously to our classification result, we establish the following theorem for the convergence of Algorithm 1 to Algorithm 2 as λ ↓ 0 for the regression case. Theorem 13 (Weakly-Regularized Continual Regression → Sequential Projections) Consider a model f (·; Θ) : X → R. Assume individual realizability, i.e., nonempty feasible sets C1 , . . . , CM (Eq. (10)). Then, as λ ↓ 0, Algorithm 1 trained with the squared loss aligns with Algorithm 2. That is, for every iteration t ∈ [k], any sequence λ ↓ 0 admits a subsequence (λj ) and a point Θ̄t ∈ Πτ (t) (Θ̄t−1 ) ≜ argminΘ̄∈Cτ (t) Θ̄ − Θ̄t−1
2
(λ )
j→∞
such that Θt j −−−−−−−→ Θ̄t ∈ Π(Θ̄t−1 ) .
Proofs for this section appear in Appendix D; as with classification, the analysis uses Γ-convergence. 4.1. Convergence for Overparameterized Homogeneous Regression Models Mirroring the classification case (Theorem 5), we first show that the 2-positively-homogeneous squared model of Definition 4 may suffer from catastrophic forgetting. Lemma 14 (Catastrophic Forgetting in Squared Models: Simplified Version) Under the same task construction of Theorem 5, thetasks are jointly separable, yet Algorithm 2 forgets catastrophically. That is, C ⋆ ̸= ∅ but d Θ̄t , C ⋆ ≥ 2 and maxm Fm (Θ̄t ) ≥ 3, for every iteration t ≥ 0. The formal statement and its proof appear in Appendix D. Specifically, we show that in the classification case (Theorem 5), the current iterate Θ̄t always lies on the boundary of the latest feasible set; thus, the dynamics of Algorithm 2 is identical for regression and classification. After establishing that global convergence is not guaranteed in the general homogeneous regression setting, we now show that local convergence can still be achieved. In contrast to our classification result, the local convergence result for regression requires the additional assumption (m) ymin = minm,i |yi | ̸= 0. This assumption is imposed to ensure that the (ε, δ)-regularity condition holds (see Lemma D.3 for more details). 11
S CHLISERMAN B UZAGLO E VRON S OUDRY
Consider Theorem 15 (Convergence Rate for Lipschitz, Smooth Homogeneous Models) a G-Lipschitz, β-smooth, r-positively-homogeneous model f (·; Θ̄) : X → R. Assume joint real(m) izability, i.e., nonempty intersection C ⋆ = ∩m Cm ̸= ∅. Let ymin = minm,i |yi | ̸= 0. Let ε hold 2 ymin εr 1 r ⋆ ⋆ ⋆ = 1 + 2(M −1)G 2 ∥Θ⋆ ∥2 and δ = β∥Θ⋆ ∥ . Then, there exists Θ ∈ C s.t. if Θ̄0 ∈ Bδ (Θ ), the (1−ε)M −1 sequential projections of Algorithm 2 converge linearly with a rate depending on the task ordering, i.e., ∀k ∈ N+ , 2 1 r2 ymin 2 ⋆ 2 ⋆ • Random: E d (Θ̄0 , C ). E max Fm (Θ̄k ) ≤ E d (Θ̄k , C ) ≤ exp − Mk2 2G2 ∥Θ ⋆ ∥2 G τiid m∈[M ] τiid τiid 2 1 r2 ymin 2 ⋆ k • Cyclic (M | k): max Fm (Θ̄k ) ≤ d (Θ̄k , C ) ≤ exp − M 2 2G2 ∥Θ⋆ ∥2 d2 (Θ̄0 , C ⋆ ). G m∈[M ]
5. Discussion Our results frame continual learning in homogeneous deep networks as a geometric process governed by sequential projections onto task-induced margin sets, unifying several empirical and theoretical observations. Below, we discuss additional connections to prior work and highlight promising directions for further inquiry. Geometry as the source of catastrophic forgetting. Our results suggest that catastrophic forgetting is a fundamental geometric consequence of nonconvexity. Unlike linear models with convex feasible sets (Evron et al., 2023), homogeneous DNNs induce nonconvex sets where even exact sequential projections may fail to reach a point in the joint feasible set. Our lower bound in Section 3.2 demonstrates that this occurs even in low-dimensional models under benign task orderings, proving that nonconvex geometry alone is sufficient to preclude global convergence. The effect of depth. The convergence rates in Theorems 6 and 15 are governed by the ratio r/G, where r is the homogeneity degree and G is the Lipschitz constant. While depth increases r (see Eq. (6)), the Lipschitz constant G typically grows exponentially with the number of layers (e.g., Szegedy et al., 2014; Virmaux and Scaman, 2018). Consequently, increased depth tends to decrease the r/G ratio, thereby degrading the local convergence rate. This provides a theoretical mechanism for the findings of Guha and Lakshman (2024), who observe that deeper networks suffer from exacerbated forgetting (see also Mirzadeh et al., 2022; Ramasesh et al., 2020). A compelling avenue for future work is to characterize how specific architectures influence the geometry of margin sets (Eq. (8)) and their corresponding regularity moduli. Characterizing convergence in deep continual models. Empirical studies have shown that task repetition can mitigate forgetting even without explicit algorithmic intervention (Lesort et al., 2023; Hemati et al., 2025). While recent analytical work has explained this in linear models through cyclic or random task orderings (Evron et al., 2022, 2023, 2026; Kong et al., 2023; Levinstein et al., 2025; Attia et al., 2025), such analyses often rely on the NTK regime to approximate deep models. Our framework more directly reflects the intrinsic nonconvex dynamics of deep architectures. We hope the analytical foundation laid here facilitates a deeper understanding of failure modes, the benefits of pretraining, and the conditions for local stability in deep continual learning. 12
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Acknowledgments The research of MS was Funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 101078075). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This work received additional support from the Israel Science Foundation (ISF, grant numbers 2549/19 and 3174/23), a grant from the Tel Aviv University Center for AI and Data Science (TAD) and from the Len Blavatnik and the Blavatnik Family foundation. GB thanks Elad Hazan and Princeton University for the financial support. The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them.
13
S CHLISERMAN B UZAGLO E VRON S OUDRY
References Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154, 2018. Haruka Asanuma, Shiro Takagi, Yoshihiro Nagano, Yuki Yoshida, Yasuhiko Igarashi, and Masato Okada. Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks. Journal of the Physical Society of Japan, 90(10):104001, Oct 2021. Amit Attia, Matan Schliserman, Uri Sherman, and Tomer Koren. Fast last-iterate convergence of sgd in the smooth interpolation regime. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025. Benjamin Aubin, Florent Krzakala, Yue Lu, and Lenka Zdeborová. Generalization error in highdimensional perceptrons: Approaching bayes error with convex optimization. Advances in Neural Information Processing Systems, 33:12199–12210, 2020. Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. Nested learning: The illusion of deep learning architectures. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Frederik Benzing. Unifying regularisation methods for continual learning. AISTATS, 2022. Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization Methods for Large-Scale Machine Learning. 2016. ISSN 0036-1445. doi: 10.1137/16M1080173. Andrea Braides. A handbook of Γ-convergence. In Handbook of Differential Equations: stationary partial differential equations, volume 3, pages 101–213. Elsevier, 2006. Thomas M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, EC-14(3): 326–334, 1965. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303–314, 1989. Minh N Dao and Hung M Phan. Linear convergence of projection algorithms. Mathematics of Operations Research, 44(2):715–738, 2019. Frank Deutsch and Hein Hundal. The rate of convergence for the cyclic projections algorithm i: Angles between convex sets. Journal of Approximation Theory, 142(1):36–55, 2006. Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 1072–1080, 2021. Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768–774, 2024. 14
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory (COLT), pages 4028–4079. PMLR, 2022. Itay Evron, Edward Moroshko, Gon Buzaglo, Maroun Khriesh, Badea Marjieh, Nathan Srebro, and Daniel Soudry. Continual learning in linear classification on separable data. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 9440–9484. PMLR, 23–29 Jul 2023. Itay Evron, Ran Levinstein, Matan Schliserman, Uri Sherman, Tomer Koren, Daniel Soudry, and Nathan Srebro. From continual learning to sgd and back: Better rates for random task orderings in continual linear models. In 37th International Conference on Algorithmic Learning Theory, 2026. Daniel Goldfarb and Paul Hand. Analysis of catastrophic forgetting for random orthogonal transformation tasks in the overparameterized regime. In International Conference on Artificial Intelligence and Statistics, pages 2975–2993. PMLR, 2023. Daniel Goldfarb, Itay Evron, Nir Weinberger, Daniel Soudry, and Paul Hand. The joint effect of task similarity and overparameterization on catastrophic forgetting - an analytical model. In The Twelfth International Conference on Learning Representations, 2024. Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, and Lorenzo Noci. The importance of being lazy: Scaling limits of continual learning. In Forty-second International Conference on Machine Learning, 2025. Etash Kumar Guha and Vihan Lakshman. On the diminishing returns of width for continual learning. In Forty-first International Conference on Machine Learning, 2024. Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022. Hamed Hemati, Lorenzo Pellegrini, Xiaotian Duan, Zixuan Zhao, Fangfang Xia, Marc Masana, Benedikt Tscheschner, Eduardo Veas, Yuxiang Zheng, Shiji Zhao, Shao-Yuan Li, Sheng-Jun Huang, Vincenzo Lomonaco, and Gido M. van de Ven. Continual learning in the presence of repetition. Neural Networks, 183:106920, 2025. Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:1810.12488, 2018. Hyunji Jung, Hanseul Cho, and Chulhee Yun. Convergence and implicit bias of gradient descent on continual linear classification. In The Thirteenth International Conference on Learning Representations, 2025. Gilad Karpel, Edward Moroshko, Ran Levinstein, Ron Meir, Daniel Soudry, and Itay Evron. Optimal l2 regularization in high-dimensional continual linear regression. In 37th International Conference on Algorithmic Learning Theory, 2026. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwińska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. doi: 10.1073/pnas.1611835114. 15
S CHLISERMAN B UZAGLO E VRON S OUDRY
Mark Kong, William Swartworth, Halyun Jeong, Deanna Needell, and Rachel Ward. Nearly optimal bounds for cyclic forgetting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pages 6109–6119. PMLR, 2021. Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath, Sebastian Goldt, and Andrew Saxe. Maslow’s hammer in catastrophic forgetting: Node re-use vs. node activation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12455–12477. PMLR, 17–23 Jul 2022. Timothée Lesort, Oleksiy Ostapenko, Pau Rodríguez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. In Conference on Lifelong Learning Agents, pages 43–65. PMLR, 2023. Ran Levinstein, Amit Attia, Matan Schliserman, Uri Sherman, Tomer Koren, Daniel Soudry, and Itay Evron. Optimal rates in continual linear regression via increasing regularization. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025. Adrian S Lewis and Jérôme Malick. Alternating projections on manifolds. Mathematics of Operations Research, 33(1):216–234, 2008. Adrian S Lewis, D Russell Luke, and Jérôme Malick. Local linear convergence for alternating and averaged nonconvex projections. Foundations of Computational Mathematics, 9(4):485–513, 2009. Boqi Li, Youjun Wang, and Weiwei Liu. Towards understanding catastrophic forgetting in two-layer convolutional neural networks. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 36057–36095. PMLR, 13–19 Jul 2025. Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of regularization-based continual learning. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup, editors, Proceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 of Proceedings of Machine Learning Research, pages 513–533. PMLR, 22–25 Aug 2023. Ziyan Li and Naoki Hiratani. Optimal task order for continual learning of multiple tasks. In Forty-second International Conference on Machine Learning, 2025. Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 21078–21100. PMLR, 23–29 Jul 2023. 16
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Ekdeep Singh Lubana, Puja Trivedi, Danai Koutra, and Robert P. Dick. How do quadratic regularizers prevent catastrophic forgetting: The role of interpolation. In ICML Workshop on Theory and Foundations of Continual Learning, 2021. Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890, 2019. Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. ICML, 2022. Angelia Nedić. Random projection algorithms for convex set intersection problems. In 49th IEEE Conference on Decision and Control (CDC), pages 7655–7660. IEEE, 2010. Binghui Peng and Andrej Risteski. Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, 2023. Jingyang Qiao, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, and Yuan Xie. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2024. Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations, 2020. Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123–146, 1995. R Tyrrell Rockafellar and Roger JB Wets. Variational analysis. Springer, 1998. Saharon Rosset, Ji Zhu, and Trevor J Hastie. Margin maximizing loss functions. In Advances in neural information processing systems, pages 1237–1244, 2004. Haozhe Shan, Qianyi Li, and Haim Sompolinsky. Order parameters and phase transitions of continual learning in deep neural networks. arXiv preprint arXiv:2407.10315, 2024. James Seale Smith, Junjiao Tian, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal-free continual learning. In CVPR Workshop on Continual Learning in Computer Vision, 2023. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. Hossein Taheri, Avishek Ghosh, and Arya Mazumdar. On the theory of continual learning with gradient descent for neural networks. arXiv preprint arXiv:2510.05573, 2025. 17
S CHLISERMAN B UZAGLO E VRON S OUDRY
Matan Tsipory, Ran Levinstein, Itay Evron, Mark Kong, Deanna Needell, and Daniel Soudry. Are greedy task orderings better than random in continual linear regression? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Gido M. van de Ven, Nicholas Soures, and Dhireesha Kudithipudi. 1.09 - continual learning and catastrophic forgetting. In John Wixted, editor, Learning and Memory: A Comprehensive Reference (Third Edition), pages 153–168. Academic Press, Oxford, third edition edition, 2025. ISBN 978-0-443-15755-4. Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018. Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, 2019. Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635–3673. PMLR, 2020. Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey. ACM Computing Surveys, 57(5):1–38, 2025. Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017. Xuyang Zhao, Huiyuan Wang, Weiran Huang, and Wei Lin. A statistical theory of regularizationbased continual learning. In Forty-first International Conference on Machine Learning, 2024.
18
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Appendix Contents A Proofs for Section 2.2
20
B Proofs for Section 3.2
23
C Proofs for Section 3.3
30
C.1 Additional Regularity Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
C.2 Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
D Proofs for Section 4
37
D.1 Proof of Theorem 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
D.2 Proof of Lemma 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
D.3 Proof of Theorem 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
19
S CHLISERMAN B UZAGLO E VRON S OUDRY
Appendix A. Proofs for Section 2.2 1 Proof of Theorem 1. For any λ ∈ (0, 1), we define cλ,r = log λ1 r . We prove in induction that for every t, every sequence λ ↓ 0 admits a subsequence (λj ) and a point Θ̄t ∈ Cτ (t) such that (λ )
Θt j −→ Θ̄t cλj ,r
Θ̄t ∈ argmin Θ − Θ̄t−1
for
2
.
Θ∈Cτ (t)
The base case, t = 0 holds trivially for the constant sequence Θλ0 = 0 = Θ̄0 . For the step, let (λ) be a sequence that converges to zero. We assume that there exists a subsequence (λj ) and a point Θ̄t−1 ∈ Cτ (t−1) that holds (λ )
j Θt−1 −→ Θ̄t−1 cλj ,r
Θ̄t−1 ∈ argmin Θ − Θ̄t−2
for
2
.
Θ∈Cτ (t−1)
For every λ, we define (λ)
(λ)
Θ̂t
≜
Θt , cλ,r
(λ)
Gt (Θ) ≜
Lτ (t) (cλ,r Θ) (λ) 2 + Θ − Θ̂t−1 . 2 cλ,r λ
Then, for every λ > 0, (λ)
(λ)
∈ arg min Lτ (t) (Θ) + λ∥Θ − Θt−1 ∥2 1 (λ) (λ) ⇐⇒ Θt ∈ arg min Lτ (t) (Θ) + ∥Θ − Θt−1 ∥2 λ 1 (λ) (λ) ⇐⇒ Θ̂t ∈ arg min Lτ (t) (cλ,r Θ) + ∥cλ,r Θ − Θt−1 ∥2 λ 1 (λ) (λ) ⇐⇒ Θ̂t ∈ arg min 2 Lτ (t) (cλ,r Θ) + ∥Θ − Θ̂t−1 ∥2 λcλ,r
Θt
(λ)
⇐⇒ Θ̂t
(λ)
∈ arg min Gt (Θ).
(λ )
Now, we look at the sequence Θ̂t j . For proving the required, it is sufficient to show a sub-sequence (λ ) 2 of Θ̂t j that converges to Θ̄t ∈ argminΘ∈Cτ (t) Θ − Θ̄t−1 . Thus, by, Lemma 3, it is sufficient to (λ )
prove that the sequence Gt j is equi-coercive and that this sequence Γ-converges to some function Gt with Θ̄t ∈ arg min Gt when j goes to ∞. Γ-convergence. For the Γ-convergence, let Gt (Θ) = 1Cτ (t) (Θ)+∥Θ− Θ̄t−1 ∥2 , where 1Cτ (t) (Θ) = 0 if Θ ∈ Cτ (t) and ∞ otherwise. • For the liminf property, let Θ̄ and a sequence Θ̂(λj ) → Θ̄. We need to prove that (λ )
Gt (Θ̄) ≤ lim inf Gt j (Θ̂(λj ) ). j→∞
(λ )
For the second term of the function, by the induction hypothesis and continuity of norm, Θ̂t j − (λ ) 2
j Θ̂t−1
→ ∥Θ̄ − Θ̄t−1 ∥2 . For the first term of the function, since Lτ (t) ≥ 0, lim inf j→∞
Lτ (t) (cλj ,r Θ̂(λj ) ) ≥ 0. λj c2λj ,r 20
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Combining both together we get that if Θ̄ ∈ Cτ (t) then (λ )
(λ ) 2
j Gt (Θ̄) = 1Cτ (t) (Θ̄) + ∥Θ̄ − Θ̄t−1 ∥2 = ∥Θ̄ − Θ̄t−1 ∥2 = lim inf Θ̂t j − Θ̂t−1
j→∞
≤ lim inf j→∞
Θ̂(λj ) )
Lτ (t) (cλj ,r λj c2λj ,r
(λ ) 2
j + Θ̂(λj ) − Θ̂t−1
(λ )
= lim inf Gt j (Θ̂(λj ) ) . j→∞
(τ (t))
(τ (t))
/ Cτ (t) , let uλ,i := crλ,r yi If Θ̄ ∈ f (xi ; Θ̂(λ) ) and consider the index realizing mini uλ,i . Since Θ̄ ∈ / Cτ (t) , we know that this index i satisfies uλ,i < crλ,r . First consider the case where uλj ,i ≤ 0 infinitely often in the sequence. Then by using the property that if u ≤ 0 then log(1 + e−u ) ≥ log 2, it holds for this sub-sequence that Lτ (t) (cλj ,a Θ̂(λj ) ) ≥ (log 2)/nτ (t) , and therefore, as for Θ̄ ∈ / Cτ (t) we have that Gt (Θ̄) = ∞, we get that (λ )
lim inf Gt j (Θ̂(λj ) ) ≥ Lτ (t) (cλ,r Θ̂(λj ) )/(λj c2λj ,r ) → ∞ = Gt (Θ̄) . j→∞
For the case where it does not hold that uλj ,i ≤ 0 infinitely often in the sequence, there exists δ ∈ (0, 1) s.t. 0 < uλj ,i ≤ crλj ,r (1 − δ). As a result, since e Lτ (t) (cλj ,r Θ̂(λj ) ) ≥
−crλ ,r j
= e− log(1/λj ) = λj , we get
1 1 1 −cr (1−δ) −u e λj ,i ≥ e λj ,r ≥ λ1−δ 2nτ (t) 2nτ (t) 2nτ (t) j
−δ Lτ (t) (cλj ,r Θ̂(λj ) ) 1 λj ≥ → ∞. =⇒ 2nτ (t) c2λj ,r (λc2λj ,r )
• For the Limsup, let Θ̄. If Θ̄ ∈ Cτ (t) , let Θ̂(λj ) = Θ̄ for every j (constant sequence). Then, for (τ (t))
every i, yi
(τ (t))
f (xi
; Θ̄) ≥ 1, and,
Lτ (t) (cλj ,r Θ̄) = ≤
1 nτ (t) 1 nτ (t)
X
(m)
log 1 + e
− crλ ,r yi j
(m)
f (xi
;Θ)
i
X − cr y(m) f (x(m) ;Θ) − cr i ≤ e λj ,r = λj . e λj ,r i i
Hence Lτ (t) (cλj ,r Θ̄)/(λj c2λj ,r ) ≤ 1/c2λj ,r → 0 as j → ∞. Then, by the induction hypothesis (λ )
and continuity of norm, Gt j (Θ̄) → ∥Θ̄ − Θ̄t−1 ∥2 = Gt (Θ̄). Otherwise, Θ̄ ∈ / Cτ (t) . Then Gt (Θ̄) = +∞, so the Γ-limsup property is trivial. Boundedness of level sets.
By the positivity of Lτ (t) , for all Θ̄, (λ )
Gt j (Θ̄) ≥
(λ ) 2
j Θ̄ − Θ̂t−1
21
.
S CHLISERMAN B UZAGLO E VRON S OUDRY
(λ )
(λ )
j Thus every sublevel set {Θ̄ : Gt j (Θ̄) ≤ C} is contained in the closed ball centered at Θ̂t−1 of √ radius C. These balls are compact sets and the equi-coercivity follows.
(λj )
By Lemma 3, there is a subsequence Θ̂t ℓ → Θ̄t . For getting the convergence in direction, τ (t) (τ (t)) note that Θ̄t ∈ Πτ (t) (Θ̄t−1 ) (as a minimizer of Gt ), which implies yi f (xi ; Θ̄t ) ≥ 1 for all i. Therefore, Θ̄t ̸= 0 (since by homogeneity f (·; 0) = 0). Since the limit is nonzero, the mapping u 7→ u/∥u∥ is continuous at Θ̄t . As a result, (λjℓ )
(λjℓ )
Θt
(λj ) Θt ℓ
=
cλjℓ Θ̂t
(λj ) cλjℓ Θ̂t ℓ
(λjℓ )
=
This concludes the proof.
22
Θ̂t
(λj ) Θ̂t ℓ
j→∞
−−−−−−−→
Θ̄t . Θ̄t
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Appendix B. Proofs for Section 3.2 In this section, we prove the lemmas used for the proof of Theorem 5. The proof is based on an analysis of the dynamics of Algorithm 2 in the construction given in Theorem 5. We begin with several lemmas. Proposition 1 (Projection onto C1 from the origin) Let a1 , b > 0 with b > a1 . Consider min x2 + y 2
x,y∈R
s.t.
a1 x2 + by 2 ≥ 1.
Then every minimizer satisfies x = 0 and y = ±b−1/2 . In particular, the minimum value is 1/b. Proof. Let (x, y) be feasible. First we show that any minimizer must satisfy a1 x2 + by 2 = 1. Assume in contradiction that r = a1 x2 + by 2 > 1 and set α := √1r ∈ (0, 1). Then (αx, αy) satisfies the constraint with equality: a1 (αx)2 + b(αy)2 = α2 (a1 x2 + by 2 ) = 1, and the objective strictly decreases: (αx)2 + (αy)2 = α2 (x2 + y 2 ) < x2 + y 2 , implying that r = 1. Second,on the boundary a1 x2 + by 2 = 1 we have y2 =
1 − a 1 x2 , b
with 0 ≤ a1 x2 ≤ 1 ⇐⇒ 0 ≤ x2 ≤
1 . a1
Substituting into the objective gives x2 + y 2 = x2 +
1 a1 1 − a 1 x2 = + x2 1 − . b b b
Since b > a1 , we have 1 − ab1 > 0, so the right-hand side is strictly increasing in x2 . Therefore it is minimized when x2 is minimal, i.e. when x2 = 0. With this choice of x = 0, the active constraint implies by 2 = 1, hence y = ±b−1/2 . This proves the claim. Proposition 2 (Projection onto C2 from the y–axis) Let a2 , b, c > 0 with c > a2 and let u = (0, b−1/2 , 0). Consider the optimization problem min
(x,y,z)∈R3
F (x, y, z) := x2 + (y − b−1/2 )2 + z 2
s.t.
a2 x2 − cy 2 + cz 2 ≥ 1.
Then every minimizer satisfies x = 0,
1 y=± √ , 2 b 23
r z=±
1 1 + . 4b c
S CHLISERMAN B UZAGLO E VRON S OUDRY
In particular, the set of minimizers is ( B :=
!) r 1 1 1 , 0, ± √ , ± + 4b c 2 b
and every Θ(2) = (0, y0 , z0 ) ∈ B satisfies 1 z02 − y02 = . c Proof. First, we show that for any solution of the problem, the constraint is active. Let (x, y, z) be feasible with a2 x2 − cy 2 + cz 2 > 1. Scaling (x, y, z) 7→ α(x, y, z) with 1 α := p ∈ (0, 1) a2 x2 − cy 2 + cz 2 preserves feasibility with equality and strictly decreases F since F (αΘ̄) = α2 Θ̄ for every Θ̄. Thus every minimizer satisfies a2 x2 − cy 2 + cz 2 = 1. Next, from feasibility we have cz 2 ≥ 1 + cy 2 − a2 x2 . Substituting into F yields F (x, y, z) ≥ x2 + (y − b−1/2 )2 +
a2 2 1 1 + cy 2 − a2 x2 = 1− x + (y − b−1/2 )2 + y 2 + . c c c
Since function (y − b−1/2 )2 + y 2 is strictly convex and is minimized at y = 12 b−1/2 , where it takes 1 value 2b . Hence every feasible point satisfies a2 2 1 1 F (x, y, z) ≥ 1 − x + + . c 2b c Equality is achieved when x = 0, y = 12 b−1/2 , and cz 2 = 1 + cy 2
⇐⇒
z2 =
1 1 + . 4b c
This yields the claimed minimizers. The final identity z02 − y02 = 1c follows by direct substitution. n o Proposition 3 (Projections onto C1 ) Let C1 := (x, y, z) ∈ R3 : a1 x2 + by 2 − bz 2 ≥ 1 , C2 := n o (x, y, z) ∈ R3 : a2 x2 − cy 2 + cz 2 ≥ 1 for a1 := 19 + ε, b := 1 − ε, a2 := 19 − ε, c := 1 + ε. Fix Θ̄t = (0, y0 , z0 ) ∈ C2 and consider the projection problem min
(x,y,z)∈R3
F (x, y, z) := x2 + (y − y0 )2 + (z − z0 )2
s.t.
Then any global minimizer (x⋆ , y ⋆ , z ⋆ ) satisfies x⋆ = 0. 24
g(x, y, z) := a1 x2 + by 2 − bz 2 − 1 ≥ 0.
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
/ C1 and C1 is closed, a Proof. The objective F and the constraint function g are C ∞ . Since u ∈ minimizer exists and is not equal to u. Then, every minimizer satisfies g(x⋆ , y ⋆ , z ⋆ ) = 0. On g = 0, ∇g(x, y, z) = (2a1 x, 2by, −2bz) ̸= 0 (otherwise x = y = z = 0 would contradict g(0, 0, 0) = −1). So LICQ holds and KKT is necessary. Next, we calculate the KKT points. We use the Lagrangian. L(x, y, z, λ) := F (x, y, z) − λg(x, y, z),
λ ≥ 0.
Stationarity gives ∂x L = 2x − 2λa1 x = 0
⇐⇒
x(1 − λa1 ) = 0,
∂y L = 2(y − y0 ) − 2λby = 0
⇐⇒
y(1 − λb) = y0 ,
∂z L = 2(z − z0 ) + 2λbz = 0
⇐⇒
z(1 + λb) = z0 .
Primal feasibility and complementary slackness imply g(x⋆ , y ⋆ , z ⋆ ) = 0, thus, since x⋆ (1−λa1 ) = 0, either x⋆ = 0 or λ = 1/a1 . y0 z0 , z = 1+λb , and g = 0 becomes Branch I corresponds to x = 0. In this case, y = 1−λb 2 2 b(y − z ) = 1. y0 z0 Branch II corresponds to λ = 1/a1 . In this case, y = 1−b/a , z = 1+b/a , and g = 0 yields 1 1 2
2
x2 = 1−b(ya1−z ) . Finally, it remains to show that any global minimizer will be in the first branch (x⋆ = 0). Branch II yields an objective value of ∥(x, y, z) − Θ̄t ∥2 ≥
1 , a1
Since a1 = 19 + ε ≤ 19 + 0.01 < 0.13,
1 > 7.6. a1 In addition, branch I contains feasible point with small objective. Consider the feasible point with x = 0, z = z0 , and r 1 y = sign(y0 ) z02 + , b which satisfies by 2 − bz 2 = 1 (hence lies on the boundary of C1 ). Then q 2 ∥(x, y, z) − Θ̄t ∥ = ( z02 + 1/b − |y0 |)2 . Using z02 ≥ y02 + 1/c and 1/b ≤ 1/c + 2ε, one gets q q p √ z02 + 1/b − |y0 | ≤ y02 + 1/c + 1/b − |y0 | ≤ 2/b ≤ 2.02, hence ∥(x, y, z) − Θ̄t ∥2 ≤ 2.02. Therefore Branch I attains objective ≤ 2.02, whereas Branch II is ≥ 1/a1 > 7.6. So every global minimizer lies in Branch I, and in particular has x = 0.
25
S CHLISERMAN B UZAGLO E VRON S OUDRY
n o Proposition 4 (Projections onto C2 ) Let C1 := (x, y, z) ∈ R3 : a1 x2 + by 2 − bz 2 ≥ 1 , C2 := n o 3 2 2 2 (x, y, z) ∈ R : a2 x − cy + cz ≥ 1 for a1 := 19 + ε, b := 1 − ε, a2 := 19 − ε, c := 1 + ε. Fix Θ̄t = (0, y0 , z0 ) ∈ C2 and consider the projection problem onto C2 : min
(x,y,z)∈R3
F (x, y, z) := x2 + (y − y0 )2 + (z − z0 )2
s.t.
g(x, y, z) := a2 x2 − cy 2 + cz 2 − 1 ≥ 0.
Then any global minimizer (x⋆ , y ⋆ , z ⋆ ) satisfies x⋆ = 0. Proof. The objective F and the constraint function g are C ∞ . Since Θ̄t ∈ C1 , we have by02 −bz02 ≥ 1, which implies y02 > z02 . However, points in C2 with x = 0 satisfy cz 2 − cy 2 ≥ 1, implying z 2 > y 2 . Thus Θ̄t ∈ / C2 . Since C2 is closed, a minimizer exists. Then, every minimizer satisfies ⋆ ⋆ ⋆ g(x , y , z ) = 0. On g = 0, ∇g(x, y, z) = (2a2 x, −2cy, 2cz) ̸= 0 (otherwise x = y = z = 0 would contradict g(0, 0, 0) = −1). So LICQ holds and KKT is necessary. Next, we calculate the KKT points. We use the Lagrangian. L(x, y, z, λ) := F (x, y, z) − λg(x, y, z),
λ ≥ 0.
Stationarity gives ∂x L = 2x − 2λa2 x = 0
⇐⇒
x(1 − λa2 ) = 0,
∂y L = 2(y − y0 ) + 2λcy = 0
⇐⇒
y(1 + λc) = y0 ,
∂z L = 2(z − z0 ) − 2λcz = 0
⇐⇒
z(1 − λc) = z0 .
Primal feasibility and complementary slackness imply g(x⋆ , y ⋆ , z ⋆ ) = 0, thus, since x⋆ (1−λa2 ) = 0, either x⋆ = 0 or λ = 1/a2 . y0 z0 Branch I corresponds to x = 0. In this case, y = 1+λc , z = 1−λc , and g = 0 becomes 2 2 c(z − y ) = 1. y0 z0 Branch II corresponds to λ = 1/a2 . In this case, y = 1+c/a , z = 1−c/a , and g = 0 yields 2 2 2
2
x2 = 1−c(za2−y ) . Finally, it remains to show that any global minimizer will be in the first branch (x⋆ = 0). Branch II yields an objective value of 1 ∥(x, y, z) − Θ̄t ∥2 ≥ . a2 Since a2 = 19 − ε,
1 > 9. a2
In addition, branch I contains feasible point with small objective. Consider the feasible point with x = 0, y = y0 , and r 1 z = sign(z0 ) y02 + , c which satisfies cz 2 − cy 2 = 1 (hence lies on the boundary of C2 ). Then q ∥(x, y, z) − Θ̄t ∥2 = ( y02 + 1/c − |z0 |)2 . 26
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Using y02 ≥ z02 + 1/b (from Θ̄t ∈ C1 ), the term y02 = 1/b. Thus q y02 + 1/c − |z0 | ≤
p y02 + 1/c − |z0 | is maximized when z0 = 0 and p √ 1/b + 1/c ≤ 2.02,
hence ∥(x, y, z) − Θ̄t ∥2 ≤ 2.02. Therefore Branch I attains objective ≤ 2.02, whereas Branch II is ≥ 1/a2 > 9. So every global minimizer lies in Branch I, and in particular has x = 0. Now given the technical lemmas above we can finally prove Theorem 5. Proof of Theorem 5. First for Part (a), let a1 := 19 + ε, b := 1 − ε, a2 := 19 − ε, c := 1 + ε. In this proof we refer to Θ̄ = (u, v) as a vector in R4 where its first and third entries are u and the two other entries are v. It holds that C1 = {Θ̄ ∈ R4 | a1 Θ̄[1]2 − a1 Θ̄[2]2 + bΘ̄[3]2 − bΘ̄[4]2 ≥ 1}, C2 = {Θ̄ ∈ R4 | a2 Θ̄[1]2 − a2 Θ̄[2]2 − cΘ̄[3]2 + cΘ̄24 ≥ 1}. Then, since ε < 0.01, Θ̄ = (4, 0, 0, 0) ∈ C1 ∩ C2 . For part (b), we first show that that for any sequence of projections, for every iteration t, Θ̄t [2] = 0 by induction of (t). The basis of t = 0 follows by initialization in the origin. For the step assume that Θ̄t [2] and assume in contradiction that Θ̄t+1 [2] = δ ̸= 0. Let Θ̃ = Θ̄t+1 [2] − δe2 . It holds that Θ̃2 = 0, and for every m ∈ 1, 2 f (Θ̃, xm ) ≥ Θ̄t+1 ≥ 1, and, ∥Θ̃ − Θ̄t ∥ < ∥Θ̄t − Θ̄t+1 ∥, in a contradiction to the fact that Θ̄t+1 is the projection of Θ̄t+1 on Cτt . Since Θ̄2 remains zero during all iterations, we identify each Θ̄ with a vector (x, y, z) ∈ R3 , where Θ̄t [1] = x, Θ̄t [3] = y, Θ̄t [4] = z and analyze the projection sequence in this 3-dimensional space, where n o n o C1 := (x, y, z) ∈ R3 : a1 x2 + by 2 − bz 2 ≥ 1 , C2 := (x, y, z) ∈ R3 : a2 x2 − cy 2 + cz 2 ≥ 1 . This will not change the dynamics of the projections. For simplicity, we assume that τ1 = 1, the case where τ1 = 2 is analogous. Under this choice, we prove that d Θ̄t , C ⋆ ≥ 2.99. For the first step, we minimize x2 + y 2 + z 2
s.t.
a1 x2 + by 2 − bz 2 ≥ 1.
At any minimizer one must have z = 0: setting z = 0 strictly decreases the objective and increases feasibility. Thus we reduce to min x2 + y 2 x,y
s.t. 27
a1 x2 + by 2 ≥ 1.
S CHLISERMAN B UZAGLO E VRON S OUDRY
Since b > a1 > 0, by Proposition 1, the minimum is attained at x = 0 and y 2 = 1/b, giving Θ̄1 ∈ A := {(0, ±b−1/2 , 0)}. For the next projection, which is onto C2 , by symmetry it suffices to project Θ̄ = (0, b−1/2 , 0) (the second case is analogous). We minimize F (x, y, z) = x2 + (y − b−1/2 )2 + z 2
s.t.
a2 x2 − cy 2 + cz 2 ≥ 1.
By Proposition 2, it follows that ( Θ̄2 ∈ B :=
!) r 1 1 1 . 0, ± √ , ± + 4b c 2 b
In particular, every Θ̄2 = (0, y0 , z0 ) ∈ B satisfies z02 − y02 =
1 c
=⇒
1 z02 ≥ y02 + . c
We continue that for all t ≥ 2, Θ̄t [1] = 0 by induction in t. Assume Θ̄t = (0, y0 , z0 ) If Θ̄t ∈ C2 and the projection is onto C1 , then, a2 · 02 − cy02 + cz02 ≥ 1, i.e. z02 ≥ y02 + 1c . Consider projecting Θ̄t onto C1 , i.e. min ∥(x, y, z) − Θ̄t ∥2 s.t. a1 x2 + by 2 − bz 2 ≥ 1. (x,y,z)
In Proposition 3, we show that any minimizer of this optimization problem satisfies x = 0. The case where Θ̄t ∈ C1 and the projection is onto C2 , is analogous and is proved in Proposition 4. As a result, we proved that for every t, Θ̄t [1] = 0. It is left to bound from below the distance between Θ̄t and the intersection C ⋆ . Let Θ̄ ∈ C ⋆ . Multiply the C1 constraint by c and the C2 constraint by b and add: c(a1 Θ̄[1]2 − a1 Θ̄[2]2 + bΘ̄[3]2 − bΘ̄[4]2 ) + b(a2 Θ̄[1]2 − a2 Θ̄[2]2 − cΘ̄[3]2 + cΘ̄[4]2 ) ≥ c + b. This gives, 2 ( + 2ε2 )Θ̄[1]2 = (ca1 + ba2 )Θ̄[1]2 ≥ (ca1 + ba2 )Θ̄[1]2 − (ca1 + ba2 )Θ̄[2]2 ≥ b + c = 2. 9 Thus, Θ̄[1]2 ≥
2
1
= 1 . 2 2 2 9 + 2ε 9 +ε
But for all t ≥ 0 we proved Θ̄t [1] = 0, so for any Θ̄ ∈ C ⋆ , ∥Θ̄t − Θ̄∥2 ≥ Θ̄[1]2 ≥ Therefore
s ⋆
d(Θ̄t , C ) ≥
1
1
. 1 2 9 +ε
>2 1 2 9 +ε 28
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
as claimed. For the forgetting, we notice that for every t, there exists m ∈ 1, 2 such that Θ̄t ∈ Cm . If m = 1, since Θ̄t is always on the boundary, it holds that 1 = y1 f (x1 ; Θ̄t ) = ⟨u2 , x1 ⟩ − ⟨v 2 , x1 ⟩ = (1 − ε)u[2]2 − (1 − ε)v[2]2 . 1 This implies, v[2]2 = − 1−ε + u[2]2 . Then, for the other task, m = 2, it holds that,
y2 f (x2 ; Θ̄t ) = ⟨u2 , x2 ⟩ − ⟨v 2 , x2 ⟩ = (−1 − ε)u[2]2 − (−1 − ε)v[2]2 1 + u[2]2 ) = (−1 − ε)u[2]2 + (1 + ε)(− 1−ε 1+ε =− 1−ε ≤ −1, and, 1 − y2 f (x2 ; Θ̄t ) ≥ 2. This implies, F2 (Θ̄t ) ≥ (1 − y2 f (x2 ; Θ̄t )) ≥ 2. If m = 2, similarly, it holds that 1 = y2 f (x2 ; Θ̄t ) = ⟨u2 , x2 ⟩ − ⟨v 2 , x2 ⟩ = (−1 − ε)u[2]2 − (−1 − ε)v[2]2 . 1 This implies, u[2]2 = − 1+ε + v[2]2 . Then, for the other task, m = 1, it holds that,
y1 f (x1 ; Θ̄t ) = ⟨u2 , x1 ⟩ − ⟨v 2 , x1 ⟩ = (1 − ε)u[2]2 − (1 − ε)v[2]2 = −(1 − ε)v[2]2 + (1 − ε)(−
1 + v[2]2 ) 1+ε
1−ε 1+ε ≤ −0.9, =−
and, 1 − y1 f (x1 ; Θ̄t ) ≥ 1.9. then, F1 (Θ̄t ) ≥ (1 − y1 f (x1 ; Θ̄t )) ≥ 1.5.
29
S CHLISERMAN B UZAGLO E VRON S OUDRY
Appendix C. Proofs for Section 3.3 In this section, we prove the lemmas used for the proof of Theorem 6. C.1. Additional Regularity Conditions We begin with defining more regularity condition that will be used to prove the required conditions and proving that several conditions imply other conditions. We begin with the following definition of MFCQ condition. Definition 1 (MFCQ condition) (e.g. Lyu and Li (2019)] Given m functions hj : Rn → R and a Set B = {hj ≥ 0, ∀j ∈ [m]} ⊆ Rn , we say the system defining set B satisfies the MFCQ at a feasible point x̄ ∈ B if there exists a vector d ∈ Rn such that for all active constraints j ∈ IB (x) = {j | hj (x̄) = 0}, ⟨∇hj (x̄), d⟩ > 0. In addition, if for all active constraints, ⟨∇hj (x̄), d⟩ > γ > 0, we say that B satisfies MFCQ with margin γ. Then, we define another regularity condition, named metric regularity and show that it is implied by MFCQ (Definition 1). Definition 2 (Metric regularity, Example 9.44 of Rockafellar and Wets (1998)) Let F : Rn → Rm be continuously differentiable, let D ⊆ Rm be a closed convex set, and let x̄ ∈ F −1 (D). In addition, ND (F (x̄)) = {v ∈ Rn | ⟨v, y − F (x̄)⟩ ≤ 0, ∀y ∈ D} . The constraint system {x | F (x) ∈ D} is said to be metrically regular at x̄ if h i ⊤ λ ∈ ND (F (x̄)) and ∇F (x̄) λ = 0 =⇒ λ = 0.
(13)
Lemma C.3 (MFCQ with margin implies Metric Regularity) Let C := {x ∈ Rn | h(x) ≥ 0}, where h : Rn → Rk is a smooth vector-valued function h(x) = (h1 (x), . . . , hk (x)). Assume that C satisfies the MFCQ with margin γ. Then, it satisfies also metric regularity. Proof. Let D be the k-dimensional positive orthant. Suppose λ satisfies the condition in (13) for D and h. Let d the vector from Definition 1. Then, λ[j] ≤ 0 for all j and X 0 = d⊤ ∇h(x̄)⊤ λ = λ[j]∇hj (x̄)⊤ d. j∈[k]
Using the MFCQ condition (∇hj (x̄)⊤ d ≥ γ) and since λ[j] ≤ 0 (by the fact that λ ∈ ND (h(x̄)), it holds that X 0≤γ λ[j] ≤ 0. j∈[k]
Then, it follows that λ = 0.
30
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Now, we prove that MFCQ and metric regularity of each set implies κ-linear regularity with respect to the intersection of the sets. For the proof we use the following claim from Rockafellar and Wets (1998). Proposition 4 (Property of metric regularity) (Example 9.44 in Rockafellar and Wets (1998)) Let F : Rn → Rm be a β smooth mapping and let D ⊆ Rm be a closed set. Define the constraint set C = F −1 (D) = {x | F (x) ∈ D}. Let x̄ ∈ C. If the constraint set satisfies metric regularity at x̄ and MFCQ with margin γ at x̄. Then, for µ = maxy∈ND (F (x̄)),∥y∥=1 ∇F 1(x̄)y , for any x near x̄ in holds that d(x, C) ≤ µd(F (x), D) Lemma C.5 (Metric regularity implies Linear Regularity) Let M . For each m ∈ M , let the set Cm be defined by a system of inequalities, Cm := {x ∈ Rn | hm (x) ≥ 0}, where hm : Rn → Rnm is a smooth vector-valued function hm (x) = (hm,1 (x), . . . , hm,nm (x)). Define the global feasible set T ⋆ as the intersection: C := m∈M Cm . Fix x̄ ∈ C ⋆ . If every hm,j is G-Lipschitz and every Cm satisfies metric regularity at x̄. Then the collection {Cm }m∈M is κ linearly regular at x̄ with κ = G∥v∥ γ . Proof. For any unit vector λ ∈ ND (hm (x̄)), where D is the nm -dimensional positive orthant, we have λ[j] ≤ 0. Then, by the fact that ∥u∥ = supw̸=0 ⟨v,w⟩ ∥w∥ , and MFCQ with margin γ and vector v, | |⟨v, ∇hm (x̄)⊤ λ⟩| ∥∇hm (x̄) λ∥ ≥ = ∥v∥ ⊤
P
j λ[j]⟨∇hm,j (x̄), v⟩|
∥v∥
≥
γ
P
|λ[j]| γ ≥ . ∥v∥ ∥v∥
∥v∥ 1 Thus, by Proposition 4, µ = sup ∥∇hm (x̄) ⊤ λ∥ ≤ γ . Now let h be the function which is components
are all of the constraints {hm,i }. Than, since C ⋆ = h−1 (D), by Proposition 4, it holds that d(x, C ⋆ ) ≤ µd(h(x), D) = µ max max{0, −hm,j (x)} = (m,j)
∥v∥ max max{0, −ht,j (x)}. γ (m,j)
(14)
Now, it holds that,
max max{0, −hm,j (x)} = max
m∈M
(m,j)
max max{0, −hm,j (x)} ,
j=1...nm
thus, for am (x) := maxj max{0, −hm,j (x)}, we get that, d(x, C ⋆ ) ≤
∥v∥ max am (x). γ m∈M
Now, Let m ∈ M and x. If x ∈ Cm , then am (x) = 0. Otherwise, let p = ΠCm (x). By definition, p ∈ Cm , so hm,j (p) ≥ 0 for all j = 1 . . . nm . For any specific component j: −hm,j (x) ≤ hm,j (p) − hm,j (x) ≤ G∥p − x∥ = Gd(x, Cm ). Since this holds for every j, it holds for the maximum: am (x) = max max{0, −hm,j (x)} ≤ Gd(x, Cm ). j
31
S CHLISERMAN B UZAGLO E VRON S OUDRY
Combining all together, we get that, d(x, C ⋆ ) ≤
∥v∥ G∥v∥ max (Gd(x, Cm )) = max d(x, Cm ). m γ m γ
Now we prove that metric regularity implies (ε, δ) regularity. In fact, we prove that for any ε > 0 there exists a δ > 0 such that (ε, δ) regularity is implied. In the proof, we use the following claim from Rockafellar and Wets (1998). Proposition 6 (Theorem 6.31 in Rockafellar and Wets (1998)) Let F : Rn → Rm be a smooth vector valued mapping and let D ⊆ Rm be a closed set. Define for every set A NAprox (u) = {v ∈ Rn | ∃ σ > 0 s.t. u ∈ ΠA (u + σv).} Define the constraint set C = F −1 (D) = {x | F (x) ∈ D}. Then for any x̄ ∈ C, if metric regularity holds at x̄, and D is convex, then, prox NCprox (x̄) = {∇F (x̄)⊤ λ | λ ∈ ND (F (x̄))}.
(15)
Lemma C.7 (Metric regularity implies (ε, δ)-Regularity) Let F : Rn → Rm be a smooth vector valued mapping, and let D ⊆ Rm be a closed convex set. Define the feasible set C = {x ∈ Rn | F (x) ∈ D}. Assume that metric regularity (Definition 2) and MFCQ holds at x̄ ∈ C with margin γ. εr Then, for every ε > 0, C is (ε, β∥x̄∥ -regular at x̄. Proof. Let ε > 0 be given. We need to find δ > 0 such that for all x, y ∈ C ∩ Bδ (x̄) and any proximal normal v ∈ NCprox (x), the inequality ⟨v, y − x⟩ ≤ ε∥v∥∥y − x∥ holds. By Proposition 6, it holds that for x near x̄, any proximal normal v ∈ NCprox (x) can be represented as: prox v = ∇F (x)⊤ λ with λ ∈ ND (F (x)). Since F is smooth vector valued mapping we have β ′ such that, ∥F (y) − F (x) − ∇F (x)(y − x)∥ ≤
β′ ∥y − x∥2 2
By Proposition 6, and using the Cauchy-Schwarz inequality on the error term, it holds that: ⟨v, y − x⟩ = ⟨∇F (x)⊤ λ, y − x⟩ = ⟨λ, ∇F (x)(y − x)⟩ = ⟨λ, F (y) − F (x) − (F (y) − F (x) − ∇F (x)(y − x))⟩ = ⟨λ, F (y) − F (x)⟩ − ⟨λ, F (y) − F (x) − ∇F (x)(y − x)⟩ ≤ ⟨λ, F (y) − F (x)⟩ +∥λ∥ ∥F (y) − F (x) − ∇F (x)(y − x)∥ | {z } ≤0
β ′ ∥λ∥ ≤ ∥y − x∥2 . 2 32
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
prox
The first term is ≤ 0 because D is convex (and thus, ND ((F (x)) = ND (F (x))), λ ∈ ND (F (x)), and F (y) ∈ D. For the second term,by MFCQ with margin γ and vector u = x̄, P P | j λ[j]⟨∇Fj (x̄), u⟩| |⟨u, ∇F (x̄)⊤ λ⟩| γ |λ[j]| γ ⊤ ∥v∥ = ∥∇F (x̄) λ∥ ≥ = ≥ ≥ ∥λ∥ . ∥u∥ ∥u∥ ∥u∥ ∥u∥ This implies, ∥λ∥ ≤
∥v∥∥u∥ . γ
Thus, combining all together, we get, ∥v∥∥u∥ ⟨v, y − x⟩ ≤ γ
β′ 2 ∥y − x∥ . 2
∥x̄∥ Thus, to satisfy the (ε, δ)-regularity condition, we denote κ = ∥u∥ γ = γ . We need to choose δ such that for every x, y ∈ Bδ (x̄)
κβ ′ ∥v∥∥y − x∥2 ≤ ε∥v∥∥y − x∥. 2 Dividing by ∥v∥∥y − x∥ (assuming nonzero, otherwise the inequality holds trivially), this requires κβ ′ 2 ∥y − x∥ ≤ ε. Since x, y ∈ Bδ (x̄), we have ∥y − x∥ ≤ 2δ. Thus, it suffices to choose δ such that: κβ ′ ε (2δ) ≤ ε =⇒ δ ≤ . 2 κβ ′ With this δ, the set is (ε, δ)-regular.
C.2. Proof of Theorem 6 In this section we prove Theorem 6. We begin with the proof of Proposition 12. Proof of Proposition 12. The statement for cyclic ordering follows directly from Corollary 5.10 in Dao and Phan (2019). Here we prove the statement for random ordering. Fix xt−1 ∈ Bδ/2 (Θ⋆ ) and let x̄ ∈ ΠC ⋆ (xt−1 ). Since Θ⋆ ∈ C ⋆ , we have ∥xt−1 − x̄∥ = d(xt−1 , C ⋆ ) ≤ d(xt−1 , Θ⋆ ) ≤ δ/2 . By the triangle inequality, d(x̄, Θ⋆ ) ≤ ∥xt−1 − x̄∥ + d(xt−1 , Θ⋆ ) ≤
δ δ + = δ. 2 2
Thus, both xt−1 and x̄ lie in Bδ (Θ⋆ ), ensuring the local regularity assumptions apply. Let τ (t) ∈ {1, . . . , M } be arbitrary and let xt ∈ ΠCτ (t) (xt−1 ). By (ε, δ)–regularity and Proposition 3.5 of Dao and Phan (2019) (with λ = 1), the exact projector ΠCτ (t) is (Ωτ (t) , γ, β)– quasi firmly Fejér monotone on Bδ/2 (Θ⋆ ) with γ=
1 , 1−ε
Ωτ (t) ≜ Cτ (t) ∩ Bδ (Θ⋆ ) .
β = 1, 33
S CHLISERMAN B UZAGLO E VRON S OUDRY
Since x̄ ∈ C ⋆ ⊆ Cτ (t) and x̄ ∈ Bδ (Θ⋆ ), we have x̄ ∈ Ωτ (t) , and hence the quasi firm Fejér inequality yields: ∥xt − x̄∥2 + ∥xt−1 − xt ∥2 ≤ γ∥xt−1 − x̄∥2 = γd2 (xt−1 , C ⋆ ) . Using d(xt , C ⋆ ) ≤ ∥xt − x̄∥ and ∥xt−1 − xt ∥ = d(xt−1 , Cτ (t) ) (exact projection), we obtain d2 (xt , C ⋆ ) ≤ γd2 (xt−1 , C ⋆ ) − d2 (xt−1 , Cτ (t) ) . Taking expectation with respect to τ (t) yields Eτ (t) [d2 (xt , C ⋆ )] ≤ γd2 (xt−1 , C ⋆ ) − Eτ (t) [d2 (xt−1 , Cτ (t) )] . By linear regularity, max d(xt−1 , Cm ) ≥
1≤m≤M
1 d(xt−1 , C ⋆ ) , κ
and therefore M
1 2 1 X 2 d (xt−1 , Cm ) ≥ d (xt−1 , C ⋆ ) . Eτ (t) [d (xt−1 , Cτ (t) )] = M M κ2 2
m=1
Overall, we showed, Eτ (t) [d (xt , C )] ≤ γ − 2
⋆
1 M κ2
d2 (xt−1 , C ⋆ ) .
1 Substituting γ = 1−ε completes the proof.
Lemma C.8 Let f (x; Θ) and C1 , . . . CM ; C ⋆ as in Section 2. Let δ and Θ be such that d(Θ, C ⋆ ) ≤ δ. Let Θ⋆ = ΠC ⋆ (Θ), and assume that f (x, ·) is G-Lipschitz in Bδ (Θ⋆ ) = {Θ ∈ Rp | ∥Θ − Θ⋆ ∥ ≤ δ}. Then, it holds that, max Fm (Θ) ≤ Gd(Θ, C ⋆ ) m
(m)
(m)
Proof. Let m ∈ [M ] and (xi , yi ). By the fact that for every such data point, it holds that (m) (m) yi f (xi ; Θ⋆ ) ≥ 1 and G-Lipschitzness, it holds that, (m)
1 − yi
(m)
f (xi
(m)
; Θ) ≤ yi
(m)
f (xi
(m)
; Θ⋆ ) − yi
(m)
f (xi
; Θ) ≤ G|y|∥Θ⋆ − Θ∥ = Gd(Θ, C ⋆ )
Since this holds for any data point, it holds also for the maximal, thus, (m)
max Fm (Θ) ≜ max max{0, 1 − yi m
i,m
(m)
f (xi
; Θ)} ≤ Gd(Θ, C ⋆ ).
Now, we use the lemmas form Section C.1 to show that if for every x, f (x; Θ) is G-Lipschitz, smooth and positively homogeneous functions to show that the regularity conditions mentioned in Proposition 12 hold. We begin with the following lemma that shows that for homogeneous models with degree r, each feasible set satisfies the MFCQ condition at Θ⋆ given in Definition 1 with margin r such the vector d in this definition is Θ⋆ . 34
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Lemma C.9 (MFCQ of homogeneous models) Consider a r-positively-homogeneous model f (·; Θ̄) : X → R that is G-Lipschitz and β-smooth. Assume joint separability, i.e., nonempty intersection C ⋆ = C1 ∩ · · · ∩ CM ̸= ∅. Let Θ⋆ ∈ C ⋆ = T ⋆ ⋆ m Cm . Then, each Cm satisfies the MFCQ condition at Θ with margin r for d = Θ . In addition, also C ⋆ satisfies the MFCQ condition at Θ⋆ with margin r for d = Θ⋆ . (m)
(m)
(m)
(m)
Proof. Let m and let Im := {i : yi f (Θ⋆ , xi ) = 1} = {i : yi f (Θ⋆ , xi ) = 1}. If Im is empty, the definition holds trivially. Otherwise, since f (x; Θ) is smooth and positively homogeneous of degree r, Euler’s theorem for homogeneous functions yield, for every x, ⟨∇Θ f (x; Θ⋆ ), Θ⋆ ⟩ = (m) (m) rf (x; Θ⋆ ). Then, for every i ∈ Im , hm,i = yi f (Θ⋆ , xi ) − 1 = 0, satisfies, (m)
⟨∇hm,i (Θ⋆ ), Θ⋆ ⟩ = yi
(m)
⟨∇Θ f (Θ⋆ , xi
(m)
), Θ⋆ ⟩ = yi
(m)
rf (Θ⋆ , xi
) = r > 0.
Then, the MFCQ condition holds for d = Θ⋆ and margin r. The proof for C ⋆ is identical.
Now we turn to prove (ε, δ) regularity. Lemma C.10 ((ε, δ) regularity of homogeneous models) Consider a r-positively-homogeneous model f (·; Θ̄) : X → R that is G-Lipschitz and β-smooth. T Assume joint separability, i.e., nonempty intersection C ⋆ = C1 ∩ · · · ∩ CM ̸= ∅. Let Θ⋆ ∈ C ⋆ = t Cm . Then, for every ε > 0, there exists δ > 0 such that every set Cm is (ε, δ)-regular at Θ⋆ . Proof. Let m ∈ M By Lemma C.9 MFCQ condition holds with margin r. In addition, by Lemma C.3 metric regularity also holds. Then, by Lemma C.7, the lemma follows.
Lemma C.11 (κ-linear regularity of homogeneous models) Consider a r-positively-homogeneous model f (·; Θ̄) : X → R that is G-Lipschitz and β-smooth. Assume joint separability, i.e., nonempty intersection C ⋆ = C1 ∩ · · · ∩ CM ̸= ∅. Let Θ⋆ ∈ C ⋆ = T G∥Θ⋆ ∥ M m Cm . Then, if f is G-Lipschitz, the collection {Cm }m=1 satisfy κ-linear regularity for κ = r around Θ⋆ . Proof. Let m ∈ M By Lemma C.9, MFCQ condition holds for the vector ∥Θ⋆ ∥ with margin r. In addition, by Lemma C.3 metric regularity also holds. Then, by Lemma C.5, the lemma follows.
Now we can turn to the proof of Theorem 6. 1 Proof of Theorem 6. For the cyclic ordering, let ε be such that (1−ε)1M −1 = 1 + 2(M −1)κ 2 . Then
by Lemma C.10 each set Cm is (ε, δ) regular. In addition, by Lemma C.11, the collection {Cm }M m=1 ⋆∥ ⋆ . As a result, the conditions of Proposition 12 satisfy κ-linear regularity for κ = G∥Θ in a around Θ r 35
S CHLISERMAN B UZAGLO E VRON S OUDRY
holds in a δ-neighborhood of Θ⋆ . By the choice of ε we get, for the cyclic order, that for any t, d(Θ̄t+M , C ⋆ ) ≤ ρ d(Θ̄t , C ⋆ ) ρ=
1 1− 2(M − 1)κ2
1/2(M −1)
1/2(M −1) 1 2(M − 1)(G∥Θ⋆ ∥/r)2 1/2(k−1) r2 = 1− 2(M − 1)G2 ∥Θ⋆ ∥2 1/2M r2 ≤ exp − 2M G2 ∥Θ⋆ ∥2 r2 = exp − , 4M 2 G2 ∥Θ⋆ ∥2 = 1−
Thus, for M | k d(Θ̄k , C ) ≤ exp − ⋆
kr2 4M G2 ∥Θ⋆ ∥2
d(Θ̄0 , C ⋆ )
1 For the random order, using Proposition 12, we get that for ε such that such that 1−ε = 1 + 2M1κ2 ,
Eτ (t) d2 (Θ̄t , C ⋆ ) ≤
1 1− 2M κ2
d2 (Θ̄t−1 , C ⋆ ).
Thus, for k = cM , k 1 1− d2 (Θ̄0 , C ⋆ ) 2M κ2 k ≤ exp − d2 (Θ̄0 , C ⋆ ) 2M κ2 kr2 = exp − d2 (Θ̄0 , C ⋆ ) 2M G2 ∥Θ⋆ ∥2
Eτ d2 (Θ̄k , C ⋆ ) ≤
By Jensen inequality, q ⋆ 2 ⋆ Eτ d (Θ̄k , C ) ≤ exp − Eτ d(Θ̄k , C ) ≤ For the forgetting, the statement follows by Lemma C.8.
36
kr2 4M G2 ∥Θ⋆ ∥2
d(Θ̄0 , C ⋆ )
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Appendix D. Proofs for Section 4 D.1. Proof of Theorem 13 (λ)
Proof of Theorem 13. As in classification, we employ the theory of Γ-convergence. Let Gt (Θ) := 1 2 2 λ Lτ (t) (Θ) + ∥Θ − Θt−1 ∥ . It holds that Θ ∈ argminΘ∈Cτ (t) Lτ (t) (Θ) + λ∥Θ − Θt−1 ∥ if and only (λ)
(λ)
if Θ ∈ argminΘ∈Cτ (t) Gt (Θ). We show that Gt
Γ-converges to the following funcation as λ ↓ 0:
( ∥Θ − Θt−1 ∥2 Gt (Θ) = +∞
if Θ ∈ Cτ (t) otherwise.
(λ)
First, for the Liminf inequality, let Θt → Θ. By the definition of the feasible set, if Θ ∈ / Cτ (t) , then Lm (Θ) > 0. Let Lm (Θ) = δ > 0. Since f is continuous, Lm is continuous. (λ) Therefore, there exists a neighborhood around Θ where Lm (·) > δ/2. Since Θt → Θ, for (λ) sufficiently small λ, Θt lies in this neighborhood. Thus: (λ)
(λ)
Gt (Θt ) =
δ 1 2 (λ) (λ) Lm (Θt ) + Θt − Θt−1 ≥ + 0. λ 2λ (λ)
(λ)
Taking the limit as λ ↓ 0, we have lim Gt (Θt ) = +∞. Since Gt (Θ) = +∞, the inequality holds. Otherwise, if Θ ∈ Cτ (t) . Since Lm (·) ≥ 0. we have: (λ)
(λ)
(λ)
Gt (Θt ) ≥ Θt
− Θt−1
(λ)
2
.
(λ)
Thus, by continuity, as Θt → Θ, the term ∥Θt − Θt−1 ∥2 → ∥Θ − Θt−1 ∥2 and (λ) (λ) lim inf λ↓0 Gt (Θt ) ≥ ∥Θ − Θt−1 ∥2 = Gt (Θ). (λ) For the Limsup property, We need to show that for any Θ, there exists a sequence Θt → Θ (λ) (λ) (λ) such that lim supλ↓0 Gt (Θt ) ≤ Gt (Θ). We choose the constant sequence Θt = Θ for all λ. If (λ)
(λ)
Θ∈ / Cτ (t) , Gt (Θ) = +∞, so the inequality lim sup Gt (Θt ) ≤ ∞ is trivially satisfied. Otherwise, (λ)
if Θ ∈ Cτ (t) , then Lm (Θ) = 0 and Gt (Θ) = Gt (Θ). (λ)
Finally for the boundedness of level sets, the function Gt (Θ) is bounded below by ∥Θ−Θt−1 ∥2 . Since the sublevel sets of this lower bound are compact, the sequence of functionals is equi-coercive. The theorem follows by Part (b) of Lemma 3 and the fact that Θt is a minimizer of Gt .
37
S CHLISERMAN B UZAGLO E VRON S OUDRY
D.2. Proof of Lemma 14 Lemma D.1 (Catastrophic Forgetting in Squared Models) Consider a squared model f (x; Θ̄) in d = 2 parameterized by Θ̄ = (u, v) where u, v ∈ R2 . Fix ε ∈ (0, 0.01) and define two tasks 1 ⊤ 1 (2) X(1) = x⊤ +ε, 1−ε , X = x = −ε, −1−ε , y(1) = y(2) = (1) . 1 = 2 9 9 Then, (a) The induced feasible sets C1 = {Θ̄ ∈ R4 | f (x1 ; Θ̄) = y1 }, C2 = {Θ̄ ∈ R4 | f (x2 ; Θ̄) = y2 } have a nonempty intersection, i.e., C ⋆ := C1 ∩ C2 ̸= ∅. (b) Algorithm 2 satisfies, for all t ≥ 0, d Θ̄t , C ⋆ ≥ 2 and maxm Fm (Θ̄t ) ≥ 3 . Proof of Lemma D.1. We now consider the dynamics where the task sets are defined by the same constraints as Theorem 5, except there is equality instead of inequality. However, we observe that the sequential projections onto these equality sets generate the exact same trajectory as the projections onto the inequality sets Cm = {Θ̄ | f (xm ; Θ̄) ≥ 1}. This follows immediately from Proposition 1, Proposition 2, and Proposition 3 in Appendix B. Specifically, those lemmas prove that for every projection step involved in the inequality-constrained dynamics, the unique minimizer satisfies the constraint with strict equality (i.e., f (xm ; Θ̄) = 1). Since the solution to the relaxed problem (≥ 1) lies on the boundary, it is necessarily the solution to the equality-constrained problem. Thus, the sequence of iterates {Θ̄t } is identical in both settings. By those lemmas we get that for every t it holds for Θ̄t = (ut , vt ) that ut = 0. Now, let Θ̄ = (u, v) ∈ C ⋆ . Multiply the C1 constraint by c and the C2 constraint by b and add: c(a1 Θ⋆ [1]2 − a1 Θ⋆ [2]2 + bΘ⋆ [3]2 − bΘ̄[4]2 ) + b(a2 Θ⋆ [1]2 − a2 Θ⋆ [2]2 − cΘ⋆ [3]2 + cΘ̄[4]2 ) = c + b = 2. This gives, 2 ( + 2ε2 )(Θ⋆ [1]2 − Θ⋆ [2]2 ) = (ca1 + ba2 )Θ⋆ [1]2 − (ca1 + ba2 )Θ⋆ [2]2 = 2. 9 In particular, 1 . 2 9 +ε
Θ⋆ [1]2 ≥ Θ⋆ [1]2 − Θ⋆ [2]2 = 1 But for all t ≥ 0 we proved Θ̄t [1] = 0, so for any Θ̄ ∈ C ⋆ , ∥Θ̄t − Θ̄∥2 ≥ Θ⋆ [1]2 ≥
1
. 1 2 9 +ε
Therefore, s ⋆
d(Θ̄t , C ) ≥
1
>2 1 2 9 +ε
as claimed. For the loss, we notice that for every t, there exists m ∈ 1, 2 such that Θ̄t ∈ Cm . 38
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
If m = 1, since Θ̄t is always on the boundary, it holds that 1 = y1 = f (x1 ; Θ̄t ) = ⟨u2t , x1 ⟩ − ⟨vt2 , x1 ⟩ = (1 − ε)ut [2]2 − (1 − ε)vt [2]2 . 1 This implies, vt [2]2 = − 1−ε + ut [2]2 . Then, for the other task, m = 2, it holds that,
f (x2 ; Θ̄t ) = (−1 − ε)ut [2]2 − (−1 − ε)vt [2]2 1 = (−1 − ε)ut [2]2 + (1 + ε)(− + ut [2]2 ) 1−ε 1+ε =− 1−ε ≤ −1 , This implies F2 (Θ̄t ) = |f (x2 ; Θ̄t ) − y2 |2 ≥ 4. If m = 2, similarly, it holds that 1 = y2 = f (x2 ; Θ̄t ) = ⟨u2 , x2 ⟩ − ⟨v 2 , x2 ⟩ = (−1 − ε)u[2]2 − (−1 − ε)v[2]2 . 1 This implies, u[2]2 = − 1+ε + v[2]2 . Then, for the other task, m = 1, it holds that,
f (x1 ; Θ̄t ) = (1 − ε)ut [2]2 − (1 − ε)vt [2]2 = −(1 − ε)vt [2]2 + (1 − ε)(− 1−ε 1+ε ≤ −0.9 , =−
and, F1 (Θ̄t ) = |f (x1 ; Θ̄t ) − y1 |2 ≥ 3.
39
1 + vt [2]2 ) 1+ε
S CHLISERMAN B UZAGLO E VRON S OUDRY
D.3. Proof of Theorem 15 In this section, we analyze Algorithm 2 for the multi-task regression problem. Each task m imposes a set of equality constraints: (m)
hi,m (Θ) = f (xi
(m)
; Θ) − yi
= 0.
Let Cm,i = {Θ ∈ Rp | hi,m (Θ) T = 0} be the feasible set for a single data point. The global solution ⋆ set is the intersection C = m,i Cm,i . We discuss the β-smooth and G-Lipschitz case. Now we turn to prove κ-linear regularity for homogeneous models with inequality constraints. Lemma D.2 (κ-Linear Regularity of Homogeneous Models) Consider a r-positively-homogeneous model f (·; Θ̄) : X → R that is G-Lipschitz and β-smooth. Assume joint separability, i.e., nonempty intersection C ⋆ = C1 ∩ · · · ∩ CM ̸= ∅. Let ymin = (m) r2 minm,i |yi | ̸= 0 and Θ⋆ ∈ C ⋆ . Let ε holding (1−ε)1M −1 = 1 + 2(M −1)G 2 ∥Θ⋆ ∥2 . Then, for δ = G∥Θ⋆ ∥ εrymin M ⋆ β∥Θ⋆ ∥ , the collection {Cm }m=1 satisfies κ-linear-regularity on Bδ (Θ ) with constant κ = rymin . (m)
Proof. We define the constraint functions as hi,m (Θ) = f (xi Homogeneous Function Theorem, (m)
⟨∇hi,m (Θ⋆ ), Θ⋆ ⟩ = ⟨∇Θ f (xi
(m)
; Θ⋆ ), Θ⋆ ⟩ = rf (xi
(m)
; Θ) − yi
(m)
)(Θ⋆ )Θ⋆ = ryi
= 0. By Euler’s
≥ rymin .
and, if hm is the function hm = (h1,m . . . , hnm ,m ), P P | j λ[j]⟨∇hj (Θ⋆ ), Θ⋆ ⟩| |⟨Θ⋆ , ∇hm (Θ⋆ )⊤ λ⟩| rymin |λ[j]| ⋆ ⊤ ∥∇hm (Θ ) λ∥ ≥ = ≥ . ∥Θ⋆ ∥ ∥Θ⋆ ∥ ∥Θ⋆ ∥ Thus, if ∇hm (Θ⋆ )⊤ λ = 0, this implies λ = 0, thus, metric regularity holds. As a result, by Proposition 4, for x near Θ⋆ it holds for µ=
1 ∥Θ⋆ ∥ ≤ , ymin r λ∈ND (hm (Θ⋆ )),∥λ∥=1 ∇hm (Θ⋆ )λ max
that d(Θ, C ⋆ ) ≤ µ max |hi,m (Θ)| ≤ i,m
∥Θ⋆ ∥ max |hi,m (Θ)|. rymin i,m
Now, by local G-Lipschitzness, for every i, m it holds that, |hi,m (Θ)| = |hi,m (Θ) − hi,m (ΠCm (Θ)| ≤ Gd(Θ, Cm ) Since this holds also for the maximal example, we get, d(Θ, C ⋆ ) ≤
G∥Θ⋆ ∥ max d(Θ, Cm ). rymin m∈M
For (ε, δ)-regularity, we have the following lemma. 40
C ONVERGENCE OF C ONTINUAL L EARNING IN H OMOGENEOUS D EEP N ETWORKS
Lemma D.3 ((ε, δ)-Regularity of Homogeneous Models) Consider an r-positively-homogeneous model f (·; Θ̄) : X → R that is G-Lipschitz and β-smooth. Assume joint separability, i.e., nonempty (m) intersection C ⋆ = C1 ∩ · · · ∩ CM ̸= ∅. Let ymin = minm,i |yi | ̸= 0 and Θ⋆ ∈ C ⋆ . Then, for every εrymin ε > 0 and δ = β∥Θ⋆ ∥ , every feasible set Cm is (ε, δ)-regular at Θ⋆ . Proof. Let Cm = {Θ | hm (Θ) = 0}. Since hm is smooth, its gradient ∇hm (Θ) is continuous. To prove (ε, δ)-regularity, we need to show that for any x, y ∈ Cm ∩ Bδ (Θ⋆ ) and any proximal normal vector v ∈ NCprox (Θ⋆ ), the following holds: m ⟨v, y − x⟩ ≤ ε∥v∥∥y − x∥. Let v ∈ NCprox (Θ⋆ ) be such a vector. By Euler’s Homogeneous Function Theorem, m (m)
⟨∇hi,m (Θ⋆ ), Θ⋆ ⟩ = ⟨∇Θ f (xi
(m)
; Θ⋆ ), Θ⋆ ⟩ = rf (xi
(m)
)(Θ⋆ )Θ⋆ = ryi
≥ rymin .
If hm is the function hm = (h1,m . . . , hnm ,m ), P P | j λ[j]⟨∇hj,m (Θ⋆ ), Θ⋆ ⟩| rymin |λ[j]| rymin ⋆ ⊤ ∥∇hm (Θ ) λ∥ ≥ ≥ ≥ ∥λ∥ ⋆ . ⋆ ⋆ ∥Θ ∥ ∥Θ ∥ ∥Θ ∥
(16)
(17)
Since hm is smooth vector valued mapping we have β ′ such that, ∥hm (y) − hm (x) − ∇hm (x)(y − x)∥ ≤
β′ ∥y − x∥2 2
By Proposition 6, and using the Cauchy-Schwarz inequality on the error term, it holds that: ⟨v, y − x⟩ = ⟨∇hm (x)⊤ λ, y − x⟩ = ⟨λ, ∇hm (x)(y − x)⟩ = ⟨λ, hm (y) − hm (x) − (hm (y) − hm (x) − ∇hm (x)(y − x))⟩ = ⟨λ, hm (y) − hm (x)⟩ − ⟨λ, hm (y) − hm (x) − ∇hm (x)(y − x)⟩ ≤ ⟨λ, hm (y) − hm (x)⟩ + ∥λ∥ ∥hm (y) − hm (x) − ∇hm (x)(y − x)∥ =
β ′ ∥λ∥ ∥y − x∥2 . 2 ⋆
∥Θ ∥ Thus, by Eq. (17), to satisfy the (ε, δ)-regularity condition, we denote κ = ry . We need to choose min δ such that for every x, y ∈ Bδ (x̄)
κβ ′ ∥v∥∥y − x∥2 ≤ ε∥v∥∥y − x∥. 2 Dividing by ∥v∥∥y − x∥ (assuming nonzero, otherwise the inequality holds trivially), this requires κβ ′ 2 ∥y − x∥ ≤ ε. Since x, y ∈ Bδ (x̄), we have ∥y − x∥ ≤ 2δ. Thus, it suffices to choose δ such that: κβ ′ ε (2δ) ≤ ε =⇒ δ ≤ . 2 κβ ′ With this δ, the set is (ε, δ)-regular.
41
S CHLISERMAN B UZAGLO E VRON S OUDRY
Proof of Theorem 15. The proof follows identically to the classification case, utilizing the established (ε, δ)-regularity and κ-linear regularity for the equality-constrained sets. For the forgetting, the statement follows by the fact that, for every m, (m)
Fm (Θ̄t ) = max f (xi i
(m) 2
; Θ⋆t ) − yi
(m)
= max f (xi i
≤ G2 d2 (Θ̄t , C ⋆ ).
42
(m)
; Θ̄t ) − f (xi
; ΠC ⋆ (Θ⋆ )
2