SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neuralnetworks

machine learning, deep learning, neural networks

SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

arXiv:2605.22743v1 [cs.LG] 21 May 2026

Javad Parsa∗ Uppsala University, ETH Zurich

Enis Simsar* ETH Zurich

Amir Joudaki ETH Zurich

Thomas Hofmann ETH Zurich

André M. H. Teixeira Uppsala University, Sweden

Abstract Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models, but composing multiple custom concepts remains challenging due to representation interference. Existing modular methods either rely on expensive post-hoc fusion or freeze adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization. Theoretically, we establish strong convergence guarantees for our algorithm and model the residual layer activations as a matrix sub-Gaussian process to derive high-probability bounds on catastrophic forgetting. We further prove that learning the LoRA basis from data minimizes residual interference energy more effectively than frozen-basis methods. Experiments on multi-concept image generation demonstrate that SeqLoRA improves identity preservation and scalability across up to 101 concepts, while avoiding costly fusion and reducing attribute interference in composed generations.

Introduction

Text-to-image diffusion models have transformed visual content generation by allowing users to create high-quality, diverse images directly from natural-language prompts [Ho et al., 2020, Rombach et al., 2022, Saharia et al., 2022, Ramesh et al., 2022]. These models are trained to invert a progressive noising process, gradually converting random noise into a coherent image that aligns with a textual description. Thanks to their stable optimization behavior and strong ability to capture diverse data modes, diffusion models now support a wide range of applications, including artistic creation, media production, data augmentation, and scientific visualization [Po et al., 2024b, Luo et al., 2025]. Largescale text-to-image systems such as Stable Diffusion [Rombach et al., 2022] and DALL·E [Ramesh et al., 2022] have seen broad adoption, while parameter-efficient fine-tuning methods, especially Low-Rank Adaptation (LoRA) [Hu et al., 2022], have made it feasible for individual users to adapt these foundation models to new visual concepts using only a small set of reference images [Ruiz et al., 2023, Gal et al., 2023]. While single-concept customization has become highly mature [Ruiz et al., 2023, Gal et al., 2023, Hu et al., 2022], a key challenge arises when users want to generate images that include multiple personalized concepts at once, for instance, two specific pets interacting in the same scene, or a particular artistic style combined with a custom object [Kumari et al., 2023, Gu et al., 2023]. The central scientific issue is representation interference: because concepts are usually learned separately, their corresponding adaptations may occupy overlapping regions in the model’s parameter space [Po ∗ Equal contribution. Correspondence to: [email protected]

Preprint.

SeqLoRA (Bilevel)

Continual Alternating

LoRACLR

Mix-of-Show

Orthogonal Adaptation

wearableglasses

scenebarn

actionfigure1

Input Images

Figure 1: Qualitative comparison of multi-concept image generation across different methods for 32 concepts. We show the original concept images (Input) and 4 generated samples (seeds) for all methods.

et al., 2024a, Liang and Li, 2024]. When these adaptations are combined at inference time, the shared directions can lead to attribute entanglement, where identity-related features from one concept bleed into another, resulting in mixed attributes, weakened identities, or even incoherent generations [Po et al., 2024a, Gu et al., 2023]. This problem is especially pronounced for semantically similar concepts, such as different human faces, whose learned weight residuals tend to be highly correlated and lie in nearly the same subspaces [Po et al., 2024a]. As the number of concepts increases, the combinatorial growth in possible concept subsets makes it infeasible to train a separate model for every combination, motivating modular methods in which independently fine-tuned components can be composed at test time without retraining [Gu et al., 2023, Po et al., 2024a]. Several recent works have proposed strategies to mitigate concept interference, each offering partial solutions with distinct trade-offs. Joint training methods such as Custom Diffusion [Kumari et al., 2023] and Break-A-Scene [Avrahami et al., 2023] optimize all concepts simultaneously within a shared model, which can reduce interference but requires concurrent access to all concept data, a requirement that is unscalable to large concept libraries and raises data privacy concerns in multiuser settings. Gradient fusion approaches, exemplified by Mix-of-Show [Gu et al., 2023], allow independent fine-tuning of each concept and then merge the resulting models through an optimizationbased fusion step to preserve identity; however, the fusion must be re-executed for every new combination of concepts, preventing truly modular composition where individual adapters can be added or removed on the fly. Orthogonal Adaptation [Po et al., 2024a] takes a different approach by sampling each concept’s LoRA basis Bi from a shared orthogonal matrix and freezing it during training, optimizing only the coefficient matrix Ai . This guarantees approximate orthogonality between concepts and enables instant merging via simple summation, but freezing Bi restricts the feasible space of the adaptation and limits per-concept fidelity. In the continual learning literature, InfLoRA [Liang and Li, 2024] designs task-specific LoRA subspaces for image classification that are orthogonal to the gradient subspaces of previously learned tasks, achieving strong stability–plasticity trade-offs for sequential classification; however, it similarly freezes the basis matrix for each task, inheriting the same expressiveness limitation. A common thread across these methods is the tension between orthogonality, which is required to prevent interference, and expressiveness, which is required for high-fidelity adaptation: existing methods do not simultaneously permit both LoRA factors to be optimized freely while enforcing the subspace orthogonality necessary for low interference composition. In this work, we resolve this tension by formulating multi-concept generation as a constrained continual learning problem. As a preview of our results, Figure 1 visualizes a qualitative comparison on a 32-concept task. Notably, for challenging concepts like ”wearableglasses”, baseline methods suffer from severe attribute entanglement, whereas SeqLoRA (our method) successfully preserves distinct concept identities. Contributions.

This paper makes the following contributions: 2

• We introduce Sequential regularized LoRA (SeqLoRA), which jointly optimizes both LoRA factors via bilevel optimization while enforcing subspace orthogonality through a closedform projection, resolving the expressiveness–interference trade-off. • We provide theoretical analyses establishing: (a) monotone descent and convergence to a critical point of the constrained problem (Theorem 1), and (b) derive high-probability catastrophic forgetting bound showing that learning the LoRA bases reduces residual interference compared to frozen bases (Theorem 2). • We evaluate SeqLoRA on multi-concept generation with up to 101 concepts, demonstrating state-of-the-art identity preservation and scalability beyond the capacity of existing fusionbased methods. Structure of the paper. The remainder of the paper is organized as follows. Section 2 introduces the continual-learning model, the LoRA parameterization, and the interference problem. Section 3 develops the bilevel optimization formulation, derives the closed-form projection, and presents the SeqLoRA algorithm with its theoretical analysis. Section 4 reports experimental results. Section 5 discusses limitations and broader impact.

Problem Formulation

2.1

Continual Learning Model

We consider a sequential concept stream τ1 , . . . , τT , where each task τi provides a small reference i dataset Di = {(xi,m , ci )}M m=1 pairing Mi reference images with a concept-specific text token ci . Only Di is accessible when task τi arrives; prior datasets are discarded. The goal is a compact, composable representation per concept such that any subset of the T concepts can be jointly rendered at inference time without retraining, balancing plasticity (adapting to new concepts) and stability (preserving previous ones). 2.2

Single-Concept Fine-Tuning with LoRA

LoRA [Hu et al., 2022] has emerged as the dominant strategy for parameter-efficient fine-tuning of diffusion models. Rather than updating the full weight matrix Wi ∈ Rn×m of every attention layer, LoRA introduces a low-rank residual for each concept i: Wi = W0 + Ai B⊤ i ,

(1) n×r

m×r

where W0 contains the pretrained parameters for that layer, Ai ∈ R and Bi ∈ R are the trainable low-rank factors with rank r ≪ min{m, n}. For a single concept i, the standard training objective minimizes the denoising loss 2 (t) Li (Ai , Bi ) = Exi ,ϵ,t ϵ − ϵθ(Wi ) (xi , t, ci ) , (2) 2

(t)

where ϵ ∼ N (0, I) is the noise target, t is a uniformly sampled diffusion timestep, xi is the noised version of the reference image, ϵθ is the noise-prediction network, and θ(Wi ) denotes the full network parameters with the adapted layer weight set to Wi . 2.3

Multi-Concept Aggregation and the Interference Problem

In a practical deployment scenario, typically each concept is fine-tuned independently, potentially by different users in parallel, producing a separate LoRA module (Ai , Bi ) per concept [Gu et al., 2023, Po et al., 2024a]. At inference time, the goal is to compose an arbitrary subset of these modules into a single model for multi-concept generation. The simplest aggregation strategy is to sum the individual LoRA residuals per layer: T X WT = W0 + Ai B⊤ (3) i . i=1

To explain why naive aggregation fails, consider a linear layer adapted for concept i with input Xi . The single-concept output is Oi (Xi ) = (W0 + Ai B⊤ i )Xi , while the merged output becomes 3

⊤ ⊤ Ôi (Xi ) = (W0 + Ai B⊤ i + Aj Bj )Xi . The discrepancy Ôi − Oi = Aj Bj Xi is the crosstalk term [Po et al., 2024a]. Concept identity is preserved when this term vanishes, but since Xi typically has full column rank, exact cancellation is impossible. A natural relaxation [Po et al., 2024a] projects Xi onto the orthogonal complement of col(Bj ). Let B̄j ∈ Rm×(m−r) span this complement; then ⊤ ⊤ Aj B⊤ j B̄j B̄j Xi = 0 since Bj B̄j = 0. Because r ≪ min{m, n}, the complement covers most of m ⊤ R , so B̄j B̄j Xi ≈ Xi . At the same time, for concept i’s residual to remain effective, the columns of Bi must not lie in col(Bj ), which requires B⊤ i Bj = 0. This establishes pairwise orthogonality of the LoRA bases as the key condition for low interference aggregation.

2.4

Desiderata

The preceding analysis motivates three requirements: (i) pairwise orthogonality B⊤ i Bj = 0 for i ̸= j to reduce inference-time interference; (ii) joint optimization of Ai and Bi to preserve expressiveness; and (iii) continual learning, accommodating new concepts sequentially without access to past data or joint retraining. Prior works [Po et al., 2024a, Liang and Li, 2024] sacrifice (ii) by freezing Bi . The following warm-up motivates why learning the basis is preferable to freezing it. Example 1 (Warm-Up: why the basis matters). Consider a single-layer linear map with input x ∈ Rm and output y ∈ Rn . For concept j with distribution Dj , define the population risk Lj (W) ≜ E(x,y)∼Dj {∥y − Wx∥22 } and the input covariance Σj ≜ Ex∼Dj {xx⊤ }. Under orthogonality constraints, the interference for concept j after learning T concepts is proportional to its uncaptured ⊥ ⊥ residual energy: Tr(PB Σj ), where PB is the orthogonal projector onto the complement of col(Bj ) j j (full derivation in Theorem 8 in Appendix E). To minimize this interference, the rank-r basis Bj must align with the top-r principal components of Σj . Consequently, randomly frozen bases (e.g., Orthogonal Adaptation [Po et al., 2024a]) fundamentally fail to capture this spectral geometry, leaving large residual interference. Importantly, the local top eigenspace is not necessarily the optimal solution for the end-to-end loss in non-linear diffusion models under orthogonality constraints. By directly minimizing the end-to-end objective, we aim to implicitly discover a maximal-energy subspace tailored for concept reconstruction, which yields the optimal projection to suppress interference. In the following section, we introduce our proposed method, which is specifically designed to satisfy all three criteria simultaneously by actively navigating this subspace trade-off.

Proposed Method

Given the sequential concept stream and problem setup defined in Section 2, SeqLoRA treats the previously learned bases {Bj }j<i as a compact memory of past concepts and enforces low interference under additive LoRA composition by constraining each new basis Bi to lie in the orthogonal complement of their column span. The optimization procedure and its solution are detailed below. 3.1

Constrained Optimization Problem

When concept i arrives, we seek LoRA factors (Ai , Bi ) that minimize the denoising loss while keeping Bi orthogonal to all previously learned bases: min

Ai , Bi ∈Bi

where

Li (Ai , Bi ),

Bi = {Bi ∈ Rm×r | B⊤ j Bi = 0,

∀ j < i, 0 ∈ Rr×r }.

(4) (5)

Problem (4) is jointly non-convex in (Ai , Bi ), but it has favorable structure: Li is differentiable with respect to each factor individually, and the constraints are linear in Bi . 3.2

Bilevel Optimization Algorithm

Several techniques for solving (4) already exist, including the BSUM algorithm by Razaviyayn et al. [2013] and the prox-linear approach introduced in [Drusvyatskiy et al., 2019]. These algorithms rely 4

on alternating between updating the Ai for fixed Bi , and Bi for fixed Ai . In our experience, such updates tend to be too aggressive, and miss important couplings between the two variable blocks that are helpful for fine-tuning models. To address these challenges, we propose a new algorithm based on rewriting (4) as a bilevel optimization problem [Dempe, 2002]: min Li (A∗i (Bi ), Bi )

Bi ∈Bi

subject to

A∗i (Bi ) = argmin Li (Ai , Bi ).

(6)

To solve the lower-level optimization problem, we use a quadratic approximation L̂i (Ai , Bi ) of Li (Ai , Bi ) 1 (k) 2 Ai − Ai F . (7) 2α Substituting this quadratic approximation into the lower-level optimization problem in (6) leads to: (k)

(k)

L̂i (Ai , Bi ; Ai ) = Li (Ai , Bi ) + ∇Ai Li (Ai , Bi ), Ai − Ai

(k+1)

(k)

(Bi ) = arg min L̂i (Ai , Bi ; Ai ) = Ai Ai

(k)

− α ∇Ai Li (Ai , Bi ),

(8)

where α is a step size. Note that the model update depends on the Bi . This dependence is accounted for in the upper-level optimization in (6), whose cost function becomes (k+1) (k) (k) Φi (Bi ) ≜ Li Ai (Bi ), Bi = Li Ai − α ∇Ai Li (Ai , Bi ), Bi , (9) which is now a function of Bi alone. The constrained upper-level problem is min Φi (Bi ).

(10)

Bi ∈Bi

The relationship between Φi and Bi follows from the chain rule. Based on (9), perturbations in Bi influence the objective in two ways: a direct effect through the partial derivative ∇Bi Li , and (k+1) an indirect effect through the dependence of the updated factor Ai (Bi ) on Bi . This indirect 2 pathway is characterized by mixed derivatives ∇Bi Ai Li , which measure how the gradient with respect to Ai varies as Bi changes. Therefore, solving (10) seeks a basis Bi that both reduces the denoising loss at the current iterate and shapes the lower-level update of Ai toward directions that further decrease the loss, while the projection onto Bi maintains orthogonality across concepts. Although (8) corresponds to a single gradient descent step, the upper-level problem (10) remains difficult due to the non-convexity of Φi and the orthogonality constraints encoded by Bi . To handle (k) this, we construct a quadratic model Φ̂i (Bi ) around Bi : (k)

(k)

Φ̂i (Bi ; Bi ) = Φi (Bi ) + ∇Bi Φi (Bi ), Bi − Bi

1 (k) 2 Bi − Bi F , 2β

(11)

(k)

where β > 0 is the upper-level step-size. Note that evaluating (8) at Bi = Bi yields (k+1) (k) Li (Ai (Bi ), Bi ), which recovers the standard alternating-minimization pattern: (i) fix Bi = (k) (k+1) Bi and update Ai , then (ii) fix Ai = Ai and update Bi . However, unlike standard alternat(k+1) (k+1) ing minimization, which descends only the partial objective Li (Ai , ·) with frozen Ai , the bilevel gradient ∇Bi Φi additionally captures the coupling between factors through a cross-Hessian correction ∇2Bi Ai Li , yielding a richer descent direction on the true reduced objective. We include this standard alternating minimization variant (without the cross-Hessian correction) as a baseline in our experiments, denoted Continual Alternating in Section 4. Minimizing this quadratic model (11) subject to the orthogonality constraints replaces (10) by (k+1)

(k)

= argmin Φ̂i (Bi ; Bi ).

(12)

Bi ∈Bi

By taking the gradient of the main cost function and dropping constant terms, the above update is equivalent to ei 2 , min Bi − B (13) F Bi ∈Bi

where

e i = B(k) − β ∇B Φi (B(k) ). B i i i 5

(14)

(k)

We use the chain rule to compute the gradient ∇Bi Φi (Bi ). So, it is straightforward to show that h i (k) (k) (k+1) (k) (k+1) (k) (k) ∇Bi Φi (Bi ) = ∇Bi Li (Ai (Bi ), Bi )|Bi =B(k) − α HAB ∇Ai Li Ai (Bi ), Bi . i (15) (k) The term HAB [·] : Rn×r → Rm×r is the cross-Hessian contraction operator. Directly instantiating this 4th-order tensor might be computationally prohibitive. Instead, we compute the required Hessianmatrix product efficiently using a nested automatic differentiation strategy (a double grad or Hessianvector product trick). For a given inner residual gA = ∇Ai Li , this operation is mathematically equivalent to: (k) (k) HAB [gA ] = ∇Bi Tr gA ⊤ ∇Ai Li (Ai , Bi ) . (16) (k)

Bi =Bi

The optimization problem (13) is a convex quadratic program with linear equality constraints. Let Bint = [B1 , . . . , Bi−1 ] ∈ Rm×(i−1)r denote the column-wise concatenation of all previously learned bases, allowing us to express the orthogonality constraint compactly as B⊤ int Bi = 0. ei The minimizer of this problem is simply the exact orthogonal projection of the unconstrained step B onto the null space of B⊤ . By solving the KKT conditions, this yields the closed-form geometric int update: h −1 ⊤ i (k+1) e i = Projspan{B }⊥ B ei . Bi = I − Bint B⊤ Bint B (17) int Bint j j<i | {z } ⊥ ≜ PB

(k+1)

⊥ The projector PB strips gradient components overlapping with past concepts. Given Bi i lower-level factor is updated by (17): (k+1) (k) (k) (k+1) Ai = Ai − α∇Ai Li Ai , Bi .

, the (18)

After iterations we freeze (Ai , Bi ), append Bi to Bint , and update Wi = W0 + P K bilevel ⊤ A B . The same procedure is applied to all adapted layers, and processing the concept j j j≤i stream i = 1, . . . , T in sequence yields the full method summarized in Algorithm 1. Remark 1. While evaluating the cross-Hessian adds marginal per-step overhead, it captures critical parameter coupling to yield a strictly richer descent direction. This accelerates convergence, requiring fewer total iterations. Consequently, this faster convergence helps to amortize the Hessian-vector product cost, ensuring highly competitive overall training efficiency (see Section 4). 3.3

Theoretical Results

SeqLoRA uses quadratic surrogate models L̂i and Φ̂i to simplify bilevel optimization. A key question is whether minimizing these local approximations guarantees convergence to a critical point of the main loss function in (4), which is addressed in the following theorem. Theorem 1. Under Assumption 1 (see Appendix A), let Li (A, B)+δBi (B) be the composite objective, (k) (k) δBi (B) is an indicator function, and let {(Ai , Bi )}K k=0 be generated by Algorithm 1. If the step 1 1 sizes satisfy 0 < α < L and 0 < β < LΦ , where LΦ ≤ L(1 + αL)2 + αρM , then: 1. Strict Monotonic Descent: The composite objective decreases monotonically at each 1 iteration k. In particular, the step sizes α∗ = 2L and β ∗ = 2L1Φ maximize the descent coefficients, yielding the sharpest guaranteed decrease. (k)

(k)

2. Asymptotic Stationarity: The sequence of iterates {(Ai , Bi )} asymptotically converges to a critical point of the constrained optimization problem (4). Proof. See Appendix B. Building on our linear warm-up (Example 1), we now bound end-to-end catastrophic forgetting in deep, non-linear diffusion models. Modeling the uncaptured layer-wise residuals as a stochastic sub-Gaussian process, the following theorem establishes a high-probability network-wide bound. 6

Theorem 2 (Hanson-Wright High-Probability Bound on Catastrophic Forgetting). Under Assumption L Q (ℓ′ ) 2 (see Appendix A), let Γℓ ≜ γℓ′ ∥WT ∥2 be the downstream amplification factor at layer ℓ ℓ′ =ℓ+1

(ℓ)

and let Cj =

T P k=j+1

(ℓ)

Ak (Bk )⊤ be the crosstalk operator for concept j. For any ξ ∈ (0, 1), there

exists an absolute universal constant c > 0 such that the end-to-end catastrophic forgetting satisfies, with probability at least 1 − ξ: q L X (ℓ) (ℓ) Lj (θT ) − Lj (θj ) ≤ Lo Γℓ Tr(Ψj ) ∥Qj ∥2F + t(ℓ) , (19) ℓ=1 (ℓ) (ℓ) L where θT = {WT }L ℓ=1 and θj = {Wj }ℓ=1 are the full model’s parameters after learning concept

T and j, respectively, and r p 2 2 (ℓ) (ℓ) ⊤ (ℓ) (ℓ) (ℓ) 2 2 2 (ℓ) t = max C1 K ∥Ψj ∥F ∥(Qj ) Qj ∥F log , C1 K ∥Ψj ∥2 ∥Qj ∥2 log , (20) ξ ξ (ℓ)

(ℓ)

(ℓ),⊥ 1/2

C1 = 1/c, and Qj ≜ Cj (Σj (ℓ)

(ℓ),⊥

energy Tr(Ψj )Tr(Σj

)

∈ Rnℓ ×mℓ . Additionally, minimizing the expected residual (ℓ)

) requires the rank-r basis Bj to span the top-r principal components

(ℓ)

of Σj within the available orthogonal complement. Thus, randomly frozen bases are generically suboptimal, yielding higher residual energy unless the remaining feature space is isotropic. Proof. See Appendix C. (ℓ),⊥

The forgetting bound in Theorem 2 is controlled at each layer by the residual covariance Σj

(ℓ) which measures the activation energy that leaks outside concept j’s learned subspace col(Bj ).

Orthogonality ensures this leaked energy is the only source of interference. Additionally, because (ℓ) Xj is produced by the network θj optimized for concept j, the LoRA bases across layers work in (ℓ−1)

(ℓ)

concert; features not captured by Bj can be absorbed by Bj , a process which encourages the per-layer residual covariance to remain small throughout the network depth. The bound is minimized (ℓ) when each Bj aligns with the top principal components of its local activations, extending the linear warm-up (Example 1) to the non-linear regime. SeqLoRA does not explicitly compute this top eigenspace to minimize interference. Instead, by minimizing the end-to-end task loss, the algorithm implicitly seeks this maximal-energy subspace to reconstruct the concept, which happily coincides with the optimal projection that suppresses catastrophic forgetting. A detailed physical interpretation appears in Appendix D.

Experiments

We evaluate SeqLoRA against state-of-the-art baselines: Mix-of-Show [Gu et al., 2023], Orthogonal Adaptation [Po et al., 2024a], LoRACLR [Simsar et al., 2025], and Continual Alternating (see Subsection 3.2) to isolate the contribution of the cross-Hessian coupling. To assess scalability, we vary the number of concepts from 8 to 101. All methods are evaluated using prompts from the CustomConcept101 dataset [Kumari et al., 2023], generating 16 images per prompt for metric computation. We evaluate identity preservation with DINO/DINOv2/DINOv3, holistic visual similarity with DreamSim, image-text alignment with CLIP-I/CLIP-T, and visual quality with HPSv2/HPSv3. Full metric definitions appear in Appendix F. Experimental Setup. We use Stable Diffusion v1.5 as the base text-to-image model, consistent with Mix-of-Show [Gu et al., 2023]. All methods are fine-tuned on the full multi-concept stream. Quantitative evaluations probe the resulting model by generating each concept individually to measure per-concept performance; Appendix I presents a qualitative compositional study where multiple concepts are rendered together under regional sketch and keypose conditioning. For SeqLoRA 7

(Algorithm 1), default hyperparameters are: bilevel iterations K = 3, inner steps 2, regularization weight ϵ = 10−8 . All experiments use 2× NVIDIA A100 GPUs. We report mean ± standard error across concepts in tables and mean performance in figures; details in Appendix G. Efficiency. Parallel methods (LoRACLR, Mix-of-Show, Orthogonal Adaptation) train per-concept adapters independently (∼78s each) and fuse them post-hoc, while SeqLoRA and Continual Alternating learn sequentially in a single model with no fusion step. The accumulating orthogonality constraints raise the per-concept cost to ∼454s for Continual Alternating and ∼660s for SeqLoRA (the latter due to cross-Hessian evaluations). However, while parallel methods can leverage distributed computing to reduce wall-clock training time (at the cost of higher total compute), they require a complex and often time-consuming optimization step to fuse the independent adapters into a single multi-concept model. At 32 concepts, post-hoc fusion itself takes ∼33 min for Mix-of-Show and ∼8 min for LoRACLR (Orthogonal Adaptation fuses in 15s via simple summation), an overhead SeqLoRA avoids entirely while also supporting truly sequential lifelong learning without data accumulation. In Table 1a, we provide a detailed comparison for the 32-concept experiment. SeqLoRA (Ours) outperforms all other methods in identity preservation and perceptual similarity metrics (CLIP-I, DINO, DINOv2, DINOv3, DreamSim), while remaining competitive on text alignment (CLIP-T) and human preference scores (HPSv2, HPSv3). This highlights the effectiveness of our bilevel optimization in preserving learned concepts without complex fusion steps. To visualize scalability, we report the mean performance across all concepts at the end of training for different concept counts in Figure 3a (see Table 3 for full quantitative). We also successfully scale SeqLoRA to 101 concepts, observing consistent trends in identity preservation and visual quality, while other methods like Mix-of-Show and LoRACLR fail due to OOM (Out of Memory) errors at this scale (detailed results are provided in the Appendix). To analyze forgetting behavior, we plot the step-based mean performance of all visible concepts at each step for the 32-concept experiment in Figure 3b for SeqLoRA and Continual Alternating. This step-based view reflects the overall health of the system during sequential learning. Figure 3b reveals that while text alignment (CLIP-T) remains comparable, SeqLoRA shows comparable or better stability across visual fidelity metrics (CLIP-I, DINO variants, and DreamSim), highlighting the stability gained from bilevel optimization. We summarize the mean forgetting metrics for SeqLoRA across different concept counts in Table 1b and visualize the evolution of generated images across steps in Figure 2. To ensure a fair visual comparison, the same random seed and the most basic prompt template (e.g., “photo of a < concept >”) are used for image generation across all training steps for a given concept. Additional experimental details, including qualitative comparison, an ablation study on hyperparameters, and qualitative forgetting analysis, are provided in the Appendix.

Limitations and Broader Impact

Limitations. The orthogonality constraint limits the number of concepts per layer to ⌊m/r⌋; however, with typical layer dimensions (e.g., m=768, r=4), this accommodates up to 192 concepts, well beyond the scales we evaluate and most practical scenarios. The bilevel formulation introduces a cross-Hessian–vector product per iteration that increases the per-concept training cost relative to alternating minimization; however, this overhead is more than offset for sequential, modular use cases by avoiding the costly post-hoc fusion step required by parallel methods (e.g., ∼33 minutes for Mix-of-Show at 32 concepts). Our theoretical results rely on Assumptions 1 and 2, which are mild standard conditions discussed in Appendix A. Current experiments use Stable Diffusion v1.5 for consistency with prior work; extending to newer architectures is a natural next step. Finally, while SeqLoRA processes concepts sequentially, exploring robustness to concept ordering is an interesting direction for future investigation. Broader Impact. SeqLoRA advances multi-concept personalization of text-to-image diffusion models, which has broad positive applications in creative design, media production, and accessibility. As with all generative modeling research, improvements in image fidelity could potentially be misused for generating misleading visual content. However, SeqLoRA does not introduce new generative capabilities beyond what existing diffusion models already provide; it improves the composability of 8

personalized adaptations within established frameworks. We encourage the responsible use of such technology in accordance with applicable guidelines and regulations.

Conclusion

This paper introduces SeqLoRA, a novel framework that resolves the expressiveness-interference trade-off in continual multi-concept generation. By formulating adaptation as a constrained bilevel optimization problem, SeqLoRA dynamically optimizes both LoRA factors while enforcing exact subspace orthogonality. We prove convergence guarantees and establish theoretical bounds on catastrophic forgetting, and theoretically prove that optimizing both LoRA bases is superior to freezing the projection subspace. Our comprehensive experiments on up to 101 concepts demonstrate that SeqLoRA outperforms existing methods in identity preservation and visual quality, while maintaining competitive text alignment. Notably, some competing methods encounter OOM errors at larger concept counts, whereas SeqLoRA scales gracefully. Furthermore, bilevel optimization shows clear superiority over standard alternating minimization for solving the constrained problem.

(a) Overall performance comparison for 32 concepts across different methods. Best results are in bold, second best are underlined. Reference Similarity Metrics Methods Continual Alternating Mix-of-Show Orthogonal Adaptation LoRACLR SeqLoRA (Ours)

Prompt Following & Aesthetics

DINO↑

DINOv2↑

DINOv3↑

DreamSim↑

CLIP-I↑

CLIP-T↑

HPSv2↑

HPSv3↑

0.463 ± 0.020 0.436 ± 0.023 0.428 ± 0.023 0.434 ± 0.023 0.468 ± 0.021

0.412 ± 0.017 0.377 ± 0.020 0.361 ± 0.020 0.372 ± 0.020 0.418 ± 0.019

0.406 ± 0.015 0.378 ± 0.017 0.359 ± 0.017 0.371 ± 0.017 0.418 ± 0.016

0.470 ± 0.015 0.448 ± 0.017 0.442 ± 0.018 0.445 ± 0.018 0.477 ± 0.016

0.688 ± 0.008 0.677 ± 0.010 0.671 ± 0.009 0.674 ± 0.010 0.693 ± 0.009

0.287 ± 0.004 0.276 ± 0.003 0.277 ± 0.003 0.277 ± 0.003 0.286 ± 0.003

0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.270 ± 0.001

9.242 ± 0.338 8.039 ± 0.457 8.336 ± 0.432 8.382 ± 0.431 9.043 ± 0.399

(b) Summary of forgetting metrics for SeqLoRA across different concept counts. # of Concepts

Measure

CLIP-I

CLIP-T

DINO

DINOv2

DINOv3

DreamSim

HPSv2

HPSv3

8 Concepts

Initial Final Forgetting

0.709 ± 0.018 0.695 ± 0.014 0.014 ± 0.006

0.286 ± 0.005 0.289 ± 0.005 -0.002 ± 0.001

0.485 ± 0.047 0.462 ± 0.041 0.022 ± 0.009

0.491 ± 0.049 0.465 ± 0.041 0.025 ± 0.009

0.468 ± 0.039 0.442 ± 0.030 0.026 ± 0.010

0.495 ± 0.040 0.478 ± 0.035 0.017 ± 0.008

0.270 ± 0.003 0.271 ± 0.003 -0.001 ± 0.000

9.513 ± 0.579 9.539 ± 0.588 -0.025 ± 0.020

16 Concepts

Initial Final Forgetting

0.703 ± 0.013 0.694 ± 0.012 0.009 ± 0.004

0.281 ± 0.004 0.284 ± 0.004 -0.002 ± 0.001

0.492 ± 0.030 0.474 ± 0.029 0.018 ± 0.007

0.436 ± 0.032 0.416 ± 0.028 0.020 ± 0.007

0.437 ± 0.027 0.414 ± 0.022 0.022 ± 0.008

0.493 ± 0.026 0.478 ± 0.024 0.014 ± 0.005

0.268 ± 0.002 0.269 ± 0.002 -0.000 ± 0.000

9.494 ± 0.416 9.476 ± 0.424 0.019 ± 0.029

32 Concepts

Initial Final Forgetting

0.696 ± 0.009 0.687 ± 0.009 0.008 ± 0.003

0.288 ± 0.003 0.290 ± 0.003 -0.001 ± 0.000

0.465 ± 0.022 0.452 ± 0.022 0.013 ± 0.004

0.413 ± 0.021 0.398 ± 0.019 0.016 ± 0.005

0.420 ± 0.018 0.402 ± 0.017 0.017 ± 0.005

0.481 ± 0.017 0.469 ± 0.016 0.012 ± 0.004

0.270 ± 0.001 0.270 ± 0.001 -0.000 ± 0.000

9.044 ± 0.396 9.043 ± 0.399 0.001 ± 0.025

Step 24

Step 28

Step 32

Table 1: Quantitative evaluation.

Step 1

Step 4

Step 8

Step 12

Step 16

Step 20

wearablesunglasses2 scenecastle

actionfigure1

scenebarn

Input Image

Not Learned

Figure 2: Evolution of generated images for selected concepts across different training steps for SeqLoRA. We show the input image followed by generated samples at steps 1, 4, 8, 12, 16, 20, 24, 28, and 32. 9

CLIP-I

CLIP-T Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

0.70

0.46

Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

0.46 0.44

0.294 0.292

Value (Mean)

0.68

0.290

Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

0.286

0.66

0.284 80

100

40 60 Number of Concepts

DINOv3

0.42

0.36 0.34

100

0.36

40 60 Number of Concepts

0.45 0.44

0.2710 0.2705 0.2700

0.43

0.2695

0.42

0.2690

100

40 60 Number of Concepts

Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

100

40 60 Number of Concepts

100

Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

0.2715

Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

HPSv3

9.5

0.34 40 60 Number of Concepts

100

0.2720

0.47

Value (Mean)

0.38

HPSv2

0.48

0.46 0.40

40 60 Number of Concepts

DreamSim

Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

0.44

0.38

Value (Mean)

40 60 Number of Concepts

0.40

Value (Mean)

0.44

0.42

0.288

0.67

Value (Mean)

0.42 Value (Mean)

Value (Mean)

DINOv2 Orthogonal Adaptation LoRACLR Mix-of-Show Continual Alternating SeqLoRA (Bilevel)

0.296

0.69

Value (Mean)

DINO

0.48

0.298

9.0

8.5

8.0

100

40 60 Number of Concepts

100

(a) Overall performance comparison across different numbers of concepts (from 8 to 101). Metrics are averaged across all targets at the end of training. CLIP-I

CLIP-T SeqLoRA (Bilevel) Continual Alternating

0.76 0.75

DINO SeqLoRA (Bilevel) Continual Alternating

0.294

0.60

0.55

0.71

0.288 0.286

0.70

0.284

0.69

0.282

0.68 5

15 Step

0.280

0.50

15 Step

SeqLoRA (Bilevel) Continual Alternating

0.625

0.450

0.282

0.575

0.280

0.550

0.278

0.525

SeqLoRA (Bilevel) Continual Alternating

15 Step

HPSv3

11.00

SeqLoRA (Bilevel) Continual Alternating

10.75 10.50 10.25

0.276

10.00 9.75 9.50 9.25

0.270

0.450 15 Step

0.272

0.475

0.400

0.274

0.500

0.425

15 Step

0.284

0.600

Value (Mean)

0.475

HPSv2

0.500

DreamSim SeqLoRA (Bilevel) Continual Alternating

0.40 0

0.525

0.50

0.45

DINOv3 0.550

0.55

Value (Mean)

0.72

Value (Mean)

0.290

0.73 Value (Mean)

SeqLoRA (Bilevel) Continual Alternating

0.60

0.292

0.74

Value (Mean)

DINOv2 SeqLoRA (Bilevel) Continual Alternating

0.65

15 Step

9.00 0

15 Step

(b) Forgetting trajectory for the 32-concept experiment. The plot shows the step-based average performance of all concepts available at each step, reflecting the overall health of the system during sequential learning.

Figure 3: SeqLoRA’s scalability and stability.

References Hedy Attouch, Jérôme Bolte, and Benar Svaiter. Convergence of descent methods for semi-algebraic and tame problems: Proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods. Mathematical Programming, 137, 01 2011. doi: 10.1007/s10107-011-0484-9. Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pages 1–12, 2023. Jerome Bolte, Shoham Sabach, and Marc Teboulle. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming, 146(1-2):459–494, 2014. doi: 10.1007/s10107-013-0701-9. URL https://inria.hal.science/hal-00916090. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 10

A Philip Dawid. Some matrix-variate distribution theory: notational considerations and a bayesian application. Biometrika, 68(1):265–274, 1981. Stephan Dempe. Foundations of Bilevel Programming, volume 61 of Nonconvex Optimization and Its Applications. Springer Science & Business Media, 2002. doi: 10.1007/978-1-4615-1503-8. Dmitriy Drusvyatskiy, Adrian S Lewis, and Catherine A Paquette. Efficiency of the prox-linear algorithm for composite optimization. Mathematical Programming, 178:503–558, 2019. Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient-based model-agnostic meta-learning algorithms. In International conference on artificial intelligence and statistics, pages 1082–1092. PMLR, 2020. Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344, 2023. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023. Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36:15890–15902, 2023. A. K. Gupta and D. K. Nagar. Matrix Variate Distributions. Chapman & Hall/CRC, 2000. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge University Press, Cambridge, 1991. Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2nd edition, 2012. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1931–1941, 2023. Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024. Ziwei Luo, Fredrik Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Taming diffusion models for image restoration: a review. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 383(2299), 2025. Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. Orthogonal adaptation for modular customization of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a. 11

Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit Bermano, Eric Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. In Computer graphics forum, volume 43, page e15063. Wiley Online Library, 2024b. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical textconditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022. Meisam Razaviyayn, Mingyi Hong, and Zhi-Quan Luo. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimization, 23 (2):1126–1153, 2013. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–22510, 2023. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022. Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. Enis Simsar, Thomas Hofmann, Federico Tombari, and Pinar Yanardag. Loraclr: Contrastive adaptation for customization of diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13189–13198, 2025. Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2 edition, 2026. Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.

Contents 1

Introduction

Problem Formulation

2.1

Continual Learning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2

Single-Concept Fine-Tuning with LoRA . . . . . . . . . . . . . . . . . . . . . . .

2.3

Multi-Concept Aggregation and the Interference Problem . . . . . . . . . . . . . .

2.4

Desiderata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Proposed Method

3.1

Constrained Optimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2

Bilevel Optimization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3

Theoretical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Experiments

Limitations and Broader Impact

Conclusion

A Mathematical Preliminaries

A.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.3 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B Proof of Theorem 1

C Proof of Theorem 2

C.1 Analysis of Catastrophic Forgetting . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Optimal basis selection inside the free complement . . . . . . . . . . . . . . . . .

D Decomposition of Catastrophic Forgetting: Mean Interference vs. Stochastic Deviation 27 E Population Risk Analysis in a Stylized Linear Setting

F Evaluation Metrics

G Detailed Evaluation Methodology

G.1 Mean Performance in Figures 3a and 3b . . . . . . . . . . . . . . . . . . . . . . .

G.2 Error Bars in Tables 1-4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

H Supplementary Qualitative Results

Multi-Concept Regional Generation 13

Supplementary Forgetting Results

K Supplementary Overall Results

L Supplementary Ablation Results

Algorithm 1 Sequential Regularized LoRA (SeqLoRA) (ℓ)

Require: Pretrained weights {W0 }L ℓ=1 ; concept datasets D1 , . . . , DT ; step sizes α, β; bilevel ′ iterations K; local iterations SB , SA ; regularization ϵ (ℓ) (ℓ) Ensure: LoRA factors {(Ai , Bi )}i,ℓ 1: for concept i = 1 to T do (ℓ,0) (ℓ,0) 2: Initialize {Ai , Bi }L ℓ=1 randomly (ℓ) (ℓ) (ℓ) 3: Bint ← [B1 , . . . , Bi−1 ] for all ℓ ∈ {1, . . . , L} ▷ Previous bases at each layer 4: for k = 0 to K − 1 do ▷ Bilevel iterations 5: 6: 7: 8: 9: 10: 11: 12:

(ℓ)

L // Step 1: Tentative lower-level update of {Ai }ℓ=1 (ℓ,k) (ℓ,k) (ℓ,k) e (ℓ) ← A A − α ∇A(ℓ) Li {A ,B }L for all ℓ i

ℓ=1 (ℓ)

// Steps 2–3: Reduced gradient + projected {Bi }L ℓ=1 update (ℓ,k) B(ℓ) ← Bi for all ℓ for s = 1 to SB do (ℓ) e (ℓ) , B(ℓ) }L ) for all ℓ gB ← ∇B(ℓ) Li ({A ℓ=1

▷ Local B-iterations ▷ Direct partial

(ℓ)

e (ℓ) , B(ℓ) }L ) for all ℓ gA ← ∇A(ℓ) Li ({A ℓ=1 i // Cross-Hessian via autodiff at current iterate (k,ℓ) (ℓ) (ℓ) ⊤ (ℓ,k) (ℓ) HAB [gA ] ← ∇B(ℓ) Tr (gA ) ∇A(ℓ) Li (Ai , Bi )

13: 14: 15:

17: 18:

24: 25: 26:

(ℓ)

(ℓ,k)

for all ℓ

i i Bi =Bi (ℓ) (k,ℓ) (ℓ) ▷ Reduced gradient ∇B(ℓ) Φi ← gB − α HAB [gA ] for all ℓ e (ℓ) ← B(ℓ) − β ∇B(ℓ) Φi for all ℓ B −1 (ℓ) (ℓ) ⊤ (ℓ) (ℓ) ⊤ e (ℓ) (ℓ) B ← I − Bint (Bint ) Bint + ϵI (Bint ) B for all ℓ ▷ Regularized

16:

19: 20: 21: 22: 23:

▷ Inner residual

projection end for (ℓ,k+1)

← B(ℓ) for all ℓ (ℓ)

(ℓ,k+1) L }ℓ=1

// Step 4: Final update of {Ai }L ℓ=1 using {Bi (ℓ,k) A(ℓ) ← Ai for all ℓ ′ do for s = 1 to SA

▷Local A-iterations at new {B(ℓ) } (ℓ,k+1) L A(ℓ) ← A(ℓ) − α ∇A(ℓ) Li {A(ℓ) , Bi }ℓ=1 for all ℓ end for

27: 28: (ℓ,k+1) 29: Ai ← A(ℓ) for all ℓ 30: 31: end for (ℓ) (ℓ) (ℓ,K) (ℓ,K) 32: Store (Ai , Bi ) ← (Ai , Bi ) for all ℓ 33: end for

Mathematical Preliminaries

This section gathers the notation, definitions, and assumptions used in the paper. A.1

Notations

We use bold lowercase letters for vectors and bold uppercase letters for matrices. For a matrix A, A⊤ , Tr(A), and vec(A) denote its transpose, trace, and vectorization, respectively. The Frobenius norm and the operator (spectral) norm are denoted by ∥A∥F and ∥A∥2 . The Kronecker product is denoted by A ⊗ B. For a matrix B, col(B) denotes its column space. The sub-Gaussian norm of a random variable Z is denoted by ∥Z∥ψ2 . We use superscripts in parentheses (e.g., W(ℓ) ) to index 15

network layers; subscripts denote either the LoRA factors associated with concept i (e.g., Ai , Bi ) or the full parameter state after learning concept i (e.g., Wi ), with the meaning clear from context. A.2

Definitions

Definition 1. [Attouch et al., 2011] The subdifferential of a PLSC function g at x ∈ Rn is defined as △

∂g(x) = {ζ ∈ Rn | ∃xk → x, g(xk ) → g(x), ζk → ζ ∈ ∂ĝ(xk )} where ∂ĝ(xk ) is the Fréchet subdifferential of g at xk ∈ Rn , defined as 1 ∂ĝ(xk ) = ζ ∈ Rn | lim inf [g(v) − g(x) − ⟨v − x, ζ⟩] ≥ 0 . v̸=x,v→x ∥v − x∥2 2

(21)

Definition 2. [Attouch et al., 2011] A point x∗ is called a critical point of a PLSC function f (x) if 0 ∈ ∂f (x∗ ). Lemma 3 (Descent lemma [Bolte et al., 2014]). Let f : Rn → R be a continuously differentiable function whose gradient ∇f is L-Lipschitz continuous. Then, for all x, y ∈ Rn : f (y) ≤ f (x) + ⟨∇f (x), y − x⟩ +

L ∥y − x∥22 . 2

(22)

Definition 3 (Operator norm of the cross-Hessian [Horn and Johnson, 2012]). For a linear map H : Rn×r → Rm×r , the operator norm is defined as: ∥H[V]∥F . V̸=0 ∥V∥F

∥H∥2 = sup A.3

(23)

Assumptions

To establish the convergence of the SeqLoRA algorithm (Theorem 1), we require the following standard regularity conditions on the optimization landscape: Assumption 1. The loss function Li (A, B) satisfies the following properties: • Li is jointly L-smooth with respect to A and B in the Frobenius norm. This inherently bounds the operator norm of the cross-Hessian: ∥∇2AB Li (A, B)∥2 ≤ L. • The cross-Hessian is ρ-Lipschitz continuous with respect to B: ∥∇2AB Li (A0 , B1 ) − ∇2AB Li (A0 , B2 )∥2 ≤ ρ∥B1 − B2 ∥F .

(24)

• The partial gradient of the loss with respect to A is bounded by a constant M > 0: ∥∇A Li (A, B)∥F ≤ M.

(25)

Discussion of Assumption 1. These are standard regularity conditions in the analysis of non-convex and bilevel optimization [Fallah et al., 2020]. Joint smoothness holds locally in any bounded region where training is stable. The bounded gradient assumption for the inner variable is a mild condition, further justified by the common deep learning practice of gradient clipping, which explicitly enforces such a bound to prevent training instability. The Lipschitz continuity of the cross-Hessian is a secondorder smoothness condition necessary for analyzing algorithms that leverage parameter coupling. Using the operator norm is the rigorous choice, as the cross-Hessian is a linear operator (a 4th-order tensor) mapping one matrix space to another. To bound the catastrophic forgetting across the continuous learning stream (Theorem 2), we employ the following structural assumptions on the network mappings and the statistical distribution of the residual activations: Assumption 2 (Multi-Layer Structure and Sub-Gaussian Residuals). The network applies LoRA (ℓ) adaptation at L layers. For a concept j at layer ℓ, let Xj ∈ Rmℓ ×p be the input activations and let (ℓ),⊥

(ℓ)

PBj denote the orthogonal projector onto the orthogonal complement of the column space of Bj . We assume: 16

1. Lipschitz Network Mappings: Between consecutive LoRA-adapted layers ℓ and ℓ + 1, the intermediate network operations compose a mapping ϕℓ+1 : Rnℓ ×p → Rmℓ+1 ×p that is γℓ -Lipschitz continuous with respect to the Frobenius norm. Furthermore, the task loss applied to the final layer output is Lo -Lipschitz. 2. Sub-Gaussian Residuals: At each LoRA-adapted layer ℓ, the residual input activation (ℓ),⊥ (ℓ),⊥ (ℓ) Xj = PBj Xj follows a matrix sub-Gaussian process: (ℓ),⊥

(ℓ),⊥ 1/2

= (Σj

(ℓ),⊥

)

(ℓ)

Z(ℓ) (Ψj )1/2 ,

(26) (ℓ)

where Σj ∈ Rmℓ ×mℓ captures the feature-dimension covariance, Ψj ∈ Rp×p captures the token-dimension correlation (sequence dependencies), and Z(ℓ) ∈ Rmℓ ×p consists of (ℓ) independent, mean-zero, unit-variance sub-Gaussian entries bounded by ∥Za,b ∥ψ2 ≤ K. Discussion of Assumption 2. These assumptions are based on the standard architecture of modern diffusion models. Item 1 (Lipschitz Mappings) is well-justified because deep networks are compositions of Lipschitz-continuous operations, a property required for stable training. Crucially, Item 2 (Sub-Gaussian Residuals) is an assumption applied only to the residual activations. The learned (ℓ) LoRA basis Bj is optimized to capture the dominant, highly-structured components of the concept’s features. The residual, therefore, represents the uncaptured “noise floor,” which can be modeled as a sub-Gaussian process. This is a mild assumption, as the sub-Gaussian class is broad and includes not only perfect Gaussians but also any bounded distribution (such as those produced by LayerNorm or Tanh) and mixtures thereof. The separable Kronecker covariance structure in (26) is a standard modeling choice in high-dimensional matrix-variate statistics [Dawid, 1981, Gupta and Nagar, 2000], adopted here for analytical tractability.

Proof of Theorem 1

We analyze the convergence of the SeqLoRA algorithm for the concept i. Proof. To prove the theorem, we first establish the following lemma. Lemma 4 (Gradient Lipschitz Continuity of Φi ). Under Assumptions 1, the reduced objective function Φi (B) ≜ Li (A∗ (B), B) where A∗ (B) = A0 − α∇A Li (A0 , B) and it has Lipschitz continuous gradients (i.e., is LΦ -smooth) with the Lipschitz constant: LΦ ≤ L(1 + αL)2 + αρM.

(27)

That is, for any B1 , B2 ∈ Rm×r , we have: ∥∇Φi (B1 ) − ∇Φi (B2 )∥F ≤ LΦ ∥B1 − B2 ∥F .

(28)

Proof. Let B1 and B2 be two arbitrary matrices. (k)

Using the definition of the lower-level update (8) for A0 = Ai Assumption 1:

and the L-smoothness from

∥A∗ (B1 ) − A∗ (B2 )∥F = ∥(A0 − α∇A Li (A0 , B1 )) − (A0 − α∇A Li (A0 , B2 ))∥F = α∥∇A Li (A0 , B1 ) − ∇A Li (A0 , B2 )∥F ≤ αL∥B1 − B2 ∥F . Using (15), the total derivative of Φi with respect to B is given by: ∇Φi (B) = ∇B Li (A∗ (B), B) −α ∇2AB Li (A0 , B) ∇A Li (A∗ (B), B) . | {z } | {z } T1

(29)

(30)

To bound the ∥∇Φi (B1 ) − ∇Φi (B2 )∥F , we use the triangle inequality. We first bound the term T1 . Using the Assumption 1 and substituting the bound from (29): ∥∇B Li (A∗ (B1 ), B1 ) − ∇B Li (A∗ (B2 ), B2 )∥F ≤ L ∥A∗ (B1 ) − A∗ (B2 )∥F + ∥B1 − B2 ∥F ≤ L αL∥B1 − B2 ∥F + ∥B1 − B2 ∥F = L(1 + αL)∥B1 − B2 ∥F . (31) To bound the term T2 in (30), let Hj = ∇2AB Li (A0 , Bj ) and Gj = ∇A Li (A∗ (Bj ), Bj ) for j ∈ {1, 2}. We need to bound ∥H1 [G1 ] − H2 [G2 ]∥F . Adding and subtracting the cross term H1 [G2 ], and applying the triangle inequality and sub-multiplicativity of operator norms: ∥H1 [G1 ] − H1 [G2 ] + H1 [G2 ] − H2 [G2 ]∥F ≤ ∥H1 [G1 − G2 ]∥F + ∥(H1 − H2 )[G2 ]∥F ≤ ∥H1 ∥2 ∥G1 − G2 ∥F + ∥H1 − H2 ∥2 ∥G2 ∥F . (32) We now bound each of the four components in the above inequality: by Assumption 1, we have ∥H1 ∥2 ≤ L and ∥G2 ∥F ≤ M . Using the ρ-Lipschitz property in Assumption 1 yields ∥H1 − H2 ∥2 ≤ ρ∥B1 − B2 ∥F Now, we use joint smoothness and (29), resulting in ∥G1 − G2 ∥F = ∥∇A Li (A∗ (B1 ), B1 ) − ∇A Li (A∗ (B2 ), B2 )∥F ≤ L ∥A∗ (B1 ) − A∗ (B2 )∥F + ∥B1 − B2 ∥F ≤ L(1 + αL)∥B1 − B2 ∥F .

(33)

Substituting these four bounds back into (32) yields: h i h i ∥H1 [G1 ] − H1 [G2 ] + H1 [G2 ] − H2 [G2 ]∥F ≤ L L(1 + αL)∥B1 − B2 ∥F + ρ∥B1 − B2 ∥F M = L2 (1 + αL) + ρM ∥B1 − B2 ∥F . (34)

Combining the bounds (31) and (34) gives us: h i ∥∇Φi (B1 ) − ∇Φi (B2 )∥F ≤ L(1 + αL) + α L2 (1 + αL) + ρM ∥B1 − B2 ∥F h i h i = L(1 + αL) + αL2 (1 + αL) + αρM ∥B1 − B2 ∥F = L(1 + αL)(1 + αL) + αρM ∥B1 − B2 ∥F = L(1 + αL)2 + αρM ∥B1 − B2 ∥F . (35) Thus, ∇Φi (B) is Lipschitz continuous with constant LΦ ≤ L(1 + αL)2 + αρM . This completes the proof of Lemma 4. Next, we show the rest of the results of Theorem 1. (k)

(k)

The tentative lower-level update is Ãi = Ai − α∇A Li (Ai , Bi ). Applying the Descent Lemma on the L-smooth loss Li with respect to Ai : D E L (k) (k) (k) (k) (k) (k) (k) (k) (k) (k) Li (Ãi , Bi ) ≤ Li (Ai , Bi ) + ∇Ai Li (Ai , Bi ), Ãi − Ai + ∥Ãi − Ai ∥2F . 2 (36) e (k) is the minimizer of (8) when Bi = B(k) , we have Since A i i (k)

(k)

Li (Ai , Bi ) + ∇Ai Li (Ai , Bi ), Ãi

(k)

− Ai

1 (k) (k) 2 (k) (k) Ãi − Ai F ≤ Li (Ai , Bi ), 2α (37)

Adding (36) into the above inequality gives us (k)

(k)

Li (Ãi , Bi ) ≤ Li (Ai , Bi ) + (

L 1 (k) (k) − )∥Ãi − Ai ∥2F . 2 2α

(38)

Algorithm 1 projects the gradient step of Φi onto Bi . This is exactly the minimizer of the proximal quadratic model in which δBi (B) is an indicator function 1 (k) (k+1) (k) (k) ∥B − Bi ∥2F + δBi (B) . (39) Bi = arg min ⟨∇Bi Φi (Bi ), B − Bi ⟩ + B 2β (k+1)

Since Bi

(k)

minimizes this model compared to the feasible point Bi , we have:

(k)

(k+1)

⟨∇B Φi (Bi ; Ai ), Bi

(k)

(k+1)

− Bi ⟩ ≤ δBi (Bi ) − δBi (Bi

)−

1 (k+1) (k) ∥B − Bi ∥2F . (40) 2β i

Applying the descent lemma to Φi (which is LΦ -smooth by Lemma 4) between iterates k and k + 1 results in (k+1)

Φi (Bi

(k)

(k+1)

) ≤ Φi (Bi ) + ⟨∇Bi Φi (Bi ), Bi

(k)

− Bi ⟩ +

LΦ (k+1) (k) ∥Bi − Bi ∥2F 2

(41)

Adding the above inequality to (40) yields: (k+1) (k+1) (k) (k) Φi (Bi ) + δBi (Bi ) ≤ Φi (Bi ) + δBi (Bi ) −

(k+1)

By definition (9), Φi (Bi ) = Li (Ai ing these and adding (38) into (42) yields:

(k+1)

, Bi

1 LΦ − 2β 2 (k)

(k+1)

∥Bi

(k)

− Bi ∥2F . (42) (k)

), and Φi (Bi ) = Li (Ãi , Bi ). Substitut-

1 L (k) (k) − ∥Ãi − Ai ∥2F 2α 2 1 LΦ (k+1) (k) − − ∥Bi − Bi ∥2F . 2β 2

(k+1) (k+1) (k) (k) Fi (Ai , Bi ) ≤ Fi (Ai , Bi ) −

(43)

where Fi (Ai , Bi ) = Li (Ai , Bi ) + δBi (Bi ). Because α < L1 and β < L1Φ , both coefficients are non-negative, resulting in a decreasing loss function. Substituting the tentative lower-level 19

(k)

(k+1)

update Ãi = Ai − α∇A Li (Ai , Bi ), using (14) and (17) upper-level update Bi = (k) (k) ⊥ PBi (Bi − β∇Bi Φi (Bi )) in the above inequality gives us α α2 L (k+1) (k+1) (k) (k) (k) (k) Fi (Ai , Bi ) ≤ Fi (Ai , Bi ) − − ∥∇Ai Li (Ai , Bi )∥2F 2 2 β LΦ β 2 (k) ⊥ − − ∥PB ∇Bi Φi (Bi )∥2F . (44) i 2 2 2 2 Maximizing α2 − α2L and β2 − LΦ2β over α ∈ (0, L1 ) and β ∈ (0, L1Φ ), respectively, gives 1 α∗ = 2L and β ∗ = 2L1Φ ≥ 2L(1+α∗ L)1 2 +2α∗ ρM = 9 L+1 ρM . 2

Summing (43) over k = 0 to ∞ reveals that the sum of the squared differences is bounded by the (0) initial suboptimality Fi − Fi∞ . Therefore, we obtain the limits: (k)

lim ∥Ãi

k→∞

(k)

− Ai ∥F = 0

(k+1)

lim ∥Bi

and

k→∞

(k)

− Bi ∥F = 0.

(45)

We now prove that the convergence of the algorithm implies the stationary conditions. First, for Ai , (k) (k) (k) (k) recall that Ai − Ãi = α∇A Li (Ai , Bi ). From (45), it trivially follows that: (k)

(k)

lim ∥∇A Li (Ai , Bi )∥F = 0,

(46)

k→∞

satisfying the critical point condition. Second, for Bi , using (39) the projected gradient mapping of the reduced objective vanishes, meaning: (k)

(k)

−∇B Φi (Bi ; Ai ) ∈ ∂δBi (Bi ) (k)

as k → ∞.

(47)

(k)

where ∂δBi (Bi ) is the sub-gradient of the indicator δBi (B) at Bi . We want to show that ∇B Φi converges to the partial gradient of the true loss ∇B Li . Using the chain rule from Lemma 4: (k) (k) (k) (k) (k) (k) (k) ∇B Φi (Bi ) = ∇B Li (Ãi , Bi ) − α∇2AB Li (Ai , Bi ) ∇A Li (Ãi , Bi ) . (48) (k)

(k)

We analyze the discrepancy D(k) = ∇B Φi (Bi ; Ai ) − ∇B Li (Ai , Bi ): (k)

(k)

∥D(k) ∥F ≤ ∥∇B Li (Ãi , Bi ) − ∇B Li (Ai , Bi )∥F (k)

(k)

(49)

(k)

− Ai ∥F . The cross-Hessian

+ α∥∇2AB Li ∥2 ∥∇A Li (Ãi , Bi )∥F . By the joint L-smoothness of Li , the first term is bounded by L∥Ãi norm is bounded by L. Furthermore, the gradient evaluates to: (k)

(k)

∥∇A Li (Ãi , Bi )∥F ≤ ∥∇A Li (Ai , Bi )∥F + L∥Ãi (k)

(k)

− Ai ∥F .

(50)

(k)

Since α∇A Li (Ai , Bi ) = Ai − Ãi , every component of the discrepancy is bounded by a (k) (k) (k) (k) constant multiple of ∥Ãi − Ai ∥F . Because ∥Ãi − Ai ∥F → 0 as established in (45), we conclude that: (k)

(k)

lim ∥D(k) ∥F = 0 =⇒ lim ∇B Φi (Bi ; Ai ) = lim ∇B Li (Ai , Bi ).

k→∞

(51)

Therefore, substituting this equivalence into (47), the algorithm perfectly satisfies the original composite stationarity condition: (k)

(k)

−∇B Li (Ai , Bi ) ∈ ∂δBi (Bi )

as k → ∞,

(52)

Concluding the proof that the sequence converges to a joint critical point of the main composite loss function Fi . 20

Proof of Theorem 2

The proof of Theorem 2 has two parts. Part 1 (Subsection C.1) establishes the high-probability (ℓ) bound (19): Lemma 5 bounds the per-layer crosstalk ∥∆j ∥F via Hanson–Wright, after Proposition 1 reduces it to the residual subspace using orthogonality; Lemma 6 propagates these local bounds through the L Lipschitz layers via a linear recursion; a union bound over ℓ ∈ [L] yields the networkwide statement. Part 2 (Subsection C.2) establishes basis optimality: Theorem 7 shows the top-r e (ℓ) minimizes residual energy among feasible rank-r eigenspace of the compressed covariance Σ j projectors, and Proposition 2 shows random frozen bases are strictly suboptimal except in the isotropic case. C.1

Analysis of Catastrophic Forgetting

To establish the end-to-end bound on catastrophic forgetting, we systematically decouple the local statistical interference from its global deterministic propagation through the deep network. We structure the proof into two complementary lemmas. First, we leverage the Hanson-Wright inequality to provide a high-probability statistical bound on the isolated activation crosstalk at any single layer. Second, the network-wide error propagation is analyzed, unrolling the Lipschitz-continuous inter-layer mappings to bound the total catastrophic forgetting as a weighted sum of these local perturbations. Finally, we synthesize these two results using a union bound across all L layers to complete the theorem. Lemma 5 (Hanson-Wright High-Probability Bound on Crosstalk). Under Assumption 2 item 2, T P (ℓ) (ℓ) (ℓ) let Cj = Ak (Bk )⊤ be the crosstalk operator for concept j and let the crosstalk be (ℓ)

k=j+1 (ℓ) (ℓ)

∆j = Cj Xj . There exists an absolute universal constant c > 0 such that for any ξ ∈ (0, 1), the crosstalk with probability at least 1 − ξ is bounded by q (ℓ) (ℓ) (ℓ) ∥∆j ∥F ≤ Tr(Ψj ) ∥Qj ∥2F + t(ℓ) , (53) where (ℓ)

r p 2 2 (ℓ) (ℓ) ⊤ (ℓ) (ℓ) (ℓ) 2 = max C1 K ∥Ψj ∥F ∥(Qj ) Qj ∥F log , C1 K 2 ∥Ψj ∥2 ∥Qj ∥22 log , (54) ξ ξ (ℓ)

(ℓ)

(ℓ),⊥ 1/2

C1 = 1/c, and Qj ≜ Cj (Σj

)

∈ Rnℓ ×mℓ .

Proof. Consider a model with L layers adapted for T concepts via SeqLoRA. At layer ℓ, the pre(ℓ) (ℓ) trained weight is W0 ∈ Rnℓ ×mℓ and the LoRA factors for concept k are Ak ∈ Rnℓ ×r and (ℓ) Bk ∈ Rmℓ ×r . In the sequential continual learning setting, the accumulated weight matrix after learning concept j at layer ℓ is: j X (ℓ) (ℓ) (ℓ) (ℓ) Wj = W0 + Ak (Bk )⊤ . (55) k=1

The composed output for concept j is: (ℓ)

(ℓ)

Oj (Xj ) = Wj Xj

(56)

while the composed output (with all T concepts) is: (ℓ)

(ℓ)

ÕT (Xj ) = WT Xj = Oj (Xj ) +

T X

(ℓ)

Ak (Bk )⊤ Xj .

(57)

(ℓ)

(58)

k=j+1

The total crosstalk at layer ℓ for concept j is: (ℓ)

(ℓ)

∆j ≜ ÕT (Xj ) − Oj (Xj ) = Cj Xj ∈ Rnℓ ×p , 21

where the crosstalk operator is: T X

(ℓ)

Cj ≜

(ℓ)

Ak (Bk )⊤ ∈ Rnℓ ×mℓ .

(59)

k=j+1

This captures the interference of the parameters added after concept j acting on concept j’s activations. We decompose the activation into its component within concept j’s basis and its residual: (ℓ)

(ℓ)

(ℓ),⊥

(ℓ)

Xj = PBj Xj + PBj Xj . (60) −1 (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) where PBj = Bj (Bj )⊤ Bj (Bj )⊤ is the projector onto col(Bj ) and its complement is (ℓ),⊥

PBj

(ℓ)

= Im(ℓ) − PBj . Since Bj is the learned LoRA basis for concept j, it captures the relevant (ℓ)

features of Xj . The first term carries most of the activation energy; the second term is the leaked energy outside the learned subspace. Proposition 1. The crosstalk depends only on the residual component: (ℓ)

(ℓ)

(ℓ),⊥

∆j = Cj Xj = Cj PBj (ℓ)

(ℓ)

(ℓ),⊥

Xj = Cj Xj

(61)

(ℓ)

That is, the signal component PBj Xj is invisible to the crosstalk. Proof. Substituting the decomposition (60): (ℓ)

(ℓ)

(ℓ),⊥

(ℓ)

Cj Xj = Cj PBj Xj + Cj PBj

(ℓ)

Xj .

(62)

We show the first term vanishes. Expanding: (ℓ)

T X

(ℓ)

C j PB j =

(ℓ)

Ak (Bk )⊤ PBj .

(63)

k=j+1 (ℓ)

(ℓ)

Now (Bk )⊤ PBj

(ℓ)

= (Bk )⊤ Bj (Gj )−1 (Bj )⊤ , where Gk

(ℓ)

= (Bk )⊤ Bk .

Since

(ℓ) (ℓ) (Bk )⊤ Bj = 0 for k ̸= j: (ℓ)

(ℓ)

Cj PBj = 0. (ℓ)

(ℓ)

(ℓ),⊥

Therefore ∆j = Cj PBj

(ℓ)

(ℓ),⊥

Xj = Cj Xj

(64)

. (ℓ)

This result implies that the entire activation energy within col(Bj ), regardless of its magnitude, does not contribute to the crosstalk, and the crosstalk sees only the residual (ℓ),⊥ (ℓ),⊥ The residual activation covariance xj = vec Xj is defined as follows n o (ℓ),⊥ (ℓ),⊥ (ℓ),⊥ ⊤ (ℓ) (ℓ),⊥ Cov(xj ) ≜ E xj (xj ) = Ψj ⊗ Σj ∈ Rmℓ p . (65) (ℓ)

It is important to note that since Bj captures most of the activation energy of concept j, the residual (ℓ),⊥

covariance Σj

may have a small energy (small eigenvalues).

Substituting the matrix sub-Gaussian process into the crosstalk (61), we obtain: (ℓ)

(ℓ)

(ℓ),⊥

∆j = Cj Xj (ℓ)

(ℓ)

(ℓ),⊥ 1/2

where Qj ≜ Cj (Σj quadratic form:

)

(ℓ)

(ℓ),⊥ 1/2

= Cj (Σj

)

(ℓ)

Z(ℓ) (Ψj )1/2 = Qj Z(ℓ) (Ψj )1/2 ,

(66)

∈ Rnℓ ×mℓ . The squared Frobenius norm of the crosstalk is a

(ℓ) (ℓ) (ℓ) (ℓ) (ℓ) z ≜ ∥∆j ∥2F = Tr (Qj Z(ℓ) (Ψj )1/2 )⊤ (Qj Z(ℓ) (Ψj )1/2 ) (ℓ) (ℓ) (ℓ) = Tr (Z(ℓ) )⊤ (Qj )⊤ Qj Z(ℓ) Ψj .

(67)

By using the standard matrix trace identity Tr(A⊤ BAC) = vec(A)⊤ (C ⊗ B) vec(A), we vectorize the independent sub-Gaussian matrix Z(ℓ) into z(ℓ) ∈ Rmℓ p . The trace perfectly rewrites as a standard vector quadratic form: z = (z(ℓ) )⊤ Hz(ℓ) , (68) (ℓ) (ℓ) ⊤ (ℓ) where H = Ψj ⊗ (Qj ) Qj . Taking the expectation of the variable z and using Tr(A ⊗ B) = Tr(A)Tr(B) [Horn and Johnson, 1991] ⊤

µz = E{(z(ℓ) ) Hz(ℓ) } = Tr(E{z(ℓ) (z(ℓ) )⊤ }H) = Tr(H) = ⊤ (ℓ) (ℓ) (ℓ) (ℓ) Tr(Ψj )Tr (Qj )(ℓ) Qj = Tr(Ψj )∥Qj ∥2F ,

(69)

where, based on the unit variance sub-Gaussian assumption, we have E{z(ℓ) (z(ℓ) )⊤ } = I. By the Hanson-Wright inequality for sub-Gaussian random vectors [Vershynin, 2026, Theorem 6.2.2], for any t > 0: t2 t P(|z − µz | > t) ≤ 2 exp −c min , , (70) K 4 ∥H∥2F K 2 ∥H∥2 where c > 0 is a universal constant. We now bound the norms of H. For the ℓ2 norm and using ∥A ⊗ B∥2 = ∥A∥2 ∥B∥2 [Horn and Johnson, 1991], we have: (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) ∥H∥2 = ∥Ψj ⊗ (Qj )⊤ Qj ∥2 = ∥Ψj ∥2 ∥(Qj )⊤ Qj ∥2 = ∥Ψj ∥2 ∥Qj ∥22 .

(71)

To compute the squared Frobenius norm ∥H∥2F , we use the property ∥A ⊗ B∥F = ∥A∥F ∥B∥F [Horn and Johnson, 1991]. Consequently, (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) ∥H∥F = ∥Ψj ⊗ (Qj )⊤ Qj ∥F = ∥Ψj ∥F ∥(Qj )⊤ Qj ∥F . (72) To derive the exact minimal t that guarantees a failure probability of at most ξ, we set the right-hand side equal to ξ: t2 t 2 1 min , (73) = log . K 4 ∥H∥2F K 2 ∥H∥2 c ξ For the minimum of two non-negative terms to equal a value x, both terms must be at least x. This yields two lower bounds for t: r 1 2 K 2 ∥H∥F t2 2 √ ≥ log =⇒ t≥ (74) log , 2 4 K ∥H∥F c ξ ξ c 2 K 2 ∥H∥2 2 1 log =⇒ t≥ log . (75) c ξ c ξ 2 To tightly satisfy both conditions, t must be exactly the maximum of the two bounds. Using (71) and (72) in the above inequalities, we have r p 2 2 (ℓ) (ℓ) ⊤ (ℓ) (ℓ) (ℓ) 2 (ℓ) 2 2 t = max C1 K ∥Ψj ∥F ∥(Qj ) Qj ∥F log , C1 K ∥Ψj ∥2 ∥Qj ∥2 log , (76) ξ ξ t

K 2 ∥H∥

≥

where C1 = 1/c. Using (69) and the above equation in the Hanson-Wright inequality results in the following crosstalk upper bound with probability 1 − ξ q (ℓ) (ℓ) (ℓ) ∥∆j ∥F ≤ Tr(Ψj ) ∥Qj ∥2F + t(ℓ) . (77) This concludes the proof of Lemma 5. Lemma 6 (End-to-End Catastrophic Forgetting). Under Assumptions 2 item 1, let Γℓ ≜ QL (ℓ′ ) ′ ℓ′ =ℓ+1 γℓ ∥WT ∥2 be the downstream amplification factor at layer ℓ. For any ξ ∈ (0, 1), the end-to-end catastrophic forgetting satisfies, with probability at least 1 − ξ: Lj (θT ) − Lj (θj ) ≤ Lo

L X

(ℓ)

Γℓ ∥∆j ∥F ,

(78)

ℓ=1 (ℓ)

(ℓ)

L where θT = {WT }L ℓ=1 and θj = {Wj }ℓ=1 are the full model after learning concept j and T , respectively.

(ℓ)

Proof. Let Oj (Xj ) = Wj Xj

(ℓ)

and ÕT (X̃j ) = WT X̃j

denote the layer-ℓ inputs under

(ℓ) (ℓ) model’s parameters Wj and the composed model’s parameters T concepts (WT ), respectively. (ℓ) (ℓ) Defining the layer-wise perturbation as eℓ ≜ ∥Oj (Xj ) − ÕT (X̃j )∥F , we have e0 = 0 at the

network input. (ℓ+1)

(ℓ+1)

(ℓ)

For ℓ ≥ 1, the inputs to the LoRA layer are Xj The output difference expands as:

= ϕℓ (Oj (Xj )) and X̂j

(ℓ)

= ϕℓ (ÕT (X̃j )).

(ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) ÕT (X̃j ) − Oj (Xj ) = WT X̃j − Wj Xj = WT X̃j − Xj + Cj Xj . (79) Taking the Frobenius norm, applying the triangle inequality, and utilizing the γℓ -Lipschitz property of the inter-layer mapping ϕℓ yields the linear recursion: (ℓ)

(ℓ)

eℓ ≤ ∥WT ∥2 ∥X̃j − Xj ∥F + ∥Cj Xj ∥F ≤ αℓ eℓ−1 + ∥∆j ∥F ,

(80)

(ℓ) (ℓ) (ℓ) (ℓ) where we define αℓ ≜ γℓ ∥WT ∥2 as the effective per-layer amplification and ∆j ≜ Cj Xj as

the injected crosstalk. Unrolling the linear recursion (80) entirely from ℓ = 1 to L, and recognizing that the cumulative QL amplification from layer ℓ to L is exactly Γℓ = ℓ′ =ℓ+1 αℓ′ , we have the following bound eL ≤

L X

(ℓ)

Γℓ ∥∆j ∥F .

(81)

ℓ=1

Finally, by the Lo -Lipschitz continuity of the downstream mapping fj , the end-to-end catastrophic forgetting is bounded directly by the final-layer perturbation: (L)

(L)

Lj (θT ) − Lj (θj ) = fj (ÕT (X̂j )) − fj (Oj (X̂j )) ≤ Lo ∥ÕT (X̂j ) − Oj (X̂j )∥F = Lo eL . (82) Using (81) in the above inequality results in L X (ℓ) Lj (θT ) − Lj (θj ) ≤ Lo Γℓ ∥∆j ∥F . (83) ℓ=1

We now synthesize the results of Lemma 5 and Lemma 6 to complete the proof of the bound (19) in Theorem 2. To ensure a global probabilistic guarantee across the entire network, we apply the statistical bound from Lemma 5 with an adjusted layer-wise failure probability of ξ/L. Substituting (ℓ) this high-probability upper bound (53) for the crosstalk norm ∥∆j ∥F into the end-to-end deterministic bound (78), we apply Boole’s inequality (the union bound) over all ℓ ∈ [L]. This yields to the inequality (19). This concludes the proof of catastrophic forgetting bound (19) in Theorem 2. (ℓ),⊥

(ℓ),⊥

(ℓ)

In Theorem 2, due to the residual model Xj = (Σj )1/2 Z(ℓ) (Ψj )1/2 in Assumption 2, the trace of the covariance matrix of residual is given by (ℓ),⊥ ⊤ (ℓ),⊥ (ℓ) (ℓ),⊥ (ℓ) (ℓ) Tr E{(Xj ) Xj } = Tr E{(Ψj )1/2 (Z(ℓ) )⊤ Σj Z (Ψj )1/2 } (ℓ) (ℓ),⊥ (ℓ) = Tr Ψj E{(Z(ℓ) )⊤ Σj Z } (ℓ) (ℓ),⊥ = Tr Ψj · Tr(Σj )Ip (ℓ),⊥

= Tr(Σj (ℓ),⊥

(ℓ),⊥

(ℓ)

)Tr(Ψj ).

(ℓ)

Also, we know Xj = PBj Xj , Σj = E{Xj (Xj )⊤ } and using the above equation results in (ℓ),⊥ (ℓ),⊥ ⊤ (ℓ),⊥ (ℓ) (ℓ),⊥ (ℓ) Tr(E{Xj (Xj ) }) = Tr(PBj Σj ) = Tr(Σj )Tr(Ψj ). (ℓ),⊥

(ℓ)

The term Tr(Σj )Tr(Ψj ) governs the expected crosstalk energy within the end-to-end bound in (19), identifying the residual covariance mass as the primary driver of catastrophic forgetting. This leads to a fundamental optimization question: among all feasible orthogonal rank-r subspaces, which one minimizes this residual energy? In the next subsection, we address this question, which proves the second part of Theorem 2 regarding minimizing the energy of interference. 24

C.2

Optimal basis selection inside the free complement

Let (ℓ)

(ℓ)

−1 (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) P<j ≜ Bint (Bint )⊤ Bint (Bint )⊤ ,

(ℓ)

Bint ≜ [B1 , . . . , Bj−1 ],

(84)

and define the currently available orthogonal complement (ℓ)

(ℓ)

Pfree,j ≜ I − P<j .

(85)

A rank-r projector P (ℓ) is feasible for concept j if P (ℓ) = (P (ℓ) )⊤ = (P (ℓ) )2 ,

(ℓ)

rank(P (ℓ) ) = r,

range(P (ℓ) ) ⊆ range(Pfree,j ).

(86)

Equivalently, P (ℓ) is any rank-r subspace that obeys the orthogonality constraints with all previously learned concepts. For any projector P (ℓ) , write P (ℓ),⊥ ≜ I − P (ℓ) . Define the compressed covariance inside the currently free subspace by e (ℓ) ≜ P (ℓ) Σ(ℓ) P (ℓ) , Σ j free,j j free,j

(87)

(ℓ)

e by and let dfree,j ≜ rank(Pfree,j ). Denote the eigenvalues of Σ j e (ℓ) ) ≥ λ2 (Σ e (ℓ) ) ≥ · · · ≥ λd e (ℓ) ) ≥ 0. λ 1 (Σ (Σ j j j free,j

(88)

Theorem 7 (Optimal feasible basis captures maximal task energy). Assume r ≤ dfree,j . Then for any feasible rank-r projector P (ℓ) , (ℓ) (ℓ) (ℓ) (ℓ) e (ℓ) . Tr(P (ℓ),⊥ Σj ) = Tr (I − Pfree,j )Σj + Tr (Pfree,j − P (ℓ) )Σ (89) j (ℓ)

Hence minimizing the residual energy Tr(P (ℓ),⊥ Σj ) over all feasible rank-r projectors is equivalent (ℓ) ⋆

(ℓ)

e ). The minimizer is the projector P to maximizing the captured energy Tr(P (ℓ) Σ j j

onto the top-r

e (ℓ) , and the minimum residual energy is eigenspace of Σ j free,j dX (ℓ) ⋆ (ℓ) (ℓ) (ℓ) e (ℓ) ). Tr (Pj )⊥ Σj = Tr (I − Pfree,j )Σj + λq (Σ j

(90)

q=r+1 (ℓ)

Proof. Because range(P (ℓ) ) ⊆ range(Pfree,j ), we have (ℓ)

(ℓ)

Pfree,j P (ℓ) = P (ℓ) Pfree,j = P (ℓ) . Therefore

(ℓ)

(91)

I − P (ℓ) = (I − Pfree,j ) + (Pfree,j − P (ℓ) ),

(92)

(ℓ) (ℓ) (ℓ) (ℓ) (ℓ) Tr(P (ℓ),⊥ Σj ) = Tr (I − Pfree,j )Σj + Tr (Pfree,j − P)Σj .

(93)

and thus

(ℓ)

Since Pfree,j − P acts entirely inside the free complement, (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) e (ℓ) . Tr (Pfree,j − P)Σj = Tr (Pfree,j − P (ℓ) )Pfree,j Σj Pfree,j = Tr (Pfree,j − P)Σ j (94) This proves the decomposition. Next, observe that (ℓ) e (ℓ) = Tr(Σ e (ℓ) ) − Tr(P (ℓ) Σ e (ℓ) ). Tr (Pfree,j − P (ℓ) )Σ j j j 25

(95)

(ℓ) e (ℓ) ) over all feasible Therefore minimizing Tr(P (ℓ),⊥ Σj ) is equivalent to maximizing Tr(P (ℓ) Σ j rank-r projectors. By the variational characterization of eigenvalues, the maximum is (ℓ)

e )= max Tr(P (ℓ) Σ j P (ℓ)

(ℓ) ⋆

attained when P (ℓ) = Pj yields

r X

(ℓ)

e ), λq (Σ j

(96)

q=1

e (ℓ) . Substituting back is the projector onto the top-r eigenspace of Σ j

r X (ℓ) ⋆ (ℓ) (ℓ) (ℓ) e (ℓ) ) − e (ℓ) ) Tr (Pj )⊥ Σj = Tr (I − Pfree,j )Σj + Tr(Σ λq (Σ j j

(97)

q=1 free,j dX (ℓ) (ℓ) e (ℓ) ). = Tr (I − Pfree,j )Σj + λ q (Σ j

(98)

q=r+1

Theorem 7 identifies the best possible basis for continual-learning: among all subspaces compatible with the orthogonality constraints, the optimal one is the dominant task covariance subspace inside (ℓ) the currently available complement. This immediately explains why learning Bj is beneficial: a e (ℓ) . frozen basis cannot adapt to the spectral structure of Σ j The next result makes this comparison explicit for random frozen bases, which are a natural abstraction of methods that choose an orthogonal basis a priori and then keep it fixed. Proposition 2 (Random frozen bases are optimal only in isotropic free complements). Let Prand be (ℓ) a Haar-random rank-r projector inside range(Pfree,j ). Then E[Prand ] =

r dfree,j

(ℓ)

Pfree,j ,

(ℓ)

e )} = E{Tr(Prand Σ j

(ℓ)

dfree,j

e ), Tr(Σ j

(99)

and therefore (ℓ) (ℓ) (ℓ) ⊥ E{Tr(Prand Σj )} = Tr (I − Pfree,j )Σj + 1 −

r dfree,j

(ℓ)

e ). Tr(Σ j

(100)

Moreover, r X (ℓ) (ℓ) ⋆ (ℓ) ⊥ e (ℓ) ) − E{Tr(Prand Σj )} − Tr (Pj )⊥ Σj = λq (Σ j q=1

r dfree,j

dfree,j

(ℓ)

e ) ≥ 0. λq (Σ j

(101)

q=1

Thus, except in degenerate isotropic or capacity-saturated cases, a learned feasible basis strictly improves upon a frozen random feasible basis in expectation. Proof. By rotational invariance of the Haar distribution on the dfree,j -dimensional free subspace, the (ℓ) expectation E[Prand ] must commute with every orthogonal transformation acting on range(Pfree,j ). (ℓ)

Hence it must be a scalar multiple of Pfree,j , say (ℓ)

E[Prand ] = αPfree,j .

(102)

Taking traces gives r = Tr(E[Prand ]) = α dfree,j , so α = r/dfree,j , proving the first identity.

(103)

e (ℓ) = P (ℓ) Σ e (ℓ) Using Σ j free,j j , e (ℓ) )} = Tr E[Prand ]Σ e (ℓ) = E{Tr(Prand Σ j j 26

r e (ℓ) ). Tr(Σ j dfree,j

(104)

(ℓ)

⊥ The formula for E[Tr(Prand Σj )] then follows directly from Theorem 7.

Finally, since the average of the top r eigenvalues is at least the average over all dfree,j eigenvalues, dfree,j r X (ℓ) 1X 1 e e (ℓ) ), λq (Σj ) ≥ λq (Σ j r q=1 dfree,j q=1

(105)

which implies r X

e (ℓ) ) − λq (Σ j

q=1

dfree,j

e (ℓ) ) ≥ 0. λq (Σ j

(106)

q=1

This proves the claim. Combining the results in Section C.1 and Section C.2 proves Theorem 2. Remark 2 (Relation to SeqLoRA). Theorem 7 should not be interpreted as saying that SeqLoRA e (ℓ) . Rather, it identifies the population quantity that matters explicitly computes the top eigenspace of Σ j for continual-learning stability under the orthogonality constraint. In the full nonlinear diffusion model, SeqLoRA seeks a favorable subspace implicitly by optimizing the end-to-end denoising loss (ℓ) (ℓ) (ℓ) over both Aj and Bj , instead of freezing Bj a priori. Theorem 7 provides a simple learning-theoretic justification for SeqLoRA. Orthogonality ensures that future concepts can only interfere with concept j through the residual covariance mass outside (ℓ) col(Bj ), while learning Bj from data minimizes exactly this exposed residual energy within the feasible complement. Proposition 2 further shows that frozen random bases are statistically optimal only when the task covariance is essentially isotropic inside the free complement. In this precise sense, optimizing both LoRA factors yields a strictly better continual-learning bias than fixing the basis in advance.

Decomposition of Catastrophic Forgetting: Mean Interference vs. Stochastic Deviation

The end-to-end catastrophic forgetting bound derived in Theorem 2 reveals a multi-scale structure. (ℓ) At each of the L adapted layers, the locally injected crosstalk ∥∆j ∥2F decomposes into two distinct (ℓ)

sources: the expected systematic interference µz and the stochastic tail deviation t(ℓ) . These local statistical fluctuations are then propagated and aggregated through the downstream network layers, controlled by the amplification factors Γℓ , to produce the final global forgetting. We now analyze the physical meaning of each component and its role in the total network-wide interference. (ℓ)

(ℓ)

Systematic Interference (µz ). The mean crosstalk µz = Tr(Ψj )∥Qj ∥2F factorizes into two (ℓ)

(ℓ)

(ℓ),⊥

physical quantities. The first, ∥Qj ∥2F = Tr(Cj Σj

), is the feature-level interference energy. (ℓ),⊥

(ℓ)

Crucially, it depends exclusively on the residual covariance Σj , not the full covariance Σj . This is a direct consequence of SeqLoRA’s orthogonality constraint, which perfectly annihilates the (ℓ) (ℓ) (ℓ) primary signal component (Cj PBj = 0). The better the LoRA basis Bj captures concept j’s (ℓ),⊥

features, the smaller the residual Σj

becomes, driving feature interference toward small values.

(ℓ) The second term, Tr(Ψj ), represents the total token sequence energy.

Together, orthogonality guarantees that the expected forgetting is confined entirely to the small, uncaptured residual subspace. (ℓ) Additionally, because the activation Xj is itself the output of a network optimized for concept j, the LoRA bases across all layers work in concert to capture the concept’s energy; any features (ℓ−1) (ℓ) (ℓ),⊥ not absorbed by Bj can be captured by Bj , ensuring the per-layer residual covariance Σj remains uniformly small throughout the network depth. 27

Stochastic Deviation (t(ℓ) ). The deviation t(ℓ) captures the random fluctuation of the crosstalk around its mean due to the realization of the sub-Gaussian noise. It is bounded by t(ℓ) = max(T1 , T2 ), where: r p 2 (ℓ) (ℓ) ⊤ (ℓ) 2 T1 = K C1 ∥Ψj ∥F ∥(Qj ) Qj ∥F log (Sub-Gaussian Term) ξ 2 (ℓ) (ℓ) (ℓ) T2 = K 2 C1 ∥Ψj ∥2 ∥(Qj )⊤ Qj ∥2 log (Sub-Exponential Term). ξ Defining the effective rank as reff (A) ≜ ∥A∥F /∥A∥2 , we determine the dominant regime by setting T1 ≥ T2 . Solving this inequality yields the threshold condition for the sub-Gaussian term to dominate: r 2 (ℓ) (ℓ) ⊤ (ℓ) reff (Ψj ) · reff (Qj ) Qj ≥ C1 log . (107) ξ The heavy-tailed sub-exponential regime only triggers if this strict inequality is reversed. Regime Analysis: Diffuse vs. Spiky Interference. The threshold condition in Equation (107) reveals that catastrophic forgetting behaves differently depending on the magnitude of the joint effective rank. This transition is governed by whether the residual geometry is diffuse or spiky: • The Sub-Gaussian Regime (Diffuse Interference): When the joint effective rank is large (ℓ) (T1 ≥ T2 ), the sub-Gaussian term dominates. Because the learned LoRA basis Bj successfully absorbs the dominant principal components of the activations, the residual (ℓ),⊥ covariance Σj retains only a flat, unstructured spectral tail. The interference is diffuse, spread evenly across many dimensions, acting as benign noise that scales gracefully as p log(2/ξ). • The Sub-Exponential Regime (Spiky Interference): Conversely, the heavier-tailed subexponential regime (T2 > T1 ) only triggers if the joint effective rank collapses (≈ 1). This occurs if the residual covariance still contains a massive, uncaptured principal direction, making the interference highly spiky and concentrated. In this state, forgetting is dictated by worst-case fluctuations along that single dominant axis, scaling poorly as log(2/ξ). The Dual Role of Orthogonality. SeqLoRA’s inter-concept orthogonality (B⊤ i Bj = 0) serves a dual mathematical purpose to protect the network. First, it shrinks the systematic mean µz by restricting interference exclusively to the residual subspace. Second, by projecting out the dominant principal (ℓ) components captured by Bj , it exposes the naturally diffuse structure of the residual spectrum, ensuring a large joint effective rank that keeps the crosstalk in the well-concentrated sub-Gaussian regime.

Population Risk Analysis in a Stylized Linear Setting

Theorem 1 establishes the optimization dynamics of SeqLoRA, and Theorem 2 provides a highprobability bound on end-to-end catastrophic forgetting in deep, non-linear networks. To complement this result, we now derive an exact, closed-form decomposition of catastrophic forgetting in a stylized linear setting with a single adapted layer. This is not intended to model the full diffusion network; rather, by removing non-linearities, it isolates the precise mechanism through which orthogonality and learned-basis adaptation interact, and makes explicit the residual-energy quantity that controls forgetting. The two analyses agree in their conclusion: the high-probability bound in Theorem 2 ⊥ and the exact decomposition below both identify Tr(PB Σj ) as the driver of forgetting, providing j convergent evidence from two distinct analytical viewpoints. Stylized population model. Consider a single linear map with fixed input features x ∈ Rm and output y ∈ Rn . For concept j, let Dj be a population distribution over (x, y), and define the squared population risk Lj (W) ≜ E(x,y)∼Dj{∥y − Wx∥22 }, 28

Σj ≜ E(x,y)∼Dj{xx⊤ }.

(108)

As in LoRA, after learning t concepts the adapted matrix is Wt = W0 +

t X

As B⊤ s .

(109)

As B⊤ s ,

(110)

s=1

Fix a concept j. After learning concept j, the model is Wj = W0 +

j X s=1

while after all later concepts have been learned, WT = Wj + Cj ,

T X

Cj ≜

As B⊤ s .

(111)

s=j+1

Let −1 ⊤ PBj ≜ Bj (B⊤ Bj , j Bj )

⊥ PB ≜ I − P Bj , j

(112)

denote the orthogonal projector onto the learned right-subspace of concept j and its orthogonal complement. In this simplified view, the role of the right LoRA factor Bj is summarized by PBj , while the cumulative effect of all future concepts is summarized by the crosstalk operator Cj . The first result shows that, under orthogonality, forgetting is driven only by the part of the concept covariance that lies outside the learned basis. Theorem 8 (Residual-subspace decomposition of forgetting). Assume that the later bases satisfy B⊤ s Bj = 0,

∀s > j.

Then the population forgetting on concept j admits the exact decomposition ⊥ ⊥ ⊤ Σ P C Lj (WT ) − Lj (Wj ) = Tr Cj PB + ⟨∇Lj (Wj ), Cj ⟩ . j Bj j j

(113)

(114)

Consequently, if ∥∇Lj (Wj )∥F ≤ εj , then ⊥ Lj (WT ) − Lj (Wj ) ≤ ∥Cj ∥22 Tr PB Σ + εj ∥Cj ∥F . j j In particular, if Wj is a stationary point of Lj , then ⊥ ⊥ ⊤ 2 ⊥ Lj (WT ) − Lj (Wj ) = Tr Cj PB Σ P C ≤ ∥C ∥ Tr P Σ . j j j Bj j 2 Bj j

(115)

(116)

Proof. Because B⊤ s Bj = 0 for every s > j, we also have B⊤ s PBj = 0,

∀s > j,

(117)

As B⊤ s PBj = 0,

(118)

since range(PBj ) = col(Bj ). Therefore Cj PBj =

T X s=j+1

⊥ which implies Cj = Cj PB . j

Now expand the population risk difference: Lj (WT ) − Lj (Wj ) = E{∥y − (Wj + Cj )x∥22 − ∥y − Wj x∥22 } = E{∥Cj x∥22 } + 2 E{(Wj x − y)⊤ Cj x}.

(119) (120)

Since ∇Lj (W) = 2 E{(Wx − y)x⊤ }, 29

(121)

⊥ the second term is exactly ⟨∇Lj (Wj ), Cj ⟩. For the first term, using Cj = Cj PB , j

E{∥Cj x∥22 } = E{x⊤ C⊤ j Cj x} = Tr Cj Σj C⊤ j ⊥ ⊥ . = Tr Cj PB Σj PB C⊤ j j j

(122) (123) (124)

This proves the identity. For the upper bound, note that for any positive semidefinite matrix M, 2 Tr(Cj MC⊤ j ) ≤ ∥Cj ∥2 Tr(M). ⊥ ⊥ Applying this with M = PB Σj PB gives j j ⊥ ⊥ ⊤ 2 ⊥ Tr Cj PB Σ P C Σ . ≤ ∥C ∥ Tr P j j j B j 2 B j j j

(125)

(126)

Finally, by Cauchy–Schwarz, |⟨∇Lj (Wj ), Cj ⟩| ≤ ∥∇Lj (Wj )∥F ∥Cj ∥F ≤ εj ∥Cj ∥F .

(127)

The stationary case follows by setting εj = 0. This result is perfectly consistent with Theorem 2, which derives the catastrophic forgetting by (ℓ),⊥ (ℓ) (ℓ),⊥ (ℓ) Tr(PBj Σj ) = Tr(Σj )Tr(Ψj ). The alignment between our deep-network concentration bounds and this population-risk analysis underscores that minimizing this residual energy is the key to sequential stability.

Evaluation Metrics

We use a comprehensive set of metrics to evaluate different aspects of the generated images: • CLIP-I / CLIP-T [Radford et al., 2021, Ruiz et al., 2023]: Measure image-to-image and text-toimage alignment, ensuring the generated images match the target concepts and prompts. • DINO / DINOv2 / DINOv3 [Caron et al., 2021, Oquab et al., 2023, Siméoni et al., 2025]: Selfsupervised features used to assess identity preservation of the customized concepts. • HPSv2/HPSv3 [Wu et al., 2023, Ma et al., 2025]: Human Preference Scores to evaluate the overall visual quality and appeal of the generated images. • DreamSim [Fu et al., 2023]: Measures holistic visual similarity (layout, pose, and semantic content) to reference images.

Detailed Evaluation Methodology

This section details the aggregation protocol used in our quantitative evaluations. Each per-concept score is first computed by averaging the relevant metric over 16 generated images obtained with different random seeds. Subsequent aggregation differs between figures (which report average performance only) and tables (which report mean ± standard error). We describe each in turn. G.1

Mean Performance in Figures 3a and 3b

Calculation of Overall Performance (Figure 3a). Figure 3a measures the scalability of the methods by reporting the final model performance across different total concept counts T ∈ {8, 16, . . . , 101}. For a given data point at x = T , we evaluate the model only after it has finished training on all T concepts. We query this final composed model to generate 16 images for every single one of the T concepts, yielding a total of 16 × T images. The metric (e.g., DINO similarity) is computed individually for all 16 × T images against their respective reference images. The value plotted on the y-axis is the unweighted arithmetic mean of these 16 × T scores. For example, the data point at T = 101 represents the grand mean of 1,616 distinct generations, providing a highly robust measure of the final system’s aggregate identity preservation. Calculation of the Forgetting Trajectory (Figure 3b). Figure 3b tracks the dynamic “health” of the model during a single 32-concept sequential training run. A data point at step k on the x-axis (where k ∈ {1, . . . , 32}) represents the state of the model immediately after learning the k-th concept. At this specific step, we evaluate the model’s retention of all concepts it has seen so far (concepts 1 through k). The model generates 16 images for each of these k concepts, yielding 16 × k images. The value plotted on the y-axis is the arithmetic mean of the scores across all 16 × k images. By continually expanding the evaluation pool to include all previously learned concepts at each step, a downward slope in this trajectory rigorously quantifies the average rate of catastrophic forgetting as the parameter space becomes increasingly constrained. G.2

Error Bars in Tables 1-4

For every (method, concept count, metric) cell in our quantitative tables we report mean ± standard error of the mean (SEM). For a given configuration with N concepts, we first compute one perconcept score by averaging the metric over the 16 generated images for that concept. The reported mean is then √ the arithmetic average of these N per-concept scores, and the SEM is computed as SEM = σ/ N , where σ is the unbiased standard deviation across the N per-concept scores. SEMs therefore capture concept-level variability, decreasing naturally as more concepts are learned, and provide an interpretable measure of the precision of each reported mean.

Supplementary Qualitative Results

To provide a comprehensive evaluation, we present qualitative comparisons for all 32 concepts used in our experiments. The concepts are ordered according to their training sequence. Figures 4, 5, and 6 31

show the results for the first 12 concepts, the next 12 concepts, and the final 8 concepts, respectively. Consistent with the main paper, we show the input images and generated samples for SeqLoRA (Bilevel), Continual Alternating, LoRACLR, Mix-of-Show, and Orthogonal Adaptation. All methods use the same set of seeds for fair comparison.

Multi-Concept Regional Generation

Beyond the per-concept comparisons in Section H, we evaluate how each method composes multiple learned concepts into a single image under regional sketch and keypose conditioning, following the regional generation protocol of Gu et al. [2023]. For every method we use the same random seed and the same regional control inputs (sketch, keypose, and per-region bounding boxes), so any visual differences are attributable to the multi-concept fusion mechanism rather than to randomness or spatial conditioning. Figure 7 shows two representative compositions: a three-concept scene combining two chairs, a table, and a vase, and a four-concept scene that adds a dog to the same setting. Across both compositions, SeqLoRA preserves per-region concept identity (chair upholstery, vase silhouette, dog appearance) while the parallel-fusion baselines (Mix-of-Show, LoRACLR, Orthogonal Adaptation) and the alternating-minimization variant (Continual Alternating) exhibit visible attribute leakage and identity collapse across regions.

Concepts: <chair>, <vase>, <table>

SeqLoRA (Ours)

Continual Alt.

LoRACLR

Mix-of-Show

Orthogonal Adapt.

(a) Three concepts. Prompt: “two chairs, a table, and a vase, in a living room”.

Concepts: <chair>, <vase>, <dogA>, <table>

SeqLoRA (Ours)

Continual Alt.

LoRACLR

Mix-of-Show

Orthogonal Adapt.

(b) Four concepts. Prompt: “a chair, a vase, a dog and a table, in a living room”.

Figure 7: Multi-concept regional generation: qualitative comparison. Each subfigure shows, in the top-left cell, the input concept reference images, followed by five generated outputs (one per method) arranged in a 3 × 2 grid. All methods use the same random seed and the same regional sketch/keypose conditioning, so visual differences reflect the underlying multi-concept fusion mechanism rather than randomness or spatial conditioning. SeqLoRA preserves per-region concept identity (e.g., the chair upholstery, vase silhouette, and dog appearance) more faithfully than the baselines, which exhibit attribute leakage and identity collapse across regions. 32

SeqLoRA (Bilevel)

Continual Alternating

LoRACLR

Mix-of-Show

Orthogonal Adaptation

transportcar10

furniturechair1

decoritemsvase2

thingskeychain1

wearableglasses

wearablejacket2

transporttank

thingscup2

wearablesunglasses2

scenecastle

actionfigure1

scenebarn

Input Images

Figure 4: Supplementary qualitative comparison for concepts 1-12 (ordered by training sequence).

Input Images

Continual Alternating

LoRACLR

Mix-of-Show

Orthogonal Adaptation

transportbike

transportcar3

transportcar4

plushiecow

thingscup3

SeqLoRA (Bilevel)

plushiepanda

luggagebackpack1

wearablesunglasses1

luggagepurse1

transportcar5

toyunicorn

transportcar2

No Img

Figure 5: Supplementary qualitative comparison for concepts 13-24 (ordered by training sequence).

SeqLoRA (Bilevel)

Continual Alternating

LoRACLR

Mix-of-Show

Orthogonal Adaptation

plushiepenguin

thingscorkscrew

thingsbottle1

petdog1

instrument1

flower2

scenewaterfall

plushielobster

Input Images

Figure 6: Supplementary qualitative comparison for concepts 25-32 (ordered by training sequence).

Step 1

Step 4

Step 8

Step 12

Step 16

Step 20

Step 24

Step 28

Step 32

Not Learned

furniturechair1

Not Learned

transportcar10

Not Learned

thingscup3

Not Learned

plushiecow

Not Learned

decoritemsvase2 thingskeychain1 wearableglasses wearablejacket2

transporttank

Not Learned

transportcar4

Not Learned

transportcar3

thingscup2 wearablesunglasses2 scenecastle

actionfigure1

scenebarn

Input Image

Figure 8: Evolution of generated images for concepts 1–16 across different training steps for SeqLoRA. We show the input image followed by generated samples at steps 1, 4, 8, 12, 16, 20, 24, 28, and 32. The same random seed and basic prompt template are used across steps for each concept to ensure visual consistency.

Step 1

Step 4

Step 8

Step 12

Step 16

Not Learned

transportcar2

Not Learned

toyunicorn

Not Learned

transportcar5

Not Learned

plushiepanda

Not Learned

plushielobster

Not Learned

flower2

Not Learned

instrument1

Not Learned

petdog1

Not Learned

thingsbottle1

Not Learned

thingscorkscrew

Not Learned

plushiepenguin

Not Learned

scenewaterfall

luggagebackpack1wearablesunglasses1 luggagepurse1

transportbike

Input Image

Step 20

Step 24

Step 28

Step 32

Figure 9: Evolution of generated images for concepts 17–32 across different training steps for SeqLoRA. We show the input image followed by generated samples at steps 1, 4, 8, 12, 16, 20, 24, 28, and 32. The same random seed and basic prompt template are used across steps for each concept to ensure visual consistency.

Table 2: Summary of forgetting results for 32 concepts across selected steps. We show the average performance across all concepts for SeqLoRA (SeqL) and Continual Alternating (Alt). Step 1

Metric

SeqL

Alt

Step 8 SeqL

Step 16 Alt

SeqL

Step 24 Alt

SeqL

Step 32 Alt

SeqL

Alt

CLIP-I 0.739 0.754 0.694 ± 0.014 0.693 ± 0.012 0.692 ± 0.011 0.682 ± 0.012 0.691 ± 0.009 0.688 ± 0.009 0.687 ± 0.009 0.683 ± 0.008 CLIP-T 0.286 0.280 0.289 ± 0.005 0.289 ± 0.006 0.284 ± 0.004 0.285 ± 0.004 0.288 ± 0.004 0.288 ± 0.004 0.290 ± 0.003 0.290 ± 0.004 DINO 0.627 0.656 0.462 ± 0.043 0.469 ± 0.041 0.473 ± 0.026 0.461 ± 0.028 0.465 ± 0.025 0.463 ± 0.025 0.452 ± 0.022 0.450 ± 0.021 DreamSim 0.600 0.624 0.475 ± 0.035 0.473 ± 0.034 0.475 ± 0.020 0.461 ± 0.021 0.470 ± 0.019 0.465 ± 0.018 0.469 ± 0.016 0.466 ± 0.015

Supplementary Forgetting Results

To further analyze the forgetting behavior, we provide the detailed numerical results for all 32 concepts across selected steps (1, 8, 16, 24, 32) for SeqLoRA. Table 2 presents the performance for four key metrics: CLIP-I, CLIP-T, DINO, and DreamSim. The concepts are ordered by their training sequence. We also provide extended visualizations of forgetting evolution for all 32 concepts in Figures 8 and 9.

Supplementary Overall Results

We also provide the detailed numerical results for the overall comparison across all metrics and different numbers of concepts (8 to 101). Table 3 presents the results for SeqLoRA, Continual Alternating, Mix-of-Show, Orthogonal Adaptation, and LoRACLR. Table 3: Supplementary overall comparison results for all metrics across different number of concepts. 80

101

CLIP-I

CLIP-T

SeqLoRA (Bilevel) 0.287 ± 0.005 0.282 ± 0.004 0.284 ± 0.004 0.286 ± 0.003 0.278 ± 0.003 0.279 ± 0.003 0.280 ± 0.002 0.279 ± 0.002 0.277 ± 0.002 Continual Alternating 0.287 ± 0.006 0.288 ± 0.004 0.286 ± 0.004 0.287 ± 0.004 0.277 ± 0.003 0.279 ± 0.003 0.278 ± 0.002 0.279 ± 0.002 0.278 ± 0.002 Mix-of-Show 0.278 ± 0.004 0.273 ± 0.003 0.274 ± 0.003 0.276 ± 0.003 0.277 ± 0.003 0.279 ± 0.002 Orthogonal Adaptation 0.274 ± 0.004 0.274 ± 0.003 0.275 ± 0.003 0.277 ± 0.003 0.278 ± 0.003 0.279 ± 0.002 0.280 ± 0.002 0.277 ± 0.002 0.278 ± 0.002 LoRACLR 0.278 ± 0.004 0.273 ± 0.003 0.275 ± 0.003 0.277 ± 0.003 0.278 ± 0.003 0.280 ± 0.002 0.280 ± 0.002 -

DINO

SeqLoRA (Bilevel) 0.502 ± 0.041 0.484 ± 0.029 0.474 ± 0.026 0.468 ± 0.021 0.478 ± 0.017 0.488 ± 0.017 0.494 ± 0.014 0.487 ± 0.013 0.484 ± 0.011 Continual Alternating 0.504 ± 0.040 0.472 ± 0.026 0.477 ± 0.025 0.463 ± 0.020 0.475 ± 0.017 0.482 ± 0.016 0.486 ± 0.015 0.482 ± 0.013 0.485 ± 0.011 Mix-of-Show 0.440 ± 0.053 0.446 ± 0.028 0.435 ± 0.026 0.436 ± 0.023 0.424 ± 0.018 0.416 ± 0.017 Orthogonal Adaptation 0.416 ± 0.048 0.440 ± 0.029 0.429 ± 0.026 0.428 ± 0.023 0.420 ± 0.018 0.417 ± 0.016 0.427 ± 0.014 0.410 ± 0.013 0.416 ± 0.011 LoRACLR 0.435 ± 0.052 0.434 ± 0.032 0.431 ± 0.028 0.434 ± 0.023 0.419 ± 0.018 0.420 ± 0.017 0.429 ± 0.015 -

DINOv2

SeqLoRA (Bilevel) 0.509 ± 0.041 0.452 ± 0.028 0.427 ± 0.021 0.418 ± 0.019 0.437 ± 0.015 0.445 ± 0.014 0.439 ± 0.013 0.435 ± 0.012 0.431 ± 0.012 Continual Alternating 0.515 ± 0.045 0.439 ± 0.022 0.433 ± 0.020 0.412 ± 0.017 0.432 ± 0.014 0.434 ± 0.014 0.429 ± 0.013 0.429 ± 0.012 0.429 ± 0.011 Mix-of-Show 0.439 ± 0.049 0.394 ± 0.026 0.375 ± 0.019 0.377 ± 0.020 0.375 ± 0.016 0.372 ± 0.015 Orthogonal Adaptation 0.405 ± 0.040 0.384 ± 0.025 0.361 ± 0.020 0.361 ± 0.020 0.368 ± 0.016 0.369 ± 0.015 0.371 ± 0.013 0.354 ± 0.013 0.358 ± 0.012 LoRACLR 0.431 ± 0.046 0.367 ± 0.028 0.361 ± 0.022 0.372 ± 0.020 0.367 ± 0.016 0.372 ± 0.015 0.370 ± 0.014 -

DINOv3

SeqLoRA (Bilevel) 0.475 ± 0.031 0.439 ± 0.023 0.425 ± 0.018 0.418 ± 0.016 0.440 ± 0.013 0.446 ± 0.013 0.439 ± 0.011 0.434 ± 0.010 0.428 ± 0.011 Continual Alternating 0.475 ± 0.030 0.427 ± 0.016 0.429 ± 0.018 0.406 ± 0.015 0.431 ± 0.012 0.435 ± 0.013 0.428 ± 0.012 0.426 ± 0.010 0.424 ± 0.010 Mix-of-Show 0.411 ± 0.035 0.385 ± 0.020 0.371 ± 0.016 0.378 ± 0.017 0.374 ± 0.013 0.372 ± 0.013 Orthogonal Adaptation 0.379 ± 0.027 0.375 ± 0.020 0.356 ± 0.017 0.359 ± 0.017 0.365 ± 0.013 0.367 ± 0.014 0.366 ± 0.011 0.350 ± 0.011 0.351 ± 0.010 LoRACLR 0.403 ± 0.032 0.361 ± 0.021 0.359 ± 0.019 0.371 ± 0.017 0.364 ± 0.013 0.369 ± 0.014 0.367 ± 0.012 -

DreamSim

SeqLoRA (Bilevel) 0.513 ± 0.035 0.494 ± 0.024 0.478 ± 0.020 0.477 ± 0.016 0.501 ± 0.014 0.512 ± 0.012 0.515 ± 0.010 0.508 ± 0.010 0.507 ± 0.008 Continual Alternating 0.510 ± 0.033 0.481 ± 0.020 0.480 ± 0.020 0.470 ± 0.015 0.495 ± 0.013 0.504 ± 0.012 0.508 ± 0.011 0.503 ± 0.009 0.504 ± 0.008 Mix-of-Show 0.447 ± 0.041 0.443 ± 0.022 0.436 ± 0.020 0.448 ± 0.017 0.442 ± 0.014 0.445 ± 0.013 Orthogonal Adaptation 0.429 ± 0.037 0.438 ± 0.021 0.430 ± 0.019 0.442 ± 0.018 0.437 ± 0.014 0.442 ± 0.013 0.446 ± 0.010 0.432 ± 0.010 0.438 ± 0.008 LoRACLR 0.445 ± 0.041 0.431 ± 0.024 0.431 ± 0.021 0.445 ± 0.018 0.437 ± 0.014 0.444 ± 0.013 0.447 ± 0.011 -

HPSv2

SeqLoRA (Bilevel) 0.710 ± 0.015 0.704 ± 0.012 0.695 ± 0.010 0.693 ± 0.009 0.707 ± 0.007 0.714 ± 0.007 0.715 ± 0.006 0.711 ± 0.006 0.705 ± 0.005 Continual Alternating 0.717 ± 0.015 0.696 ± 0.012 0.698 ± 0.009 0.688 ± 0.008 0.705 ± 0.007 0.711 ± 0.007 0.711 ± 0.006 0.708 ± 0.006 0.706 ± 0.005 Mix-of-Show 0.677 ± 0.018 0.677 ± 0.011 0.677 ± 0.009 0.677 ± 0.010 0.678 ± 0.008 0.681 ± 0.007 Orthogonal Adaptation 0.664 ± 0.016 0.673 ± 0.011 0.668 ± 0.009 0.671 ± 0.009 0.675 ± 0.008 0.677 ± 0.008 0.677 ± 0.007 0.670 ± 0.007 0.669 ± 0.006 LoRACLR 0.674 ± 0.017 0.664 ± 0.012 0.669 ± 0.009 0.674 ± 0.010 0.674 ± 0.008 0.678 ± 0.008 0.677 ± 0.007 -

SeqLoRA (Bilevel) 0.271 ± 0.003 0.269 ± 0.002 0.270 ± 0.001 0.270 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 Continual Alternating 0.270 ± 0.003 0.270 ± 0.002 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 Mix-of-Show 0.272 ± 0.002 0.271 ± 0.002 0.272 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 Orthogonal Adaptation 0.272 ± 0.003 0.271 ± 0.001 0.272 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 LoRACLR 0.272 ± 0.002 0.270 ± 0.002 0.271 ± 0.001 0.271 ± 0.001 0.271 ± 0.001 0.272 ± 0.001 0.271 ± 0.001 -

HPSv3

Metric Method

SeqLoRA (Bilevel) 9.539 ± 0.588 9.476 ± 0.424 9.171 ± 0.509 9.043 ± 0.399 9.136 ± 0.283 8.828 ± 0.326 8.898 ± 0.252 8.772 ± 0.231 8.904 ± 0.195 Continual Alternating 9.529 ± 0.552 9.827 ± 0.328 9.636 ± 0.407 9.242 ± 0.338 9.427 ± 0.303 9.008 ± 0.347 9.115 ± 0.261 8.991 ± 0.228 9.032 ± 0.200 Mix-of-Show 8.847 ± 0.735 8.968 ± 0.444 8.446 ± 0.503 8.039 ± 0.457 8.013 ± 0.337 7.688 ± 0.314 Orthogonal Adaptation 8.886 ± 0.788 9.135 ± 0.431 8.723 ± 0.413 8.336 ± 0.432 8.343 ± 0.312 8.101 ± 0.314 8.104 ± 0.290 8.029 ± 0.244 8.240 ± 0.205 LoRACLR 9.089 ± 0.722 9.035 ± 0.400 8.681 ± 0.432 8.382 ± 0.431 8.388 ± 0.304 8.142 ± 0.308 8.268 ± 0.270 -

Supplementary Ablation Results

We provide the ablation study results for SeqLoRA with 32 concepts in Table 4, where we vary the number of local steps, the regularization weight ϵ, and the number of bilevel iterations. The results show that the default configuration (first row) provides a strong balance across all metrics. Reducing 38

the number of local steps or bilevel iterations to 1 yields essentially identical identity-preservation scores (DINO, DINOv2, DINOv3, DreamSim) but lowers the human preference score (HPSv3). Increasing the number of local steps to 5 further improves HPSv3 with a marginal change in identity preservation, while smaller values of ϵ (10−4 or 10−5 ) slightly reduce identity scores. Overall, the default configuration is chosen as it maintains near-optimal identity preservation while sustaining a competitive human preference score. Table 4: Ablation study on hyperparameters for SeqLoRA with 32 concepts. Factor

Value

DINO↑

DINOv2↑

DINOv3↑

DreamSim↑

CLIP-I↑

CLIP-T↑

HPSv2↑

HPSv3↑

Default

0.468 ± 0.021

0.418 ± 0.019

0.418 ± 0.016

0.477 ± 0.016

0.693 ± 0.009

0.286 ± 0.003

0.270 ± 0.001

9.043 ± 0.399

Local Steps

1 Step 5 Steps

0.466 ± 0.021 0.469 ± 0.021

0.418 ± 0.017 0.417 ± 0.020

0.414 ± 0.015 0.415 ± 0.017

0.479 ± 0.016 0.475 ± 0.016

0.694 ± 0.009 0.690 ± 0.009

0.287 ± 0.003 0.286 ± 0.003

0.270 ± 0.001 0.271 ± 0.001

8.814 ± 0.422 9.378 ± 0.346

Epsilon

10−4 10−5

0.459 ± 0.021 0.466 ± 0.021

0.406 ± 0.018 0.422 ± 0.019

0.405 ± 0.016 0.417 ± 0.017

0.470 ± 0.017 0.476 ± 0.016

0.686 ± 0.009 0.692 ± 0.009

0.288 ± 0.003 0.287 ± 0.004

0.271 ± 0.001 0.271 ± 0.001

8.838 ± 0.402 8.914 ± 0.402

Bilevel Iters

1 Iter 4 Iters

0.466 ± 0.021 0.464 ± 0.020

0.418 ± 0.019 0.408 ± 0.018

0.417 ± 0.016 0.410 ± 0.016

0.477 ± 0.017 0.474 ± 0.016

0.695 ± 0.009 0.691 ± 0.009

0.288 ± 0.003 0.288 ± 0.003

0.270 ± 0.001 0.271 ± 0.001

8.784 ± 0.377 8.855 ± 0.390

Record · ID 216823 · SHA-256 ab9063b7f971fb8c
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.