Conceptio › Archive › arXiv CS

arXiv CSopen access

Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

Haiyun Qiu1 , Xingyu Wu1,∗ , Kay Chen Tan1

arXiv:2604.22464v1 [cs.LG] 24 Apr 2026

Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China [email protected], [email protected], [email protected]

Abstract Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbonecentric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.

Introduction

While fine-tuning pre-trained models is standard for task adaptation (Wolf et al., 2019; Dodge et al., 2020), the proliferation of independent checkpoints imposes prohibitive storage and deployment overheads (McMahan et al., 2017; Huang et al., 2024). Model merging offers a training-free alternative by directly integrating multiple task-specific models into a unified, versatile model (Yang et al., 2026; Yadav et al., 2025; Li et al., 2025; Ilharco et al., 2023; Yadav et al., 2023; Du et al., 2024; Zhou et al., 2025). However, conventional merging methods assume simultaneous model availability, rendering them ineffective in streaming scenarios where tasks and models arrive sequentially. Consequently, continual model merging (CMM) has emerged to enable sequential integration under a bounded memory footprint (Tang et al., 2025; Qiu et al., 2025b; Yang et al., 2025). Existing CMM methods face a fundamental saturation-redundancy dilemma. On one hand, integrating task-specific parameters into a fixed-capacity backbone via weighted averaging or orthogonal projection inevitably triggers parameter saturation and catastrophic forgetting (Tang et al., 2025; Yang et al., 2025; Qiu et al., 2025a). Conversely, Mixture-of-Experts (MoE) variants mitigate task interference by allocating dedicated experts for each task (Cai et al., 2025; Qiu et al., 2025b). However, such indiscriminate expansion overlooks inter-expert commonalities, causing expert redundancy and architectural bloat. Resolving this dilemma requires adaptively retaining critical experts while Preprint.

pruning redundant ones during merging. Nevertheless, quantifying intrinsic inter-expert correlations to orchestrate such autonomous evolution remains highly non-trivial. Beyond architectural challenges, MoE-based merging methods are further constrained by an explicit routing bottleneck during inference. Typically, they rely on parameterized gating networks to activate experts (Lu et al., 2024; Shen et al., 2026; Qiu et al., 2025b). However, these auxiliary networks inevitably introduce extra learnable weights and necessitate data-driven optimization to sustain routing accuracy. In the context of continual merging, frequent network tuning and inherent data dependency restrict their adaptability in strictly data-free and training-free scenarios. Consequently, a pressing objective is to propose an expert activation mechanism that efficiently captures the intrinsic alignment between input features and experts, bypassing the need for external routing networks. Resolving these intertwined challenges necessitates extracting intrinsic expert representations to quantify correlations among experts and their alignment with input features. However, parameterspace metrics such as cosine similarity are inadequate due to the inherent permutation invariance and basis rotation sensitivity of deep neural networks (Entezari et al., 2022; Ainsworth et al., 2023; Kim et al., 2023). Functionally equivalent experts may manifest as completely disparate parameter configurations, rendering redundancy identification and alignment evaluation intractable. Therefore, we shift from raw parameters to their underlying subspace structures. Grounded in manifold geometry (Edelman et al., 1998; Kornblith et al., 2019), we extract transformation-invariant expert representations. This geometric perspective bridges two critical operations: measuring inter-expert similarities for dynamic evolution and evaluating feature-subspace alignment for adaptive activation. Synthesizing these geometric insights, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing) for efficient expert management and activation in CMM. To address the saturation-redundancy dilemma, MADE-IT introduces a manifold-aware dynamic expert evolution strategy that considers both expert diversity and architectural parsimony. We extract expert principal subspaces via singular value decomposition (SVD) and characterize them as unique points on the Grassmann manifold. We then define a projection-based subspace affinity metric to quantify geometric similarities between candidate and existing experts. Guided by a distribution-aware threshold adaptation mechanism, MADE-IT autonomously orchestrates the expert population by deciding whether to create new experts for specificity or consolidate redundant ones via subspace merging. Furthermore, to address the routing bottleneck, we design a data-free, training-free implicit routing mechanism. This mechanism determines the optimal activation path by evaluating the projection alignment of input features onto expert principal subspaces while leveraging the hierarchical dependencies among modular experts. Our main contributions are summarized as follows: • We reveal the inadequacies of parameter-space metrics in quantifying the geometric correlations underlying expert redundancy and feature matching. In response, we introduce manifold geometry as a robust foundation for characterizing intrinsic expert representations. • We propose a manifold-aware dynamic expert evolution strategy that enables the adaptive, differentiated management of modular expert populations based on subspace affinity, achieving a dynamic balance between expert diversity and architectural parsimony. • We design a data-free and training-free implicit routing mechanism that exploits the geometric alignment between input features and expert principal subspaces, facilitating efficient expert activation without auxiliary gating networks. • Extensive experiments demonstrate that MADE-IT achieves state-of-the-art performance, consistently outperforming strong baselines in accuracy and robustness while significantly reducing expert redundancy, particularly in generic modules and early layers.

Background and Motivation

2.1

Problem Definition

Consider a pre-trained backbone parameterized by θ(0) and a sequence of task-specific models {θ(t) }Tt=1 fine-tuned on associated datasets {Dt }Tt=1 . The task vector for task t is defined as τ (t) = θ(t) − θ(0) (Ilharco et al., 2023). Conventional merging performs one-shot merging of all models via θmerged = F(θ(0) ; θ(1) , . . . , θ(T ) ), where F denotes a merging operator. In contrast, we focus on the continual merging scenario where task-specific models arrive sequentially. At step t, we integrate the 2

(t−1)

incoming model θ(t) with the current merged model θmerged to construct an updated model: (t)

(t−1)

θmerged = F(θ(0) ; θmerged , θ(t) ), t ≥ 2.

(1)

Typically, θmerged is initialized as the first arriving model θ(1) . Our objective is to sequentially merge incoming models in a data-free and training-free manner, while mitigating catastrophic forgetting. 2.2

Related Work

Model Merging. Conventional model merging has evolved from simple weight averaging (Wortsman et al., 2022; Ainsworth et al., 2023) to sophisticated techniques involving importance-based reweighting (Matena and Raffel, 2022; Jin et al., 2023), pre-merging sparsification (Ilharco et al., 2023; Yadav et al., 2023; Du et al., 2024), and dynamic integration (Yang et al., 2024; Lu et al., 2024; Huang et al., 2024; Shen et al., 2026). However, these approaches rely on simultaneous model availability and fail in streaming environments. Consequently, continual model merging has emerged to integrate sequentially arriving models while mitigating catastrophic forgetting (Tang et al., 2025; Qiu et al., 2025b; Yang et al., 2025; Qiu et al., 2025a). Pioneering CMM approaches employ orthogonal projection to reduce interference but inevitably dilute task-specific knowledge over time due to capacity constraints (Tang et al., 2025; Yang et al., 2025). While the recent MoE-based variant MINGLE (Qiu et al., 2025b) alleviates this bottleneck through dedicated expert allocation, it introduces redundant experts and relies on data-driven optimization of parameterized gating networks. Continual Learning. Continual learning aims to mitigate catastrophic forgetting in sequential task learning (McCloskey and Cohen, 1989) through memory replay (Wang et al., 2022; Smith et al., 2023), regularization constraints (Jung et al., 2020; Wu et al., 2024), and dynamic architectural expansion (Zhou et al., 2024; Marouf et al., 2024). However, these approaches demand frequent access to raw data and computationally intensive joint training. In contrast, CMM offers an efficient parameter-centric paradigm (Tang et al., 2025; Qiu et al., 2025b). By sequentially merging incoming task-specific models, CMM inherently circumvents the privacy risks associated with data sharing and significantly enhances the scalability and plasticity of foundation architectures. 2.3

Challenges and Motivation

The Saturation-Redundancy Dilemma. Existing CMM methods either suffer from capacity saturation within a bounded parameter space or incur redundancy through unconstrained expert expansion. For instance, the backbone-centric approach OPCM (Tang et al., 2025) incrementally integrates task vectors τ (t) into a fixed-capacity backbone by projecting them onto the orthogonal complement of (t−1) historical updates via a projection mapping Pα and a scaling factor λ(t) : (t−1) (t−1) (t) λ(t−1) τmerged + Pα τ (t) (0) θmerged = θ + . (t)

(2) λ While ensuring architectural consistency, backbone-centric approaches inevitably trigger parameter saturation and representation conflicts, exacerbating catastrophic forgetting (Yang et al., 2025; Qiu et al., 2025a). Conversely, the MoE-based approach MINGLE (Qiu et al., 2025b) circumvents interference by assigning a dedicated expert fi and an input-dependent gate gi to each sequential task. For a given input X, the output is formulated as: (t)

θmerged (X) = θ(0) (X) +

t X

gi (X) · fi (X).

(3)

i=1

While avoiding interference, such unconditional expansion overlooks intrinsic commonalities among experts. Consequently, the expert population scales linearly with the model stream, yielding significant expert redundancy and architectural bloat. To solve this dilemma, we develop an adaptive, dynamic expert evolution strategy that harmonizes expert diversity with architectural parsimony. The Bottleneck of Explicit Routing. MoE-based merging is further constrained by an explicit routing bottleneck. As defined in Eq. 3, expert contributions are modulated by parameterized gating networks, typically formulated as linear projections with trainable parameters Wtg and bgt : gt (X) = Wtg⊤ X + bgt . 3

(4)

Crucially, these gating networks require continuous data access and frequent network tuning to ensure routing accuracy, limiting the applicability and flexibility of MoE-based CMM methods in strictly data-free and training-free scenarios. To bypass this bottleneck, we design an implicit routing mechanism for efficient expert activation without resorting to auxiliary parameterized networks.

Method

Motivated by the above, we propose MADE-IT for adaptive CMM. As illustrated in Fig. 1, we first resolve the saturation-redundancy dilemma via a manifold-aware dynamic expert evolution strategy (Section 3.1). By representing experts via their principal subspaces on the Grassmann manifold, this strategy utilizes geometric affinity to automatically consolidate redundant experts while isolating specialized ones. Second, to circumvent the routing bottleneck, we design an implicit routing mechanism (Section 3.2). By exploiting geometric feature-subspace alignment, this mechanism facilitates efficient expert activation, bypassing the need for parameterized routing networks. Due to page limitations, the detailed algorithm of MADE-IT is outlined in Appendix B. Principal Subspace Extraction

Architecture Block N

Task Model 𝑊 (")

𝑑%

Candidate Expert

𝐸 !(#) ← SVD𝑟(∆𝑊 (#) )

… Block n Norm

𝑟

Projection-Based Subspace Affinity

Differentiated Expert Evolution

Existing Experts 𝐸 (%)

Creation

+ !(#) 𝑈

Subspace Merging

Yes

MLP Norm

Consolidation

…

⊖

Base Model 𝑊 ($)

Σ- !(#)

∆𝑊 (")

Attention …

𝑑%

Block 1

𝑉- !(#) 𝑑&

𝑑&

Feature Projection Alignment

Subspace Affinity Metric

𝑡−1 𝑡

Adaptive Threshold

𝑟

(a) Expert Evolution Expert Dependency and Path Consistency

Feature Projection Alignment

Intermediate Features 𝐡

Expert Dependency Graph

Task Identity Λ

Projected & Component 𝐡

Activation Path

Λ&'()*+

Anchor Module

Subspace 𝑉( (b) Implicit Routing

Figure 1: Overview of MADE-IT. (a) Expert Evolution: Expert principal subspaces are extracted from the module-wise weight updates via truncated SVD. Projection-based subspace affinity and adaptive thresholds guide the autonomous consolidation or creation of modular experts. (b) Implicit Routing: During inference, expert responsiveness is evaluated using intermediate features via feature projection alignment. Anchored by the highest-diversity module, the task-identity constraint is propagated across the expert dependency graph to resolve bifurcations and navigate a valid activation path.

3.1

Manifold-Aware Dynamic Expert Evolution

Grounded in the low-rank nature of fine-tuned model updates (Hu et al., 2022; Gargiulo et al., 2025), our dynamic expert evolution strategy isolates essential knowledge increments within principal subspaces. Geometrically, characterizing these subspaces as unique points on the Grassmann manifold (Edelman et al., 1998; Hamm and Lee, 2008) yields compact, high-fidelity expert representations. 4

3.1.1

Principal Subspace Extraction (t)

(t)

(0)

Given an incoming fine-tuned model for task t, let ∆Wc = Wc − Wc ∈ Rdo ×di denote the weight update of module c relative to its pre-trained counterpart. Leveraging the low-rank prior of (t) ∆Wc , we apply truncated SVD to extract its principal directions. Specifically, we set a truncation rank r = ⌊ρ · min(do , di )⌋ controlled by a ratio ρ ∈ (0, 1], and construct a lightweight modular expert via a rank-r approximation: (t)⊤ Ec(t) = Ũc(t) Σ̃(t) , c Ṽc

(5)

(t) (t) (t) where Ũc ∈ Rdo ×r and Ṽc ∈ Rdi ×r are the top-r left and right singular vectors, while Σ̃c ∈ (t) (t) r×r R contains the corresponding singular values. The columns of Ũc and Ṽc form orthonormal (t) (t) bases for the principal output subspace Uc and input subspace Vc , respectively. Geometrically,

these subspaces correspond to unique points on the Grassmann manifolds G(r, do ) and G(r, di ). This decomposition effectively projects expert knowledge from the ambient parameter space onto these compact manifolds, enabling efficient storage and manipulation. 3.1.2

Projection-Based Subspace Affinity

As experts may vary in specificity and commonality across modules with different depths and types, (k) |Ec | we maintain an independently evolving expert set for each module c, denoted by Ec = {Ec }k=1 . To manage the expert population, we design a projection-based subspace affinity metric that is intrinsic to subspace geometry and invariant to basis selection. (k)

Projection Operators and Affinity Metric. For a reference expert Ec ∈ Ec and a new expert ′(t) (k) ′(t) Ec , let S̃c and S̃c denote orthonormal bases for a target subspace S ∈ {U , V}. The associated (k) (k)⊤ ′(t) ′(t)⊤ orthogonal projection operators are PS (k) = S̃c S̃c and PS ′(t) = S̃c S̃c , which uniquely characterize these subspaces and remain invariant to orthogonal rotations of the basis. Then, we quantify the correlation between subspaces using the Hilbert-Schmidt inner product of their projection operators, ⟨PS (k) , PS ′(t) ⟩HS = Tr(PS (k) PS ′(t) ), which measures subspace overlap. To ensure computational efficiency, we leverage the trace-Frobenius equivalence, where the HilbertSchmidt inner product of two projection operators is equivalent to the squared Frobenius norm of the inner product between their compact basis matrices (see Appendix A for detailed derivation): Tr(PS (k) PS ′(t) ) = ∥S̃c′(t)⊤ S̃c(k) ∥2F .

(6)

This equivalence obviates the explicit construction of the dense d × d projection matrices, reducing the computational cost to an r × r interaction. Finally, we normalize and aggregate the affinities of input and output subspaces to derive the final metric A ∈ [0, 1]: ′(t)⊤

(k)

′(t)⊤

(k)

Ṽc ∥2F Ũc ∥2F + ∥Ṽc . (7) 2r Here, A = 0 denotes maximal subspace orthogonality, reflecting high specificity of the new candidate expert, while A = 1 indicates perfect alignment and full expert redundancy. A(Ec(k) , Ec′(t) ) =

∥Ũc

Geometric Interpretation. We further analyze the geometric properties of the normalized subspace affinity for a single subspace type, defined as AS = 1r ∥S̃1⊤ S̃2 ∥2F , where S̃1 and S̃2 represent the corresponding orthonormal bases. Spectrally, the relative geometry of two Grassmannian subspaces is characterized by their principal angles {ϕi }ri=1 . Since the singular values of the interaction matrix S̃1⊤ S̃2 precisely yield {cos ϕi }ri=1 , our metric quantifies Pr the spectral mean of the squared cosines of these principal angles, formulated as AS = 1r i=1 cos2 ϕi . Crucially, AS efficiently aggregates alignment information across all principal directions via the Frobenius norm, bypassing the additional SVD calculation required for the full angle spectrum. Furthermore, we establish a linear mapping between AS and the standard Chordal distance dC = √12 ∥S̃1 S̃1⊤ − S̃2 S̃2⊤ ∥F on the √ manifold, yielding AS = 1 − 1r d2C . Unlike the raw chordal distance dC ∈ [0, r] which scales with the rank r, AS provides a scale-invariant measure normalized within [0, 1], ensuring a robust and consistent decision boundary across heterogeneous network modules with varying ranks. Finally, AS preserves basis invariance and symmetry. For any orthogonal rotation Q ∈ O(r), the relation 5

∥S̃1⊤ (S̃2 Q)∥2F = ∥S̃1⊤ S̃2 ∥2F holds, confirming that the measure captures the intrinsic geometry rather than basis artifacts. Concurrently, the cyclic property of the trace ensures geometric symmetry, preventing evaluation bias from the sequential task arrival order. 3.1.3

Differentiated Expert Evolution

To accommodate expert correlations as tasks arrive, we employ a distribution-aware adaptation (t) mechanism instead of a fixed threshold. At each step t, an adaptive threshold γA = µ(t) + βσ (t) is calibrated for merging decisions based on the historical statistics of affinity scores, where µ(t) and σ (t) denote the mean and standard deviation of the collected scores, and β serves as a margin coefficient. Guided by the adaptive thresholds, the modular expert population evolves via two criteria: (k)

′(t)

(t)

Consolidation. If max1≤k≤|Ec | A(Ec , Ec ) ≥ γA , indicating high functional overlap between (k∗ ) (k) ′(t) the candidate and the most similar expert Ec , where k ∗ = arg maxk A(Ec , Ec ), we mitigate

expert redundancy via subspace merging (Gargiulo et al., 2025). We concatenate the bases as Uc = ′(t) (k∗ ) ′(t) (k∗ ) ′(t) (k∗ ) [Ũc , Ũc ] and Vc = [Ṽc , Ṽc ], and form the block-diagonal matrix Σc = diag(Σ̃c , Σ̃c ). After resolving non-orthogonality via polar decomposition to obtain Ûc and V̂c , the updated expert (k∗ ) Ec is finally extracted from the top-r singular components of the reconstructed form Ûc Σc V̂c⊤ . Creation. Otherwise, it implies that the new candidate expert occupies a specific subspace that is not effectively covered by existing experts. Thus, we integrate the new expert into the architecture to preserve diverse expertise and specialization. 3.2

Implicit Routing for Expert Activation

This mechanism is based on our key observations of the geometric alignment between input features and expert principal subspaces, and the hierarchical dependencies among evolved modular experts. 3.2.1

Feature Projection Alignment

During inference, to efficiently activate the optimal expert from the current set Ec at each module, we extract the intermediate activations hc from the pre-trained backbone through the forward pass, which serve as a shared input to evaluate the responsiveness of all experts. (k)

We quantify the compatibility between hc and an expert Ec by focusing on its input-sensitive directions and measuring the match through feature projection alignment (FPA). We first project (k) hc onto the principal input subspace of Ec using the orthogonal projection operator PṼ (k) = c

(k)

(k)⊤

(k)

, yielding the projected component ĥc = PṼ (k) hc . The FPA score M is then defined as Ṽc Ṽc c the cosine similarity between the original input and its projection: (k)

M(hc , Ec(k) ) =

h⊤ c ĥc

(k)

∥hc ∥2 · ∥ĥc ∥2

(8)

A larger M ∈ [0, 1] prioritizes experts whose operational domain aligns most closely with the input signal, ensuring that selected experts induce meaningful feature updates. 3.2.2

Expert Dependency and Path Consistency

Sole reliance on local expert selection may lead to semantically inconsistent paths across modules. To enforce global coherence, we construct an expert dependency graph G = (V, L), where V represents the set of all modular experts, and L denotes valid expert dependency links between (k) (k) adjacent modules. Let Λ(Ec ) denote the set of source task identities associated with expert Ec . A (k) (j) vertical dependency edge (Ec , Ec+1 ) ∈ L connects experts in adjacent modules if and only if they (k) (j) share a common task origin, i.e., Λ(Ec ) ∩ Λ(Ec+1 ) ̸= ∅. Based on G, we design an implicit routing strategy to navigate a semantically valid path in a data-free and training-free manner. The process begins by designating the module with the highest expert diversity as the primary anchor c∗ , where the optimal expert Ec∗∗ with the maximum FPA score (Eq. 8) 6

is activated. This decision initializes the active source set as a global constraint Λactive ← Λ(Ec∗∗ ), which is then propagated to other modules along the graph edges. During this traversal, we prune inconsistent branches by retaining only those candidate experts that maintain vertical consistency with Λactive , meaning that their source set intersection is non-empty. In cases of pathway bifurcation, where multiple experts in a module c satisfy the propagated constraint, the module is treated as a new anchor. We resolve the ambiguity by selecting the expert with the highest FPA score and refining the constraint Λactive ← Λactive ∩ Λ(Ec∗ ). This cycle of constraint propagation and conflict resolution iterates recursively until a unique and globally consistent path is established.

Experiments

4.1

Experiment Setup

Models and Datasets. Following (Tang et al., 2025), we employ CLIP-ViT (Radford et al., 2021) backbones and construct 8, 14, and 20-task groups for merging ViT-B/32, ViT-B/16, and ViT-L/14 models to evaluate scalability and robustness. For fair comparison, we adopt publicly available fine-tuned checkpoints (Tang et al., 2024). Detailed settings are provided in Appendix C. Evaluation Metrics. We adopt average accuracy (ACC) and backward transfer (BWT) (Lin et al., PT (T ) 2022) for evaluation. ACC is the average accuracy across all tasks: ACC = T1 i=1 Ai (θmerged ). BWT quantifies forgetting through the average performance degradation after the final merge: PT −1 (T ) (i) 1 BWT = T −1 i=1 [Ai (θmerged ) − Ai (θmerged )], where Ai (·) denotes the accuracy on the i-th task. Baselines. We compare MADE-IT with three categories of baselines: (1) Non-merging paradigms including pre-training, standard fine-tuning, and continual fine-tuning; (2) Continual adaptations of conventional merging methods including Stochastic Weight Averaging (SWA) (Izmailov et al., 2018), Task Arithmetic (Ilharco et al., 2023), Ties-Merging (Yadav et al., 2023), Layer-Wise AdaMerging (Yang et al., 2024), and LoRA-WEMoE (Shen et al., 2026); and (3) Continual merging methods including OPCM (Tang et al., 2025) and MINGLE (Qiu et al., 2025b). Implementation Details. To evaluate task order sensitivity, we conduct 10 runs for each experiment using random seeds ranging from 42 to 51. All experiments share global hyper-parameters across model architectures and task sequences, with rank ratio ρ = 0.1 and margin coefficient β = 1.0. Sensitivity analysis is provided in Section 4.5. Table 1: Comparative results of MADE-IT against three categories of baselines across three CLIP-ViT architectures. We report the average accuracy (ACC) and backward transfer (BWT) averaged over ten task orders (mean±std). Best results are in bold. ’Continual’ is abbreviated as ’C.’ for conciseness. ViT-B/32 ViT-B/16 ViT-L/14 8 tasks 14 tasks 20 tasks 8 tasks 14 tasks 20 tasks 8 tasks 14 tasks 20 tasks Pre-Trained 48.1 56.9 55.6 55.4 62.0 59.8 64.9 69.1 65.6 Fine-Tuned 90.4 89.3 89.8 92.4 91.3 91.6 94.3 93.4 93.5 C. Fine-Tuned 79.8 67.4 62.6 82.9 72.2 68.2 90.0 70.9 77.7 Average (SWA) 66.3±0.0 65.4±0.0 61.1±0.0 72.3±0.0 69.7±0.0 64.8±0.0 80.0±0.0 77.5±0.0 71.1±0.0 C. Task Arithmetic 67.5±0.0 66.5±0.0 60.0±0.0 77.1±0.0 70.9±0.6 64.2±0.0 82.1±0.0 77.9±0.0 70.3±0.0 C. Ties-Merging 49.0±10.2 66.2±0.6 59.9±0.7 66.8±3.7 70.5±0.8 63.0±1.6 64.3±7.0 78.0±0.6 68.3±0.9 C. LW AdaMerging 53.4±3.2 59.8±1.6 59.7±7.4 59.9±2.3 64.3±1.2 61.5±1.1 68.8±2.9 73.1±5.7 66.9±1.1 C. LoRA-WEMoE 68.8±7.8 63.8±3.4 49.6±15.4 72.6±3.7 67.9±2.9 55.0±7.0 75.6±7.8 74.0±5.0 56.9±19.8 OPCM 75.5±0.5 71.9±0.3 65.7±0.2 81.8±0.3 77.1±0.5 70.3±0.2 87.0±0.4 83.5±0.2 76.0±0.2 MINGLE 85.8±0.8 81.6±1.4 77.1±2.0 88.3±0.6 84.9±0.8 81.9±0.9 91.8±0.2 88.8±0.7 85.5±1.3 MADE-IT (Ours) 87.1±1.6 84.3±0.6 81.4±1.3 90.4±0.6 87.7±0.4 83.0±0.1 93.1±0.7 91.7±0.7 89.6±0.7 Average (SWA) -11.5±2.2 -8.0±1.3 -7.1±2.1 -9.7±1.5 -7.1±1.4 -7.3±1.7 -7.3±1.4 -5.8±1.0 -6.4±1.5 C. Task Arithmetic -9.6±1.5 -1.3±1.6 -3.4±1.0 -4.2±1.0 -1.3±0.4 -3.6±0.4 -7.1±0.8 -1.8±0.3 -3.3±0.3 C. Ties-Merging -15.3±8.0 1.9±0.6 -1.5±0.7 -5.5±0.4 1.4±0.7 -1.5±1.2 -13.0±5.7 -1.1±0.4 -2.9±1.0 C. LW AdaMerging -32.5±3.6 -24.1±1.7 -22.7±4.3 -27.8±2.7 -22.1±1.4 -21.4±1.2 -24.3±3.3 -19.6±1.7 -21.7±1.1 C. LoRA-WEMoE -20.4±9.0 -20.2±3.9 -24.5±10.0 -18.0±6.2 -18.8±3.4 -25.8±7.9 -17.8±5.9 -16.8±5.3 -27.9±17.2 OPCM -6.3±1.1 -6.0±1.0 -7.8±1.5 -4.8±0.7 -5.1±1.4 -6.3±2.2 -2.6±1.0 -4.3±0.7 -6.5±1.8 MINGLE -0.6±0.4 -1.1±0.3 -2.2±0.8 -0.4±0.1 -0.9±0.1 -1.9±0.4 -0.6±0.1 -1.0±0.3 -2.6±0.9 MADE-IT (Ours) -2.2±1.2 -2.6±0.6 -3.8±3.1 -1.2±0.6 -1.9±0.6 -3.0±0.9 -0.6±0.4 -1.0±0.5 -1.7±0.9

BWT (%) ↑

ACC (%) ↑

Method

4.2

Main Results

As detailed in Table 1, MADE-IT consistently outperforms merging baselines across all architectures and sequence lengths in ACC, while preserving competitive BWT scores. In the challenging 20task scenario, MADE-IT achieves 81.4% accuracy with the ViT-B/32 backbone, outperforming the strongest baseline MINGLE by a significant margin of 4.3% and surpassing OPCM by a substantial 15.7%. On the ViT-B/16 and the larger ViT-L/14 architectures, our method establishes new stateof-the-art accuracies, further widening the performance gap over existing techniques. Notably, MADE-IT significantly narrows the disparity between merged models and the theoretical upper bound of individually fine-tuned models. These results underscore the effectiveness of our method in achieving higher accuracy and long-horizon stability without data access or auxiliary training. 4.3

Analysis of Manifold-Aware Expert Evolution

RenderedSST2 GTSRB CIFAR10 SVHN PCAM CIFAR100 Food101 KMNIST MNIST STL10 EMNIST FER2013 Cars OxfordIIITPet RESISC45 FashionMNIST DTD Flowers102 SUN397 EuroSAT 387

378

369

360

351

342

333

324

315

306

297

288

279

270

261

252

243

234

225

216

207

198

189

180

171

162

153

144

135

126

117

108

18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Expert Index

Generic-to-Specific Expert Allocation. Fig. 2 visualizes the module-wise expert allocation trajectories across 20 tasks for the ViT-L/14 architecture, revealing a distinct generic-to-specific hierarchical evolution pattern consistently observed across all architectures (see Appendix D for comprehensive visualizations and analysis). Shallower modules predominantly share a compact set of universal experts by merging redundant ones with high geometric affinity. Conversely, deeper layers transition towards expert specialization to preserve task-specific fidelity. This adaptive allocation autonomously resolves the saturation-redundancy dilemma by harmonizing shallow-layer parameter efficiency with deep-layer plasticity, closely aligning with the feature hierarchy of deep networks.

Figure 2: Module-wise visualization of dynamic expert evolution across 20 sequential tasks using ViT-L/14. In each column, consistent colors indicate shared experts among tasks within the module. Quantitative Analysis of Expert Evolution. Fig. 3 provides a quantitative allocation breakdown for the ViT-B/32 architecture to assess parameter efficiency. Component-wise (Fig. 3a), MLP modules exhibit the highest expert retention, highlighting their pivotal role in encoding task-specific knowledge, whereas generic modules achieve reduction rates near 95%. Depth-wise (Fig. 3b), expert retention increases toward deeper layers. This adaptive allocation confirms that MADE-IT maximizes parameter efficiency through expert sharing in generic modules and early layers, while preserving multi-task capacity via increased diversity in task-sensitive modules and deeper layers.

Avg. # Experts

80 15 60 10

5 0

20 Overall

MLP

Attention Module Type

Full Experts (20)

Other

Avg. # Experts

Reduction Rate

ViT-B/32

100

80 15 60 10

(a)

20 0

4 5 6 7 8 Transformer Block

10 11

Reduction Rate (%)

Reduction Rate

Avg. # Experts

ViT-B/32

Reduction Rate (%)

Full Experts (20)

(b)

Figure 3: Quantitative analysis of dynamic expert evolution across components and network depths.

4.4

Analysis of Projection-Based Subspace Affinity

Quantitative Performance Evaluation. Table 2 demonstrates that MADE-IT using subspace affinity consistently outperforms its cosine similarity variant across all architectures, yielding significantly higher accuracy and reduced catastrophic forgetting. This robust superiority confirms that, unlike parameter-space metrics susceptible to ambient high-dimensional noise, our geometric measure effectively captures intrinsic inter-expert correlations to orchestrate optimal expert evolution. Table 2: Quantitative comparison of similarity metrics across all architectures in the 8-task setting. ViT-B/32 ViT-B/16 ViT-L/14 ACC BWT ACC BWT ACC BWT w/ Cosine Similarity 84.0±2.6 -4.5±2.4 85.1±2.2 -4.9±2.2 92.1±0.7 -1.5±0.6 w/ Projection-Based Subspace Affinity (Ours) 87.1±1.6 -2.2±1.2 90.4±0.6 -1.2±0.6 93.1±0.7 -0.6±0.4 Strategy

Distributional Analysis. Fig. 4 compares the density distribution of pairwise similarity scores. Comparative results reveal that cosine similarity scores cluster tightly around zero due to the inherent orthogonality of high-dimensional parameters, rendering it incapable of distinguishing subtle inter-expert variations. Conversely, our projection-based subspace affinity exhibits a remarkably smoother distribution with a broader dynamic range. This enhanced discriminability unlocks rich informational signals of inter-expert correlations for dynamic evolution.

Cosine Similarity

Projection-Based Subspace Affinity (Ours)

ViT-B-32 30

Density

25 20 15 10 5

Visualization of Semantic Correlations. Fig. 5 provides 0 −0.1 0.0 0.1 0.2 0.3 0.4 0.5 Similarity Value microscopic validation via task-wise similarity heatmaps. While cosine similarity yields near-zero off-diagonal val- Figure 4: Density distribution of pairues that obscure latent relationships, our subspace affinity wise similarity scores across 20 models. naturally manifests distinct clusters (e.g., the MNIST family). This confirms that our metric guides the collaboration of functionally aligned experts with high geometric correlations. Projection-Based Subspace Affinity (Ours)

1 SUN397 2 Cars 3 RESISC45 4 EuroSAT 5 SVHN 6 GTSRB 7 MNIST 8 DTD 9 Flowers102 10 PCAM 11 FER2013 12 OxfordIIITPet 13 STL10 14 CIFAR100 15 CIFAR10 16 Food101 17 FashionMNIST 18 EMNIST 19 KMNIST 20 RenderedSST2

Cosine Similarity

1.0

0.8

0.6

0.4

0.2

0.0 1

11 13 15 17 19

Projection-Based Subspace Affinity (Ours)

(a)

Cosine Similarity

1.0

0.8

0.6

0.4

0.2

0.0 1

11 13 15 17 19

(b)

Figure 5: Similarity heatmaps for (a) mlp.fc1 and (b) self_attn.k.proj in the 6-th layer of ViT-B/32.

4.5

Hyperparameter Sensitivity Analysis

Sensitivity to Rank Ratio ρ. As shown in Fig. 6a, performance plateaus within ρ ≤ 0.6 and precipitously declines as ρ approaches unity. Crucially, unconstrained ranks inject noise as expert subspaces degenerate toward the full parameter space, deteriorating the feature discriminability essential for implicit routing. Consequently, adopting a compact ρ = 0.1 enhances both parameter efficiency and expert specificity via rigorous denoising, ensuring multi-task performance. Sensitivity to Margin Coefficient β. As shown in Fig. 6b, increasing β relaxes the creation criterion, facilitating the formation of diverse expert ensembles that enhance cross-task performance. However, a performance plateau emerges beyond β = 1.0, where further threshold elevation yields diminishing returns and inflates the parameter footprint due to redundant experts. Consequently, adopting β = 1.0 balances representational fidelity with architectural parsimony across complex task sequences. 9

BWT

ACC

100

BWT 0

−2 90

−2

−8 75

ACC (%)

−6 80

BWT (%)

ACC (%)

−4 85

−4

−10

−12

−6

BWT (%)

ACC 100

70 65

−8

−14 60

0.1

0.2

0.4

0.6

0.8

1.0

-1.5

-1.0

-0.5

0.0

0.5

(a)

(b)

1.0

1.5

2.0

−10

Figure 6: Sensitivity analysis of the (a) rank ratio ρ and (b) margin coefficient β across the 8-task setting on the ViT-L/14 architecture. Shaded areas denote the standard deviation.

Conclusion

This paper resolves the saturation-redundancy dilemma and the routing bottleneck in continual merging by proposing MADE-IT, which reframes expert management and activation through manifold geometry. MADE-IT utilizes robust subspace affinity to guide autonomous expert evolution, harmonizing expert diversity with architectural parsimony. Furthermore, its inherently training-free implicit routing mechanism achieves efficient expert activation via feature-subspace alignment. Extensive evaluations confirm that MADE-IT significantly enhances multi-task accuracy while mitigating catastrophic forgetting, establishing it as an adaptive and scalable continual merging approach.

References Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101–mining discriminative components with random forests. In European Conference on Computer Vision. Springer, 446–461. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A Survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering 37, 07 (2025), 3896–3915. Gong Cheng, Junwei Han, and Xiaoqiang Lu. 2017. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE 105, 10 (2017), 1865–1883. Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3606–3613. Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. 2018. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 (2018). Adam Coates, Andrew Ng, and Honglak Lee. 2011. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 215–223. Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, 2921–2926. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 (2020). 10

Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim K Goh, Ho-Kin Tang, Daojing He, et al. 2024. Parameter competition balancing for model merging. In Advances in Neural Information Processing Systems, Vol. 37. 84746–84776. Alan Edelman, Tomás A Arias, and Steven T Smith. 1998. The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 2 (1998), 303–353. Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2022. The role of permutation invariance in linear mode connectivity of neural networks. International Conference on Learning Representations (2022). Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. 2025. Task singular vectors: Reducing task interference in model merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18695–18705. Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. 2013. Challenges in representation learning: A report on three machine learning contests. In International Conference on Neural Information Processing. Springer, 117–124. Jihun Hamm and Daniel D Lee. 2008. Grassmann discriminant analysis: A unifying view on subspace-based learning. In International Conference on Machine Learning. 376–383. Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12, 7 (2019), 2217–2226. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. 2024. Emr-merging: Tuning-free high-performance model merging. In Advances in Neural Information Processing Systems, Vol. 37. 122741–122769. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In International Conference on Learning Representations. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations. Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. 2020. Continual learning with node-importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems, Vol. 33. 3647–3658. Do-Yeon Kim, Dong-Jun Han, Jun Seo, and Jaekyun Moon. 2023. Warping the space: Weight space rotation for class-incremental few-shot learning. In International Conference on Learning Representations. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International Conference on Machine Learning. PMLR, 3519–3529. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3D object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 554–561. 11

Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. (2009). Technical Report. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 2002. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (2002), 2278–2324. Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. 2025. Deep model fusion: A survey. IEEE Transactions on Neural Networks and Learning Systems (2025). Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. 2022. Beyond not-forgetting: Continual learning with backward knowledge transfer. In Advances in Neural Information Processing Systems, Vol. 35. 16165–16177. Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. 2024. Twinmerging: Dynamic integration of modular expertise in model merging. In Advances in Neural Information Processing Systems, Vol. 37. 78905–78935. Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. 2024. Weighted ensemble models are strong continual learners. In European Conference on Computer Vision. Springer, 306–324. Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems, Vol. 35. 17703–17716. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation 24 (1989), 109–165. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. PMLR, 1273–1282. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Vol. 2011. Granada, 7. Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE, 722–729. Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 3498–3505. Zihuan Qiu, Lei Wang, Yang Cao, Runtong Zhang, Bing Su, Yi Xu, Fanman Meng, Linfeng Xu, Qingbo Wu, and Hongliang Li. 2025a. Null-space filtering for data-free continual model merging: Preserving transparency, promoting fidelity. arXiv preprint arXiv:2509.21413 (2025). Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, and Hongliang Li. 2025b. MINGLE: Mixtures of null-space gated low-rank experts for test-time continual model merging. In Advances in Neural Information Processing Systems. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763. Li Shen, Anke Tang, Enneng Yang, Guibing Guo, Yong Luo, Lefei Zhang, Xiaochun Cao, Bo Du, and Dacheng Tao. 2026. Efficient and effective weight-ensembling mixture of experts for multi-task model merging. IEEE Transactions on Pattern Analysis and Machine Intelligence 48, 3 (2026), 2329–2341. James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. 2023. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11909–11919. 12

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 1631–1642. Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks 32 (2012), 323–332. Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. 2024. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv e-prints (2024), arXiv–2406. Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Lefei Zhang, Bo Du, and Dacheng Tao. 2025. Merging on the fly without retraining: A sequential approach to scalable continual model merging. In Advances in Neural Information Processing Systems. Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. 2018. Rotation equivariant CNNs for digital pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 210–218. Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. 2022. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision. Springer, 631–648. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019). Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning. PMLR, 23965–23998. Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. 2024. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In International Conference on Learning Representations. Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017). Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 3485–3492. Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. 2025. A survey on model MoErging: Recycling and routing among specialized experts for collaborative learning. Transactions on Machine Learning Research (2025). Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, Vol. 36. 7093–7115. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2026. Model merging in LLMs, MLLMs, and beyond: Methods, theories, applications, and opportunities. Comput. Surveys 58, 8 (2026), 1–41. Enneng Yang, Anke Tang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, and Jie Zhang. 2025. Continual model merging without data: dual projections for balancing stability and plasticity. In Advances in Neural Information Processing Systems. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024. AdaMerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations. 13

Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. 2024. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23554–23564. Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. 2025. HM3: Hierarchical multiobjective model merging for pretrained models. In Advances in Neural Information Processing Systems.

Theoretical Properties of Subspace Affinity

This section provides detailed derivations for the theoretical properties of our proposed subspace affinity. Let S̃1 , S̃2 ∈ Rd×r denote the orthonormal basis matrices spanning two r-dimensional subspaces S1 and S2 . The orthogonal projection operators are defined as P1 = S̃1 S̃1⊤ and P2 = S̃2 S̃2⊤ . The individual affinity component is defined as AS = 1r ∥S̃1⊤ S̃2 ∥2F . A.1

Trace-Frobenius Equivalence

The computational efficiency of AS is facilitated by the equivalence between the Hilbert-Schmidt inner product of projection operators and the squared Frobenius norm of their basis interaction matrix. Lemma 1 (Trace-Frobenius Equivalence). Given projection operators P1 , P2 and their orthonormal bases S̃1 , S̃2 , the following identity holds: Tr(P1 P2 ) = ∥S̃1⊤ S̃2 ∥2F .

(9)

Proof. Expanding the trace of the product of the projection operators yields: Tr(P1 P2 ) = Tr(S̃1 S̃1⊤ S̃2 S̃2⊤ ).

(10)

Leveraging the cyclic property of the trace, Tr(ABCD) = Tr(DABC), we cyclically permute the matrices: Tr(S̃1 S̃1⊤ S̃2 S̃2⊤ ) = Tr(S̃2⊤ S̃1 S̃1⊤ S̃2 ) = Tr((S̃1⊤ S̃2 )⊤ (S̃1⊤ S̃2 )). (11) Recall that for any matrix M , Tr(M ⊤ M ) = ∥M ∥2F . Setting M = S̃1⊤ S̃2 ∈ Rr×r , we obtain: Tr(P1 P2 ) = ∥S̃1⊤ S̃2 ∥2F ,

(12)

which proves the lemma. This identity justifies the computational reduction from the d × d ambient parameter space to an r × r compact subspace interaction. A.2

Relation to Principal Angles

Proposition 1 (Spectral Interpretation). Let 0 ≤ ϕ1 ≤ · · · ≤ ϕr ≤ π/2 be the principal angles between S1 and S2 . The affinity AS represents the mean squared cosine of these principal angles: r

AS =

1X cos2 ϕi . r i=1

(13)

Proof. In manifold geometry, the cosines of the principal angles {cos ϕi }ri=1 are equal to the singular values {σi }ri=1 of the basis interaction matrix M = S̃1⊤ S̃2 . Let the singular value decomposition of M be M = U ΣV ⊤ , where U, V ∈ O(r) are orthogonal matrices, and Σ = diag(cos ϕ1 , . . . , cos ϕr ). The squared Frobenius norm of M can be expanded as: ∥M ∥2F = Tr(M ⊤ M ) = Tr(V ΣU ⊤ U ΣV ⊤ ).

(14)

Applying U ⊤ U = Ir , V ⊤ V = Ir and leveraging the cyclic property of the trace, this simplifies to: ∥M ∥2F = Tr(Σ2 ) =

r X

cos2 ϕi .

(15)

i=1

Pr Dividing by the rank r yields AS = 1r i=1 cos2 ϕi , completing the proof. This confirms that AS ∈ [0, 1] serves as a normalized measure of total spectral alignment. 14

A.3

Relation to Chordal Distance

Proposition 2 (Equivalence to Normalized Chordal Distance). The affinity AS is linearly related to the squared Chordal distance d2C = 12 ∥P1 − P2 ∥2F via the mapping: 1 AS = 1 − d2C . r

(16)

Proof. Expanding the Frobenius norm definition of the Chordal distance yields: 1 Tr((P1 − P2 )⊤ (P1 − P2 )) 2 1 = (Tr(P21 ) + Tr(P22 ) − 2 Tr(P1 P2 )). 2

d2C =

(17)

Utilizing the idempotence (P2 = P) and the property that Tr(P) = rank(P) = r: d2C =

1 (r + r − 2 Tr(P1 P2 )) = r − Tr(P1 P2 ). 2

(18)

By applying the Trace-Frobenius Lemma, we substitute Tr(P1 P2 ) = ∥S̃1⊤ S̃2 ∥2F : d2C = r − ∥S̃1⊤ S̃2 ∥2F .

(19)

Substituting the definition AS = 1r ∥S̃1⊤ S̃2 ∥2F , we obtain AS = 1 − 1r d2C , completing the proof. This confirms that AS operates as a scale-invariant metric for subspace proximity.

Workflow of MADE-IT

We summarize the main workflow of MADE-IT in Algorithm 1, which comprises two primary phases: Manifold-Aware Dynamic Expert Evolution and Implicit Routing. Dynamic Expert Evolution. The procedure commences by initializing the expert repositories for all target modules (line 1). Upon the arrival of each fine-tuned model for a specific task, the algorithm extracts the principal subspace from the module-wise weight updates via truncated SVD to construct a candidate modular expert (lines 4-5). Subsequently, it computes the projection-based subspace affinity between the newly extracted candidate expert and all existing experts within the respective module (lines 6-7). Based on an adaptive threshold calibrated from historical statistics (lines 8-9), the algorithm performs branching logic for expert evolution: if the maximum affinity is below the threshold, the mechanism instantiates the candidate as a new expert and assigns its task identity (lines 11-13); otherwise, it consolidates the candidate into the most similar reference expert via subspace merging and updates the corresponding task identity set (lines 14-18). Finally, the globally maintained expert set is updated (lines 20-21). Implicit Routing. During multi-task inference, we first initialize the activated expert set (line 23), identify the anchor module with the highest expert diversity (line 24), and perform a backbone forward pass to obtain intermediate features (line 25). The optimal expert for the anchor module is activated based on the Feature Projection Alignment scores, and the global pathway constraint is initialized according to its task identity set (lines 26-29). This constraint is then propagated across the expert dependency graph to activate functionally consistent candidates and resolve potential path bifurcations in other modules (lines 30-37). Finally, the activated experts are integrated into the pre-trained backbone for model inference (line 38).

Details of Experiment Settings

C.1

Model and Dataset Details

Following the experimental setup from (Tang et al., 2025), we use ViT-B/32, ViT-B/16 and ViT-L/14 models, each independently fine-tuned on 20 image classification datasets. These vision tasks include SUN397 (Xiao et al., 2010), Stanford Cars (Krause et al., 2013), RESISC45 (Cheng et al., 2017), EuroSAT (Helber et al., 2019), SVHN (Netzer et al., 2011), GTSRB (Stallkamp et al., 2012), MNIST 15

Algorithm 1 MADE-IT Procedure. Input: Pre-trained model θ(0) , sequentially arriving fine-tuned models {θ(t) }Tt=1 , target module set C, rank ratio ρ, margin coefficient β, input x Output: Expert set E(T ) , output y 1: Initialize: E(0) = {Ec }c∈C ← ∅ # Expert Evolution 2: for each task t = 1 to T do 3: for each module c ∈ C do 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16:

▷ extract principal subspace via SVDr (Wc(t) − Wc(0) ) with low rank r ← ⌊ρ · min(di , do )⌋ ′(t) ′(t) ′(t) ′(t)⊤ Ec ← Ũc Σ̃c Ṽc via Eq.(5) ▷ compute subspace affinity with existing experts (k) ′(t) (k) Compute A(Ec , Ec ), for Ec ∈ Ec via Eq.(7) ▷ adapt threshold with mean µ and std σ of historical affinity scores (t) γA = µ(t) + βσ (t) ▷ evolve experts (t) if Ec = ∅ ∨ max A(Ec(k) , Ec′(t) ) < γA then 1≤k≤|Ec | ′(t) Ec ← Ec ∪ {Ec } ′(t) Λ(Ec ) ← {t}

else (k) ′(t) k ∗ = arg maxk A(Ec , Ec ) ∗ (k ) (k∗ ) ′(t) Ec ← SubspaceMerge(Ec , Ec ) ∗ ∗ (k ) (k ) Λ(Ec ) ← Λ(Ec ) ∪ {t} end if end for

17: 18: 19: 20: ▷ update expert set 21: E(t) ← {Ec }c∈C 22: end for 23: Initialize: Eactivated ← ∅ 24: c∗ ← arg maxc∈C |Ec |

# Implicit Routing

25: {hc }c∈C ← Forward(x; θ (0) ) 26: ▷ activate expert by computing projection alignment scores 27: Ec∗∗ ← arg maxE∈Ec∗ M(hc∗ , E) via Eq. (8) 28: Λactive ← Λ(Ec∗∗ ) 29: Eactivated ← Eactivated ∪ {Ec∗∗ } 30: for c ∈ C \ {c∗ } do (k)

(k)

31: E∗c ← {Ec | Λ(Ec ) ∩ Λactive ̸= ∅} 32: if |E∗c | > 1 then 33: E∗c ← {arg maxE∈E∗c M(hc , E)} 34: end if 35: Λactive ← Λactive ∩ Λ(E∗c ) 36: Eactivated ← Eactivated ∪ E∗c 37: end for 38: y ← f (x; Integrate(θ (0) , Eactivated ))

(LeCun et al., 2002), DTD (Cimpoi et al., 2014), Flowers102 (Nilsback and Zisserman, 2008), PCAM (Veeling et al., 2018), FER2013 (Goodfellow et al., 2013), Oxford-IIIT Pet (Parkhi et al., 2012), STL-10 (Coates et al., 2011), CIFAR-100 and CIFAR-10 (Krizhevsky and Hinton, 2009), Food-101 (Bossard et al., 2014), Fashion-MNIST (Xiao et al., 2017), EMNIST (Cohen et al., 2017), KMNIST (Clanuwat et al., 2018), and Rendered SST-2 (Socher et al., 2013). These open-source datasets and fine-tuned checkpoints are obtained from Hugging Face (Wolf et al., 2019). During fine-tuning, the visual encoder is updated while the text encoder remains fixed. Training follows a consistent setup: cross-entropy loss, Adam optimizer, cosine annealing schedule, learning rate of 1 × 10−5 , batch size of 128, and 4000 training steps. C.2

Task Grouping and Sequence Protocols

To comprehensively evaluate the scalability and long-horizon stability of continual merging methods under varying workloads, following the protocols in (Tang et al., 2025; Qiu et al., 2025b), we partition the 20 benchmark datasets into three progressive task groups: • Short-Range Benchmark (8-Task Group): (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6) GTSRB, (7) MNIST, (8) DTD. • Medium-Range Benchmark (14-Task Group): (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6) GTSRB, (7) MNIST, (8) DTD, (9) Flowers102, (10) PCAM, (11) FER2013, (12) OxfordIIITPet, (13) STL10, (14) CIFAR100. • Long-Range Benchmark (20-Task Group): (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6) GTSRB, (7) MNIST, (8) DTD, (9) Flowers102, (10) PCAM, (11) FER2013, (12) OxfordIIITPet, (13) STL10, (14) CIFAR100, (15) CIFAR10, (16) Food101, (17) FashionMNIST, (18) EMNIST, (19) KMNIST, (20) RenderedSST2. To mitigate the impact of task order and ensure statistical reliability, we conduct 10 independent runs for each task group, each with a distinct randomly permuted task sequence generated with seeds ranging from 42 to 51, as detailed in Table 3. Consequently, we report the mean and standard deviation of both average accuracy (ACC) and backward transfer (BWT) metrics across these trials, providing a robust and consistent evaluation of continual merging performance.

Group

Order

8 Tasks

1 2 3 4 5 6 7 8 9 10

04 → 05 → 07 → 08 → 03 → 06 → 01 → 02 07 → 08 → 05 → 04 → 02 → 06 → 03 → 01 03 → 06 → 04 → 02 → 01 → 08 → 05 → 07 06 → 08 → 02 → 01 → 03 → 07 → 04 → 05 07 → 06 → 03 → 08 → 05 → 01 → 04 → 02 07 → 02 → 03 → 08 → 05 → 04 → 01 → 06 07 → 01 → 04 → 03 → 08 → 05 → 02 → 06 08 → 05 → 06 → 07 → 01 → 04 → 03 → 02 01 → 04 → 05 → 02 → 06 → 03 → 07 → 08 08 → 03 → 01 → 02 → 06 → 05 → 07 → 04

14 Tasks

1 2 3 4 5 6 7 8 9 10

09 → 13 → 08 → 07 → 14 → 12 → 06 → 03 → 10 → 04 → 05 → 01 → 02 → 11 09 → 10 → 11 → 14 → 07 → 13 → 04 → 02 → 06 → 08 → 03 → 12 → 05 → 01 05 → 08 → 12 → 06 → 11 → 01 → 10 → 04 → 14 → 03 → 02 → 13 → 09 → 07 03 → 10 → 09 → 12 → 04 → 13 → 01 → 06 → 11 → 02 → 14 → 08 → 07 → 05 08 → 14 → 09 → 06 → 12 → 13 → 05 → 03 → 04 → 11 → 10 → 01 → 07 → 02 03 → 12 → 13 → 01 → 11 → 04 → 10 → 05 → 14 → 08 → 09 → 07 → 02 → 06 07 → 01 → 12 → 10 → 02 → 08 → 13 → 04 → 05 → 11 → 14 → 03 → 06 → 09 05 → 12 → 04 → 11 → 03 → 08 → 10 → 01 → 09 → 13 → 14 → 07 → 06 → 02 10 → 07 → 09 → 02 → 03 → 13 → 01 → 12 → 14 → 04 → 11 → 06 → 05 → 08 01 → 02 → 11 → 06 → 08 → 12 → 07 → 05 → 10 → 14 → 03 → 13 → 09 → 04

20 Tasks

Table 3: Task sequences used for robustness evaluation. Task IDs correspond to the index in the respective dataset list. Dataset Order (by Task ID)

1 2 3 4 5 6 7 8 9 10

20 → 06 → 15 → 05 → 10 → 14 → 16 → 19 → 07 → 13 → 18 → 11 → 02 → 12 → 03 → 17 → 08 → 09 → 01 → 04 09 → 14 → 06 → 03 → 07 → 04 → 18 → 01 → 17 → 19 → 08 → 20 → 13 → 16 → 11 → 12 → 15 → 05 → 10 → 02 09 → 15 → 16 → 11 → 03 → 13 → 08 → 10 → 12 → 02 → 20 → 01 → 05 → 19 → 07 → 06 → 04 → 18 → 17 → 14 17 → 04 → 11 → 19 → 18 → 10 → 07 → 15 → 12 → 13 → 08 → 02 → 01 → 06 → 05 → 03 → 20 → 16 → 14 → 09 14 → 16 → 04 → 20 → 15 → 17 → 07 → 11 → 06 → 18 → 12 → 01 → 19 → 09 → 10 → 05 → 08 → 02 → 13 → 03 02 → 06 → 17 → 04 → 19 → 18 → 08 → 16 → 20 → 01 → 10 → 13 → 07 → 09 → 05 → 11 → 15 → 14 → 03 → 12 19 → 01 → 09 → 14 → 06 → 20 → 17 → 04 → 08 → 02 → 15 → 03 → 16 → 13 → 12 → 07 → 10 → 05 → 11 → 18 15 → 07 → 08 → 02 → 10 → 06 → 17 → 20 → 05 → 19 → 16 → 01 → 18 → 09 → 13 → 11 → 04 → 14 → 12 → 03 10 → 05 → 07 → 11 → 01 → 03 → 17 → 15 → 18 → 04 → 14 → 19 → 02 → 06 → 13 → 20 → 08 → 12 → 09 → 16 01 → 11 → 02 → 15 → 03 → 10 → 12 → 19 → 16 → 13 → 07 → 05 → 09 → 04 → 14 → 20 → 06 → 18 → 17 → 08

C.3

Baseline Details

We provide an elaborate description of the three baseline categories to facilitate a clear comparison. Non-Merging Methods. • Pre-trained models are used as zero-shot baselines, establishing the lower bound of performance without task-specific adaptation. • Fine-tuned models are independently adapted for each task, serving as specialized references that preserve optimal task-specific performance. • C. Fine-Tuned refers to a standard sequential learning baseline where a single model is iteratively fine-tuned on arriving tasks. Continual Adaptations of Conventional Merging Methods. • Simple Weight Averaging (SWA) (Izmailov et al., 2018) maintains a running average of model parameters to stabilize optimization. At each step, the current merged model is updated by averaging its previous state with the new model parameters. • Continual Task Arithmetic (C. TA) (Ilharco et al., 2023) treats task-specific updates as vectors and linearly accumulates them onto the pre-trained backbone using a global scaling coefficient. • Continual Ties-Merging (C. Ties-Merging) (Yadav et al., 2023) prunes redundant parameters and resolves sign conflicts to mitigate task interference, and merges the pruned task vector into the accumulated backbone. • C. LW AdaMerging (Yang et al., 2024) advances Task Arithmetic by learning layer-wise scaling coefficients. • Continual LoRA Weight-Ensembling MoE (C. LoRA-WEMoE) (Shen et al., 2026) refers to an adaptation of the WEMoE method for parameter-efficient continual merging. Following the setting in (Qiu et al., 2025b), we compress the MLP experts using LoRA and aggregate them via a gating function. Continual Merging Methods. • Orthogonal Projection-based Continual Merging (OPCM) (Tang et al., 2025) projects each task vector τ (t) onto the orthogonal complement of previous updates using a pro(t−1) (t) jection mapping Pα , then generates the new merged model: θmerged = θ(0) + (t−1)

(t−1) (t) λ(t−1) τmerged +Pα τ (t−1) (t−1) , where τmerged = θmerged − θ(0) and λ(t) denotes a time-varying λ(t)

scaling factor. • Mixture of Null-Space Gated Low-Rank Experts (MINGLE) (Qiu et al., 2025b) represents an MoE-based CMM approach that assigns each incoming model a dedicated expert fi , built upon the null-space projection mechanism of OPCM. It further adapts the associated taskspecific gate function gi at test time using unlabeled test data. The resulting merged model Pt (t) output for a given input X is formulated as: θmerged (X) = θ(0) (X) + i=1 gi (X) · fi (X).

Comprehensive Results

D.1

Detailed Per-Task Performance

We present the per-task average accuracy across 10 randomized task sequences for the 20-task continual merging experiments in Table 4, Table 5, and Table 6. The overall results of these pertask accuracy are summarized in Table 1 of the main text. Across all three CLIP-ViT backbones, MADE-IT consistently improves performance on the majority of tasks. These per-task breakdowns further substantiate the overall effectiveness of MADE-IT, reinforcing the main findings reported in the paper. 18

Table 4: Per-task results of merging 20 ViT-B/32 models. Method C. Fine-Tuned Average (SWA) C. Task Arithmetic C. Ties-Merging C. LW AdaMerging C. LoRA-WEMoE OPCM MINGLE MADE-IT (Ours)

SUN397 53.9 64.2 62.0 62.5 63.1 51.4 64.4 67.8 75.5

Cars 38.2 59.6 53.7 49.1 60.0 45.8 51.1 58.3 77.6

Method FER2013 OxfordIIITPet C. Fine-Tuned 58.3 68.5 Average (SWA) 50.2 84.1 C. Task Arithmetic 51.4 82.3 C. Ties-Merging 49.5 81.3 C. LW AdaMerging 43.2 83.7 C. LoRA-WEMoE 44.6 72.5 OPCM 58.5 82.9 MINGLE 65.0 85.5 MADE-IT (Ours) 70.3 91.8

RESISC45 64.7 64.8 60.9 55.8 63.5 63.3 66.0 83.5 86.6 STL10 86.7 97.0 94.9 95.2 96.8 86.1 95.9 97.0 91.8

EuroSAT 98.7 60.9 58.1 50.9 60.1 43.5 71.7 90.0 86.5

SVHN 45.4 47.3 48.5 54.6 35.6 42.9 66.1 82.9 78.0

GTSRB 34.4 43.1 48.9 49.3 32.1 34.6 56.0 91.8 96.3

MNIST 86.7 71.8 79.4 82.0 51.8 58.9 90.2 98.0 80.8

DTD 58.4 46.4 46.1 46.7 45.4 46.5 40.4 65.3 75.1

CIFAR100 CIFAR10 Food101 FashionMNIST EMNIST 40.2 70.5 50.0 90.7 72.4 69.8 92.7 80.4 71.3 15.0 64.6 91.4 71.9 73.9 17.8 63.7 91.2 70.2 73.7 17.8 67.0 89.9 81.6 63.7 16.8 40.1 63.8 63.8 48.1 10.3 67.6 92.8 74.0 76.3 22.4 72.6 94.1 81.5 85.4 50.4 71.9 89.7 88.3 84.8 71.0

Flowers102 57.5 66.5 61.1 58.5 66.6 47.5 64.9 74.0 84.5

PCAM 67.7 63.9 73.4 69.9 60.2 60.1 80.2 66.9 84.9

KMNIST 54.5 11.5 12.2 16.9 10.7 12.8 18.3 65.2 71.4

RenderedSST2 54.5 61.8 59.9 59.8 59.1 55.7 64.6 67.1 71.3

Flowers102 64.1 71.8 67.5 61.3 71.8 64.3 72.3 82.6 93.5

PCAM 75.2 63.1 67.1 68.5 57.9 51.7 79.6 80.6 88.6

KMNIST 51.5 12.4 16.7 24.9 11.0 16.6 23.4 79.9 85.2

RenderedSST2 72.8 62.5 61.8 61.5 59.2 52.7 66.8 74.0 57.2

Flowers102 83.2 80.0 78.6 77.9 80.6 66.2 84.8 94.8 98.2

PCAM 78.3 73.6 70.4 72.3 52.4 54.2 76.3 75.3 86.4

KMNIST 59.2 10.4 10.6 12.6 9.8 10.9 14.7 88.6 80.8

RenderedSST2 78.8 66.1 63.6 58.4 66.6 56.5 71.0 79.8 79.9

Table 5: Per-task results of merging 20 ViT-B/16 models. Method C. Fine-Tuned Average (SWA) C. Task Arithmetic C. Ties-Merging C. LW AdaMerging C. LoRA-WEMoE OPCM MINGLE MADE-IT (Ours)

SUN397 62.7 67.1 65.8 64.2 65.5 62.7 67.9 71.5 77.4

Cars 58.0 64.6 57.5 52.9 65.7 60.2 55.9 64.9 85.2

Method FER2013 OxfordIIITPet C. Fine-Tuned 60.5 84.5 Average (SWA) 50.9 89.6 C. Task Arithmetic 50.7 89.3 C. Ties-Merging 50.4 87.9 C. LW AdaMerging 46.8 88.9 C. LoRA-WEMoE 45.6 91.2 OPCM 59.5 91.8 MINGLE 67.6 92.7 MADE-IT (Ours) 72.3 94.7

RESISC45 67.6 69.3 63.8 60.9 69.8 69.4 73.7 85.3 91.8 STL10 90.5 98.0 97.0 96.3 98.1 92.3 97.7 97.4 92.7

EuroSAT 99.1 63.4 59.5 53.0 59.4 37.7 77.5 90.0 95.2

SVHN 46.0 62.4 64.7 62.8 50.1 52.1 74.4 87.5 67.8

GTSRB 29.2 52.7 54.0 48.8 44.2 39.9 63.2 90.1 88.0

MNIST 93.9 80.7 88.0 88.4 61.1 63.1 94.1 97.1 89.6

DTD 61.9 46.6 45.3 45.0 47.1 45.3 49.2 62.7 76.1

CIFAR100 CIFAR10 Food101 FashionMNIST EMNIST 38.8 73.6 61.9 89.7 83.3 72.9 94.2 85.9 73.3 15.6 68.0 93.1 80.3 75.7 18.1 63.1 91.7 78.0 75.0 23.4 69.2 91.4 86.6 67.2 17.2 41.3 64.3 78.1 48.0 23.5 73.2 94.7 83.1 81.3 26.5 74.0 95.3 87.7 87.4 73.5 78.1 89.3 91.6 84.4 62.1

Table 6: Per-task results of merging 20 ViT-L/14 models. Method C. Fine-Tuned Average (SWA) C. Task Arithmetic C. Ties-Merging C. LW AdaMerging C. LoRA-WEMoE OPCM MINGLE MADE-IT (Ours)

SUN397 69.5 70.7 70.4 69.7 68.8 62.1 73.1 75.9 82.6

Cars 73.6 77.7 74.1 70.3 78.6 68.1 78.3 83.4 92.7

Method FER2013 OxfordIIITPet C. Fine-Tuned 68.0 92.1 Average (SWA) 52.7 94.2 C. Task Arithmetic 55.7 94.2 C. Ties-Merging 57.6 93.5 C. LW AdaMerging 49.2 93.5 C. LoRA-WEMoE 46.3 84.5 OPCM 61.8 95.4 MINGLE 67.7 96.0 MADE-IT (Ours) 74.7 95.6

RESISC45 78.3 76.4 73.9 65.3 75.9 68.7 82.4 87.8 90.6 STL10 94.5 99.2 98.6 97.8 99.3 87.6 99.2 98.7 98.4

EuroSAT 99.2 75.3 66.3 47.9 65.7 53.2 80.2 88.7 90.2

SVHN 59.3 69.5 69.9 76.1 58.3 47.5 80.8 91.1 96.3

GTSRB 49.3 62.1 65.6 63.6 51.6 49.4 80.4 94.5 98.5

MNIST 98.6 93.7 95.1 94.7 79.9 69.8 97.4 98.4 95.4

DTD 69.7 57.7 56.6 54.4 57.4 49.1 61.6 70.8 82.2

CIFAR100 CIFAR10 Food101 FashionMNIST EMNIST 60.5 85.7 74.8 93.1 89.0 81.7 97.0 90.7 77.4 16.1 79.1 96.6 87.6 80.8 17.6 74.0 95.6 84.7 79.7 20.2 77.2 95.8 91.1 68.2 18.6 52.1 70.5 73.3 50.0 18.7 83.0 97.8 90.9 86.0 26.4 81.4 97.1 90.6 90.6 60.7 86.9 97.4 94.7 92.9 78.4

D.2

Additional Results of Expert Evolution

RenderedSST2 GTSRB CIFAR10 SVHN PCAM CIFAR100 Food101 KMNIST MNIST STL10 EMNIST FER2013 Cars OxfordIIITPet RESISC45 FashionMNIST DTD Flowers102 SUN397 EuroSAT 196

192

188

184

180

176

172

168

164

160

156

152

148

144

140

136

132

128

124

120

116

112

108

104

100

19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Expert Index

We provide detailed visualizations of the dynamic expert evolution for the ViT-B/32 and ViT-B/16 architectures in Figure 7. Consistent with the evolution trajectories analyzed for the ViT-L/14 architecture in the main text, the visualizations across all architectures demonstrate a clear genericto-specific hierarchical transition. Specifically, in modules located at shallower network depths, the allocation maps are dominated by consistent color blocks, indicating that diverse tasks frequently share a compact set of universal experts. This phenomenon confirms that our manifold-aware strategy effectively identifies high geometric affinity among the principal subspaces of experts in early layers, thereby promoting collaborative sharing. Conversely, as network depth increases, the visualizations exhibit increasingly divergent color variations. This reflects a shift towards expert specialization, where the strategy isolates and retains distinct experts to accommodate highly decoupled, task-specific requirements. Such robust cross-architecture consistency reinforces our conclusion that MADE-IT effectively leverages the intrinsic functional hierarchy of deep neural networks to autonomously manage expert redundancy in complex model streams.

Expert Index

18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

196

192

188

184

180

176

172

168

164

160

156

152

148

144

140

136

132

128

124

120

116

112

108

104

100

18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Expert Index

(a) ViT-B/32 RenderedSST2 GTSRB CIFAR10 SVHN PCAM CIFAR100 Food101 KMNIST MNIST STL10 EMNIST FER2013 Cars OxfordIIITPet RESISC45 FashionMNIST DTD Flowers102 SUN397 EuroSAT

(b) ViT-B/16

387

378

369

360

351

342

333

324

315

306

297

288

279

270

261

252

243

234

225

216

207

198

189

180

171

162

153

144

135

126

117

108

RenderedSST2 GTSRB CIFAR10 SVHN PCAM CIFAR100 Food101 KMNIST MNIST STL10 EMNIST FER2013 Cars OxfordIIITPet RESISC45 FashionMNIST DTD Flowers102 SUN397 EuroSAT

Figure 7: Complete module-wise visualizations of dynamic expert evolution across 20 sequential tasks using the ViT-B/32, ViT-B/16, and ViT-L/14 architectures, respectively. In each visualization, consistent colors denote shared experts among tasks within the corresponding module. Complementing the analysis in the main text, Figure 8 provides the complete quantitative allocation breakdown across all evaluated architectures (ViT-B/32, ViT-B/16, and ViT-L/14). The consistent trends across varying model architectures and scales substantiate the robustness of our dynamic expert evolution strategy. From a component-wise perspective (Figures 8a-8c), MLP modules consistently exhibit the highest expert retention with an average of 11 to 12 experts, followed by Attention modules with approximately 8 to 9 experts. This reinforces the critical function of MLP modules in encoding task-specific knowledge. Conversely, other modules maintain significantly high reduction rates, validating their predominantly task-agnostic nature that facilitates extensive expert sharing. 20

From a depth-wise perspective (Figures 8d-8f), the number of retained experts progressively increases with network depth, with reduction rates dropping from around 80% in shallow layers to about 40% in deeper ones. This validates the premise that earlier layers prioritize capturing shared, generalized representations, whereas deeper layers demand specialized functional pathways to satisfy task-specific requirements. Reduction Rate

20 Attention

Other

Module Type

20 Overall

MLP

20 4

20 Overall

MLP

Attention

Module Type

Other

Transformer Block

10 11

80 15 60 10

5 0

20 0

Transformer Block

(d)

ViT-L/14

100

Avg. # Experts

Reduction Rate (%)

Avg. # Experts

(c)

ViT-B/16

100

Reduction Rate

60 10

10 11

100

80 15 60 10

5 0

20 0

Reduction Rate (%)

Avg. # Experts

ViT-B/32

80 15

(b)

Other

Module Type

(a) Full Experts (20)

Attention

Avg. # Experts

MLP

Reduction Rate (%)

Overall

100

Reduction Rate (%)

Avg. # Experts

ViT-L/14

100

Avg. # Experts

ViT-B/16

100

Reduction Rate (%)

Avg. # Experts

ViT-B/32

Reduction Rate (%)

Full Experts (20)

10 12 14 16 18 20 22

Transformer Block

(e)

(f)

Figure 8: Comprehensive quantitative analysis of dynamic expert evolution across components and network depths for all evaluated architectures. The top row (a-c) shows the average number of retained experts and reduction rates by module type (MLP, Attention, Other). The bottom row (d-f) illustrates the layer-wise distribution across Transformer blocks.

D.3

Additional Results of Density Distributions

To decipher the mechanism underlying the performance gains of our subspace affinity, we provide comprehensive density distributions of pairwise similarity scores for the ViT-B/32, ViT-B/16, and ViT-L/14 architectures in Figure 9. Consistent with the ViT-B/32 observations discussed in the main text, the comparative results across different architectures and model scales demonstrate a clear concentration phenomenon when utilizing cosine similarity, where similarity scores cluster tightly around zero. This consistent collapse is fundamentally attributed to the inherent orthogonality of high-dimensional parameter spaces. Such concentration renders parameter-space metrics incapable of distinguishing the subtle differences among task-specific updates. In contrast, our projectionbased subspace affinity consistently exhibits a much smoother distribution with a broader dynamic range across all evaluated architectures. This enhanced geometric discriminability provides critical informational signals that enable the effective and robust quantification of intrinsic inter-expert correlations necessary for dynamic expert evolution. Cosine Similarity

Projection-Based Subspace Affinity (Ours)

ViT-B-32

ViT-B-16

ViT-L-14 35

Density

20 15

10 5

5 0

Density

−0.1

0.0

0.1

0.2

0.3

Similarity Value

(a)

0.4

0.5

5 −0.2

0.0

0.2

Similarity Value

(b)

0.4

0.6

−0.2

0.0

0.2

Similarity Value

0.4

0.6

(c)

Figure 9: Density distribution of pairwise similarity scores across 20 task-specific models.

D.4

Additional Results of Hyperparameter Sensitivity

Sensitivity to Rank Ratio ρ: We investigate the influence of the truncation rank ratio ρ ∈ {0.1, 0.2, 0.4, 0.6, 0.8, 1.0} across all architectures in Figure 10. Consistent with the ViT-L/14 observations in the main text, both average accuracy and backward transfer curves across the ViTB/32 and ViT-B/16 architectures exhibit a characteristic performance plateau within the ρ ≤ 0.6 regime, followed by a sharp degradation as ρ approaches 1.0. This pattern corroborates the low-rank prior of fine-tuning updates, indicating that a minimal set of principal parameters is sufficient to represent task-specific functionalities with high fidelity. Moreover, the sharp performance degradation at higher ratios highlights the necessity of compact subspaces for our implicit routing mechanism. While higher rank ratios theoretically preserve more information, they invariably introduce substantial task-irrelevant noise. As these expert subspaces gradually expand to approximate the ambient full parameter space, feature projections become increasingly indistinguishable. This degeneration obscures the discriminability required by our implicit routing mechanism, generating misleading routing decisions and exacerbating catastrophic forgetting. Consequently, a compact ρ = 0.1 not only optimizes parameter efficiency through low-rank truncation but also preserves the geometric specificity of experts via denoising, ensuring precise expert activation and superior multi-task performance. BWT

ACC

BWT

100

ACC 0

100

−2

−4

−6

BWT

−8 70 −10

−10 60

−12

0.1

0.2

0.4

0.6

0.8

0.1

0.2

0.4

0.6

0.8

−8 75 −10 −12 65

−14

1.0

−6 80

−12

−14 40

−4

ACC (%)

−8

ACC (%)

BWT (%)

−6

−2

−4

ACC (%)

−2

BWT (%)

ACC 100

−14 60

1.0

0.1

0.2

0.4

0.6

(a)

(b)

(c)

0.8

1.0

Figure 10: Sensitivity analysis of the rank ratio ρ across the 8-task setting on (a) ViT-B/32, (b) ViT-B/16, and (c) ViT-L/14. Shaded areas denote the standard deviation. Sensitivity to Margin Coefficient β: We investigate the influence of the margin coefficient β ∈ {−1.5, −1.0, −0.5, 0.0, 0.5, 1.0, 1.5, 2.0} across all architectures in Figure 11. Consistent with the ViT-L/14 observations in the main text, both average accuracy and backward transfer scores across the ViT-B/32 and ViT-B/16 models exhibit a monotonic upward trend as β increases, particularly in the regime β < 0.5. This indicates that raising the adaptive threshold relaxes the creation criterion, forming diverse expert ensembles capable of capturing distinct task-specific features. Notably, a characteristic performance plateau emerges within the interval β ∈ [1.0, 2.0], where further elevating the threshold yields diminishing performance gains while unnecessarily inflating the overall parameter footprint due to redundant experts. Across all architectures, the performance inflection point consistently aligns near β = 1.0. Therefore, we adopt β = 1.0 as the default configuration, as it strikes an optimal balance between task-sensitive representational fidelity and architectural parsimony, ensuring robust performance across varying model scales and task sequences. 0 −2

−4

−6

BWT

−10

−12

BWT 0

−2

85 −6 80

ACC (%)

90 −4

BWT (%)

ACC (%)

−8

ACC

100

ACC

100

BWT (%)

ACC (%)

BWT

−4

−6

80 −8

BWT (%)

ACC

100

−8

−14 -1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

−10

-1.5

-1.0

-0.5

0.0

0.5

(a)

(b)

(c)

1.0

1.5

2.0

−10

Figure 11: Sensitivity analysis of the margin coefficient β across the 8-task setting on (a) ViT-B/32, (b) ViT-B/16 models, and (c) ViT-L/14. Shaded areas denote the standard deviation.

Record · ID 134560 · SHA-256 bd61169e9514d282
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.