arXiv CSopen access

MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification

MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification

Weisen Jiang 1 Shuhao Chen 2 3 Sinno Jialin Pan 1

arXiv:2605.14289v1 [cs.LG] 14 May 2026

Abstract

organizations and users often finetune a shared seed model on their own private data, resulting in a collection of specialized experts. While these experts could in principle be unified by aggregating private data to train a single Mixtureof-Experts (MoE) model (Jacobs et al., 1991; Fedus et al., 2022), data sharing is often infeasible due to confidentiality, regulatory, and ethical constraints. This raises a fundamental question (called Privacy-Preserving Mixture-of-Experts Unification): How can we unify independently trained experts into one deployable MoE model while strictly preserving data privacy? Although federated learning (Li et al., 2020; Kairouz et al., 2021; Zhang et al., 2021) enables collaborative training without data sharing, it requires costly synchronized optimization across many clients and often suffers from performance degradation under heterogeneous client data distributions (Wei et al., 2020).

Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are distributed across clients and cannot be shared due to privacy constraints, making unified MoE training challenging. We propose MetaMoE, a privacypreserving framework that unifies independently trained, domain-specialized experts into a single MoE using public proxy data as surrogates for inaccessible private data. Central to MetaMoE is diversity-aware proxy selection, which selects client-domain–relevant and diverse samples from public data to effectively approximate private data distributions and supervise router learning. These proxies are further used to align expert training, improving expert coordination at unification time, while a context-aware router enhances expert selection across heterogeneous inputs. Experiments on computer vision and natural language processing benchmarks demonstrate that MetaMoE consistently outperforms recent privacy-preserving MoE unification methods. Code is available at https://github.com/ ws-jiang/MetaMoE.

Several approaches have been proposed to unify independently finetuned experts. BTM (Li et al., 2022) ensembles expert predictions, allowing embarrassingly parallel training but failing to produce a single deployable model for downstream fine-tuning or RLHF (Ouyang et al., 2022). Model averaging methods such as Model Soup (Wortsman et al., 2022) merge parameters directly, which is computationally efficient but fragile when experts are diverse. BTX (Sukhbaatar et al., 2024) extends this line by transplanting expert feed-forward network (FFN) sublayers into a shared MoE architecture with a learned router. However, its requirement for client-specific data to train this router limits its use in privacy-sensitive scenarios.

1. Introduction Large foundation models (Meta, 2024a;b; Qwen, 2024) have become indispensable across domains such as computer vision (CV) (Radford et al., 2021; Wei et al., 2024) and natural language processing (NLP) (Jiang et al., 2023; 2024; Chen et al., 2024; Lin et al., 2025; Jiang & Pan, 2026). In practice,

When private data cannot be shared, public data can be used as proxies to approximate unavailable private distributions and provide supervision for router learning. Most recently, FlexOlmo (Shi et al., 2025) adopts this strategy by training a router on proxy data while anchoring experts to a public model. However, its reliance on similarity-based proxy selection often produces redundant and narrowly concentrated proxies, limiting coverage of domain-relevant modes and weakening router supervision. Moreover, because experts are trained exclusively on private data and never exposed to proxies, they remain domain-isolated, further exacerbating the mismatch between expert behavior and proxy-based routing. These limitations motivate the need for a more

1 Department of Computer Science and Engineering, The Chinese University of Hong Kong 2 Department of Computer Science and Engineering, Hong Kong University of Science and Technology 3 Department of Computer Science and Engineering, Southern University of Science and Technology. Correspondence to: Weisen Jiang <[email protected]>.

Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).

1

Title Suppressed Due to Excessive Size

2.2. Model Merging and Mixture-of-Experts

principled proxy selection and alignment strategy for expert unification under privacy constraints.

Model Merging (Yang et al., 2024; Wortsman et al., 2022; Ilharco et al., 2023; Yadav et al., 2023; Rame et al., 2023; Li et al., 2024) explores how to combine multiple independently trained models into a single, stronger model without requiring costly joint training. Branch-Train-Merge (BTM) (Li et al., 2022) trains domain experts independently and ensembles outputs at inference time, but does not yield a single unified model (hindering downstream SFT/RLHF (Ouyang et al., 2022) and incurring inference overhead). Model Soup (Wortsman et al., 2022) shows that averaging the weights of finetuned models often improves both accuracy and robustness with no additional inference cost, yet it is fragile when experts diverge in function space, leading to degraded performance in heterogeneous settings. Branch-Train-MiX (BTX) (Sukhbaatar et al., 2024) inserts experts into MoE layers and learns a post-hoc router via additional fine-tuning on private data. More recently, FlexOlmo (Shi et al., 2025) advances this direction in federated settings by anchoring experts to a shared public model and aligning them with router embeddings, where proxy samples are selected for each client based solely on similarity. In contrast, our MetaMoE leverages a relevance-and-diversity criterion via a relevance-weighted DPP to select proxies, and employs a context-aware router, thereby combining heterogeneous experts into a unified MoE model under privacy constraints.

We propose MetaMoE, a privacy-preserving framework for unifying independently trained experts into a single MoE model via diversity-aware proxy selection. Specifically, MetaMoE selects proxy samples using a relevance-weighted determinantal point process (DPP), an extension of standard DPPs (Macchi, 1975; Kulesza & Taskar, 2012) that augments diversity with client-specific relevance scores, ensuring that proxy samples are both diverse and representative of each client domain. Each client then performs proxy-aligned expert training by finetuning FFN sublayers on private data alongside proxy samples. Because the same proxy data are later used for router training, this exposure aligns expert behavior with the supervision available at unification time, mitigating domain isolation and enabling effective coordination across experts. A context-aware router that incorporates both token-level and sequence-level context further improves expert assignment, after which experts’ FFN sublayers are merged into MoE layers and jointly finetuned on the union of proxy data. Experiments on both CV and NLP benchmarks demonstrate that MetaMoE consistently outperforms recent state-of-the-art baselines. Our contributions are three-fold: (i) We study privacy-preserving MoE unification and propose MetaMoE, a framework that unifies independently finetuned experts without sharing private data, and provide formal privacy guarantees. (ii) We propose diversity-aware proxy selection via a relevance-weighted DPP, addressing the limitations of similarity-only proxy sampling for router learning. (iii) We design a proxy-aligned expert training strategy and a context-aware router, and demonstrate through extensive CV and NLP experiments that MetaMoE achieves superior performance over recent methods.

The Mixture-of-Experts (MoE) (Jacobs et al., 1991) framework enhances model flexibility by combining specialized experts via a gating mechanism. Modern MoE architectures (Riquelme et al., 2021; Fedus et al., 2022) scale Transformers by activating only a sparse subset of experts per token, expanding capacity without proportional compute cost (Shazeer et al., 2017). Key designs include the Switch Transformer (Fedus et al., 2022) with top-1 routing and variants (Shazeer et al., 2017; Riquelme et al., 2021; Roller et al., 2021; Dai et al., 2024) exploring top-k gating, stochastic routing, and random assignment for better accuracy, efficiency, and load balance. While MoE is effective for scaling, these methods rely on centralized access to all training data and synchronized training. To overcome these limits under privacy constraints, we propose MetaMoE, which introduces proxy-data-driven routers and decentralized expert training.

2. Related Works 2.1. Federated Learning Federated Learning (FL) (Li et al., 2020; Kairouz et al., 2021; Zhang et al., 2021) enables collaborative training without centralizing raw data. Classical methods such as FedAvg (McMahan et al., 2017) aggregate local updates, with extensions for parameter-efficient tuning (Hu et al., 2022) or differential privacy (Dwork & Roth, 2014). Yet scaling FL to large models (Meta, 2024a;b; Radford et al., 2021) is difficult due to costly synchronization, degraded generalization (Zhang et al., 2023), and privacy leakage from gradients (Wang et al., 2020; Darzi et al., 2024). In contrast, MetaMoE trains experts independently and asynchronously, then merges them via proxy-data-driven routing, avoiding FL’s communication bottlenecks while preserving privacy.

2.3. Determinantal Point Processes (DPPs) Determinantal Point Processes (DPPs) (Macchi, 1975; Kulesza & Taskar, 2012; Lavancier et al., 2015) are probabilistic models designed to capture negative interactions among items, which are useful for diverse subset selection. Formally, consider a discrete set Z = {1, 2, . . . , N }, where each element corresponds to an item with feature vector xi . A DPP defines a probability distribution over all 2N 2

Title Suppressed Due to Excessive Size

public data

Client

Client

train on RelevanceWeighted DPP

Attention

Client

train on RelevanceWeighted DPP

Attention

train on RelevanceWeighted DPP

Attention

Attention

... FFN

FFN

FFN

proxy data

proxy data

proxy data expert

expert

seed model

FFN

expert

train on

FFN

...

Context-Aware Router

FFN

...

Attention

unioned proxy data

FFN Mixture of Experts

Figure 1. Illustration of MetaMoE.

subsets of Z. To specify the distribution, we construct a positive semi-definite kernel matrix L ∈ RN ×N , Lij = κ(xi , xj ), where κ(·, ·) is a kernel function encoding similarity. The probability of sampling a subset S ⊆ Z is: P(S) =

det(LS ) , det(L + I)

owns a private dataset Dp from its local domain. Sharing {Dp }K p=1 directly is prohibited due to privacy constraints. Each client adapts the seed model locally to obtain a domainspecialized expert Mp . Our objective is to unify these experts {Mp }K p=1 into a Mixture-of-Experts (MoE) model MMoE that can be deployed back to all clients such that each client can achieve multi-domain capabilities. In the MoE, experts encode domain-specific knowledge, while the router coordinates the experts to enable effective collaboration.

(1)

where LS is the submatrix indexed by S, det(·) is the determinant, and I is the identity matrix. The denominator det(L + I) is constant with respect to the choice of S, so for subset selection it can be ignored; maximizing the selection probability P(S) is thus equivalent to maximizing det(LS ).

The core challenge lies in training the router. Conventional MoE training (Jacobs et al., 1991; Shazeer et al., 2017) assumes centralized access to all client data, but in our setting only the public dataset D0 is globally accessible. Thus, the router must be learned without directly observing {Dp }K p=1 , while still generalizing across all clients’ domains. We address this challenge by selecting proxy samples from D0 to approximate each Dp , enabling the router to coordinate domain-specific experts in a privacy-preserving manner. Figure 1 illustrates our proposed MetaMoE, consisting of three stages (proxy data selection, proxy-aligned expert training, and context-aware router training) and will be detailed in the following sections.

By reproducing kernel Hilbert space representation (Schölkopf et al., 2001), one may write κ(xi , xj ) = ϕ(xi )⊤ ϕ(xj ) for some feature map ϕ(·). Then det(LS ) equals the squared volume of a parallelotope spanned by {ϕ(xi ) | i ∈ S}, so similar items are less likely to be selected together, promoting diversity.

3. Methodology 3.1. Problem Formulation Denote by M0 a seed model and by D0 a publicly available dataset. We consider K clients, where each client p 3

Title Suppressed Due to Excessive Size Client Samples Proxy Samples (Random)

Client Samples Proxy Samples (FlexOlmo)

(a) Random.

(b) FlexOlmo.

Client Samples Proxy Samples (MetaMoE)

(c) MetaMoE (Ours).

Figure 2. t-SNE visualization of selected proxy samples with random selection, FlexOlmo selection (relevance only), and our selection (relevance + diversity) for Pets with ViT-B/32 as the seed model. As can be seen, our selection yields a more diverse and representative proxy dataset that covers the private-data manifold more effectively (see Section 4.3 for further analysis).

3.2. Proxy Data Selection via Relevance-Weighted DPP

agonal. According to (1) and (3), the unnormalized logprobability of selecting a subset S under relevance-weighted DPP is given by eS ) = 2 P log det(L (4) i∈S log ri + log det(LS ), P where the first term 2 i∈S log ri encourages relevance by favoring samples closer to the client’s data Dp , while the second term log det(LS ) is the standard DPP repulsion term that enforces diversity.

Since client private data is inaccessible for router training, we construct a proxy dataset D̂p for each client p from the public dataset D0 to serve as a proxy for Dp . Effective proxy data should satisfy two criteria: they should be both relevant to the private data Dp and sufficiently diverse to avoid redundancy. Relevance ensures that the proxy samples resemble the private data so that the router trained on proxies learns domain-appropriate decision boundaries, rather than being distracted by unrelated public samples. Diversity, on the other hand, ensures that the selected proxy samples cover different regions of the private-data manifold, rather than clustering around a narrow region of highly similar samples, thereby providing broader coverage and improving the router’s generalization.

The proxy dataset D̂p is selected by e S ), D̂p = arg max log det(L

which yields a relevance-weighted and diverse cover of the private-data manifold. In contrast to FlexOlmo’s relevanceonly proxy selection (Shi et al., 2025), which often collapses onto redundant samples and provides a narrow view of a client’s domain (see Figure 2(b)), the relevance-weighted DPP explicitly balances relevance and diversity. The diversity term discourages near-duplicate proxies, providing the router with a richer supervision signal that better spans the private-domain manifold (see Figure 2(c)) and enables more effective expert coordination. Because all proxies are selected from a public dataset, choosing client-specific proxy subsets does not violate privacy constraints (see Section 3.6 and Appendix E.4).

Determinantal point processes (DPPs) (Macchi, 1975; Kulesza & Taskar, 2012; Lavancier et al., 2015) naturally enforce diversity, but a vanilla DPP (Section 2.3) ignores whether the chosen samples align with the client’s domain Dp . Hence, naively applying a vanilla DPP may select a diverse yet irrelevant proxy dataset. To overcome this, we propose a relevance-weighted DPP, which augments the kernel with client-specific relevance scores: κ̃(xi , xj ) = g(xi , Dp ) κ(xi , xj ) g(xj , Dp ),

(2)

where κ(xi , xj ) measures similarity between public samples (e.g., cosine similarity), and g(xi , Dp ) quantifies the relevance of xi to Dp (e.g., via a classifier distinguishing D0 from Dp , see Appendix A). This yields the relevanceweighted kernel matrix e = Diag(r) L Diag(r), L

(5)

S⊆Z,|S|=m

Since exact maximum a posteriori (MAP) inference in (5) is NP-hard, we adopt efficient greedy algorithms with approximation guarantees (Kulesza & Taskar, 2012; Gillenwater et al., 2012; Han et al., 2017). We first restrict a candidate pool (the top-n public samples ranked by g(xi , Dp )), and then perform greedy MAP inference to construct D̂p by iteratively adding the sample that maximizes the marginal gain e ). Using Cholesky updates (Horn & Johnson, in log det(L D̂p

(3)

where Lij = κ(xi , xj ), r = [g(x1 , Dp ), . . . , g(xN , Dp )], and Diag(r) denotes the diagonal matrix with r on the di4

Title Suppressed Due to Excessive Size

1985), the computational cost is reduced from O(nm3 ) to O(nm) (see Appendix B), where n is the candidate pool size and m is the target proxy set size.

1 T

(l) t=1 zt

PT

is a sequence-level embedding capturing global context (T is the length of x), and λ ∈ [0, 1] is a learnable weight. This blending balances token semantics with broader context cues, and routing distribution is computed as

3.3. Proxy-Aligned Expert Training For each client p, an expert is initialized by branching from the shared seed model M0 . Only the feed-forward network (FFN) sublayers are finetuned using a combination of private data Dp and client-specific proxy data D̂p , while all other parameters remain frozen.

(l)

(l)⊤ (l) (l)⊤ (l) e1 , . . . , z̃t eK ],

π (l) (zt ) = softmax[z̃t

(7)

(l)

where ep is the learnable routing vector for expert p, ini(l) tialized as the mean embedding of Dp ∪ D̂p , i.e., ep = P (1:l) (1:l) 1 (x), with Mp (·) denoting x∈Dp ∪D̂p Mp |Dp ∪D̂p | the first l layers of Mp . This domain-aware initialization injects each expert’s domain characteristics directly into the router, giving it meaningful expert–token priors.

Training experts solely on private data, as in (Shi et al., 2025), produces highly specialized models but leads to domain isolation: since the router is trained later without access to private data, it struggles to coordinate experts adapted to heterogeneous domains.

3.5. Final MoE Training

We address this mismatch by incorporating proxy data during expert training. Since router learning relies exclusively on proxy data, exposing each expert to its corresponding proxies calibrates expert representations to the same data distribution used for routing. Specifically, the router is trained on the union D̂1 ∪ · · · ∪ D̂K ; aligning expert training with this supervision improves routing compatibility while preserving data privacy. Compared with FlexOlmo, which trains experts only on private data, proxy-aligned training in MetaMoE preserves domain-specific expertise while enabling more effective expert coordination.

At the final stage, we aggregate all proxy datasets D̂1 ∪ · · ·∪ D̂K and finetune the unified MoE model. This process updates the router while jointly adapting the FFN experts under supervision from proxy data, ensuring that experts are not treated as isolated components but instead operate cohesively within an MoE architecture. Hence, the resulting model MMoE unifies domain-specific expertise with a privacy-preserving router, yielding a unified MoE model that generalizes across heterogeneous client domains. The overall procedure of MetaMoE is summarized in Algorithm 1.

3.4. Context-Aware Router for Expert Unification

3.6. Privacy Analysis

After collecting domain-specific experts {Mp }K p=1 , we merge their FFN sublayers into MoE modules at each Transformer layer. We denote FFN(l) p (·) as the l-th FFN sub-

We specify the threat model and provide formal privacy guarantees for the artifacts communicated by MetaMoE. Throughout the MetaMoE pipeline, each client p communicates only the following artifacts to the central server: (i) indices of selected public proxy samples (a subset of D0 ), (ii) final expert weights (FFN sublayers of Mp ), and (l) (iii) routing vectors ep (mean embeddings used to initialize the router, (7)). Raw private samples, gradients, and intermediate activations are never shared. This privacy model, commonly referred to as data residency (McMahan et al., 2017), is standard in privacy-preserving collaborative learning. Notably, MetaMoE is strictly more conservative than federated learning (FL) (Li et al., 2020; Kairouz et al., 2021; Zhang et al., 2021): it transmits only final expert weights in a single one-shot communication, whereas FL iteratively exchanges gradient updates that are susceptible to model inversion attacks (Wang et al., 2020).

(l)

layer of the p-th expert Mp and let zt be the t-th token representation of input sequence x in the l-th layer. The corresponding MoE module is then formulated as (l)

(l)

MMoE (zt ) =

X

(l)

(l)

[π (l) (zt )]p · FFN(l) p (zt ), (6) (l)

p∈Top-k(π (l) (zt ))

where π (l) (zt ) is the router’s score distribution over experts, and Top-k(·) is the top-k selection. (l)

Conventional routers rely solely on zt , making routing decisions based on token-level features. However, this can be unreliable: tokens with similar surface forms may belong to different domains and require different experts. Such routing collisions are particularly problematic here, since the router is trained only on proxies and never directly observes the true client data distributions.

Among the three communicated artifacts, proxy indices are deterministic functions of public data and carry no private information. Expert weights, as outputs of deep network training, are shared in the same manner as in FL. The rout(l) ing vectors ep , while computed over both private and proxy data, also carry negligible private information, as we for-

To mitigate this, we introduce a context-aware router. In(l) stead of routing purely from zt , we form a blended rep(l) (l) (l) (l) resentation z̃t = (1 − λ) zt + λ zx , where zx = 5

Title Suppressed Due to Excessive Size

Algorithm 1 MetaMoE. Require: public dataset D0 ; client private datasets {Dp }K p=1 ; kernel κ(·, ·); proxy dataset size m; optional candidate pool size n (m ≤ n ≪ |D0 |); 1: for each client p = 1, 2, . . . , K do 2: // select proxy samples by relevance-weighted DPP 3: Train a binary classifier g(x, Dp ) to distinguish D0 vs. Dp ; 4: Compute relevance r(x) = g(x, Dp ) for each x ∈ D0 ; 5: Let Cp ⊆ D0 be the top-n elements of D0 ordered by r(x); 6: Construct kernel matrix L ∈ Rn×n with Lij = κ(xi , xj ) for each xi , xj ∈ Cp ; e = Diag(r) L Diag(r); 7: Form r = [ r(x) ]x∈Cp and relevance-weighted kernel matrix L ⋆ 8: Greedily build D̂p of size m by iteratively adding x = arg max log det(L̃D̂p ∪{x}) − log det(L̃D̂p ); x∈Cp \D̂p

// proxy-aligned expert training Finetune the FFN sublayers of the p-th client model on D̂p ∪ Dp , and freeze all other sublayers; P (l) (1:l) 1 11: For each layer l, compute router vector ep = |D̂ ∪D (x); x∈D̂p ∪Dp Mp p p| 12: end for 13: // build the MoE model with context-aware router 14: Merge all clients’ FFN sublayers into a single MoE model MMoE ; (l) 15: Collect {ep }K p=1 for all layers l = 1, . . . , L to initialize the router; 9: 10:

(l)

16: Compute the routing distribution π (l) (zt ) for each layer l using (7); 17: Finetune MMoE on the union of proxy datasets D̂1 ∪ · · · ∪ D̂K ; 18: return Trained MoE model MMoE .

(l) m) ep − m µproxy /N , which cannot be evaluated without knowing N . Since µpriv is the coarsest possible summary of private data (a single vector), finer-grained distributional properties (e.g., variance, class proportions, individual sam(l) ples) are a fortiori unrecoverable from ep .

mally establish below. We analyze the routing vector for a fixed client p and layer l. Let N = |Dp | and m = |D̂p | denote the private and (1:l) proxy dataset sizes, respectively, and let f (·) = Mp (·) denote the first l layers of Mp used as thePencoder. Define the mean private embedding µpriv = N1 x∈Dp f (x) and P 1 the mean proxy embedding µproxy = m x∈D̂p f (x). The routing vector (7) can be decomposed as e(l) p =

N m µpriv + µproxy . N +m N +m

(iii) MetaMoE exposes strictly less private information than FlexOlmo. FlexOlmo (Shi et al., 2025) shares per-expert routing embeddings computed as mean embeddings over private data alone (Section 3.3.2 of the FlexOlmo paper), directly exposing µpriv . In contrast, MetaMoE dilutes the private component by averaging over both private and proxy data (8), providing a strictly stronger privacy guarantee.

(8)

(l)

An adversary observes ep and knows µproxy , m, and f (all derived from public information), but does not know N (the private dataset size is never communicated). We establish three formal guarantees (full derivations in Appendix F):

4. Experiments 4.1. Experiments on CV Tasks

(i) Per-sample sensitivity is bounded by O(1/m). Let B = maxx ∥f (x)∥2 be the embedding norm bound. Replacing any single private sample xk ∈ Dp with an arbitrary x′k ′(l) yields a perturbed routing vector ep satisfying ′(l) ∥e(l) p −ep ∥2 =

Datasets. We evaluate on three benchmarks from distinct domains: (i) Pets (Parkhi et al., 2012), with 37 cat and dog breeds, (ii) Flowers (Nilsback & Zisserman, 2008), with 102 flower categories, and (iii) EuroSAT (Helber et al., 2019), with 10 land-use classes from satellite imagery. Each dataset serves as a client domain, covering fine-grained categorization, natural object recognition, and remote sensing. As the public dataset D0 , we adopt ImageNet (Deng et al., 2009), which is a large-scale visual corpus providing 1.28M images across 1,000 categories.

∥f (x′k )−f (xk )∥2 2B 2B ≤ ≤ , (9) N +m N +m m

where the last inequality uses N + m ≥ m. Thus the proxy set size m alone provides a universal sensitivity ceiling, independent of both N and the domain gap. (ii) The mean private embedding µpriv is unrecoverable. From (8), isolating µpriv requires computing µpriv = (N +

Models. We adopt CLIP ViT-B/32 and ViT-B/16 (Rad6

Title Suppressed Due to Excessive Size

ford et al., 2021) as the seed models, which consist of a transformer-based image encoder and a text encoder.

architecture. In contrast to BTX and FlexOlmo, which also employ MoE-style unification but rely on proxy data without explicit distribution alignment, MetaMoE yields clear gains by combining relevance-weighted DPP-selected proxy subsets with proxy-aligned expert training, allowing routers to coordinate experts more effectively. Quantitatively, MetaMoE achieves an average accuracy of 94.52% with CLIP ViT-B/32 and 96.24% with CLIP ViT-B/16, outperforming the strongest baseline FlexOlmo (92.92% and 93.53%) by about 1.6 and 2.7 points, respectively.

Implementation Details. For each client, we build a proxy dataset by first selecting a candidate pool of n = 3000 ImageNet samples that are most similar to its private data according to the relevance score g(x, Dp ), then choosing m = 500 proxy samples via relevance-weighted DPP with the cosine similarity kernel κ(xi , xj ) = cos(zi , zj ), where zi is xi ’s embedding extracted from M0 . Client models are initialized from the seed model M0 and finetuned on both private and proxy data. For proxy-aligned expert training, we finetune the FFN sublayers of the visual encoder using LoRA (Hu et al., 2022) (rank 16, scaling factor α = 32) for 10 epochs with the SGD optimizer (momentum 0.9, weight decay 0.0001, learning rate 0.01, batch size 128, constant learning rates schedule). For router training, we adopt top-1 routing and finetune for 5 epochs using SGD (lr=0.001).

Table 1. Accuracy of CV Tasks when using CLIP ViT-B/32 as the seed model.

Baselines. We compare our method against various baselines: (i) ZeroShot directly evaluates the seed model M0 without any adaptation, providing a capacity-only lower bound. (ii) BTM (Li et al., 2022) ensembles predictions from independently trained experts at inference, but does not produce a unified model for downstream use. (iii) ModelSoup (Wortsman et al., 2022) merges experts by averaging their weights into a single model, avoiding inference overhead from ensembling but losing specialization when experts diverge. (iv) BTX (Sukhbaatar et al., 2024) integrates experts by inserting their FFN sublayers into MoE layers, averaging the remaining parameters, and then fine– tuning the mixed model on public data (since private data are unavailable), following the setup of FlexOlmo (Shi et al., 2025). (v) FlexOlmo (Shi et al., 2025) aligns experts without centralizing private data by anchoring them to a shared public model and introducing per-expert router embeddings, which are later finetuned on proxy data selected by similarity to private domains. (vi) Separate Experts evaluates each independently finetuned expert across all domains. (vii) UnrestrictedMoE trains a unified MoE model directly on the merged private datasets from all clients, thereby achieving strong performance but breaking privacy constraints.

Pets

Flowers

EuroSAT

Average

UnrestrictedMoE

92.45

96.43

98.15

95.68

ZeroShot Expert I (Pets) Expert II (Flowers) Expert III (EuroSAT)

85.77 92.40 84.25 82.64

61.59 59.03 96.91 52.42

29.81 22.74 27.21 97.91

59.06 58.06 69.46 77.66

BTM ModelSoup BTX FlexOlmo MetaMoE

90.81 87.90 88.44 91.36 91.91

85.10 70.52 75.07 90.62 93.67

95.07 64.19 59.38 96.79 97.98

90.33 74.20 74.30 92.92 94.52

Table 2. Testing Accuracy of CV Tasks when using CLIP ViT-B/16 as the seed model. Pets

Flowers

EuroSAT

Average

UnrestrictedMoE

94.30

97.73

98.23

96.75

ZeroShot Expert I (Pets) Expert II (Flowers) Expert III (EuroSAT)

88.53 94.44 85.99 86.56

68.09 65.69 98.13 60.05

34.00 30.10 24.05 98.43

63.54 63.41 69.39 81.68

BTM ModelSoup BTX FlexOlmo MetaMoE

93.21 89.94 89.89 94.09 94.22

84.65 74.18 78.52 89.40 97.08

97.38 74.14 75.20 97.11 97.41

91.75 79.42 81.20 93.53 96.24

4.2. Experiments on NLP Tasks Datasets. We evaluate on three benchmarks for commonsense reasoning: (i) CommonsenseQA (Talmor et al., 2019) (denoted as CSQA), which tests general world knowledge through discrimination among semantically related concepts, (ii) CosmosQA (Huang et al., 2019), which emphasizes narrative reasoning by requiring inference of implicit causes and effects, and (iii) SocialIQA (Sap et al., 2019), which focuses on social commonsense involving human actions, motivations, and social implications. Proxy data are selected from Alpaca (Taori et al., 2023), a publicly available collection of 52K instruction–response pairs that spans diverse instruction-tuning tasks. Note that the public dataset (Alpaca) has no domain overlap with the client datasets.

Results. Tables 1 and 2 report the testing accuracy on the three client domains when using CLIP ViT-B/32 and CLIP ViT-B/16 as the seed model, respectively. Across both backbones, our proposed MetaMoE consistently outperforms all privacy-preserving baselines. Compared with BTM, which ensembles predictions from independent experts without parameter sharing, MetaMoE integrates experts at the parameter level and leverages a router to coordinate experts, thereby achieving better performance. Compared with ModelSoup—which averages model parameters—MetaMoE achieves much higher accuracy by explicitly retaining expert specialization within a unified MoE

Models. We evaluate on two LLaMA models across dif7

Title Suppressed Due to Excessive Size Table 3. Accuracy of NLP Tasks when using LLaMA-3.2-3B as the seed model. CSQA

CosmosQA

SocialIQA

Average

UnrestrictedMoE

75.51

78.39

71.80

75.23

ZeroShot Expert I (CSQA) Expert II (CosmosQA) Expert III (SocialIQA)

62.49 74.94 63.06 65.44

62.68 70.18 78.69 68.04

56.19 60.54 58.96 72.42

60.45 68.55 66.90 68.63

BTM ModelSoup BTX FlexOlmo MetaMoE

74.61 73.71 69.62 73.30 74.94

75.44 75.24 72.06 73.33 76.05

68.17 71.75 71.75 70.88 72.26

72.74 73.57 71.14 72.50 74.42

MoE further benefits from DPP-selected proxies that approximate private distributions more accurately, leading to more effective routing. Quantitatively, MetaMoE achieves 74.42% average accuracy on LLaMA-3.2-3B and 81.59% on LLaMA-3.1-8B, outperforming the strongest baselines. 4.3. Effectiveness of Relevance-Weighted DPP We conduct an ablation study to evaluate the effect of incorporating relevance-weighted DPP into proxy data selection (Section 3.2) for MetaMoE and the recent method FlexOlmo. The configuration denoted as “✗” in Table 5 corresponds to FlexOlmo’s original similarity-based approach, which selects public samples based on their relevance scores to the client’s private data, without enforcing diversity. As shown in Table 5, DPP consistently boosts accuracy across both NLP and CV tasks, highlighting its effectiveness as a selection strategy. For FlexOlmo, incorporating DPP leads to clear accuracy gains across all tasks (e.g., +0.85 on LLaMA3.2-3B and +2.32 on LLaMA-3.1-8B), demonstrating that existing methods can benefit significantly from our proposed relevance-weighted DPP proxy selection. MetaMoE further boosts these benefits, with consistent improvements in all settings, ultimately achieving the best overall performance. These findings demonstrate that relevance-weighted DPP is essential for selecting proxy samples that effectively train the router to coordinate experts.

Table 4. Accuracy of NLP Tasks when using LLaMA-3.1-8B as the seed model. CSQA

CosmosQA

SocialIQA

Average

UnrestrictedMoE

81.90

85.70

76.41

81.34

ZeroShot Expert I (CSQA) Expert II (CosmosQA) Expert III (SocialIQA)

69.37 81.24 71.42 73.63

75.98 77.29 86.23 76.98

57.57 69.19 67.86 78.10

67.64 75.91 75.17 76.24

BTM ModelSoup BTX FlexOlmo MetaMoE

81.24 80.26 75.76 75.18 81.33

80.84 84.25 81.04 81.11 85.80

77.28 77.02 73.39 76.10 77.64

79.79 80.51 76.73 77.46 81.59

ferent scales: LLaMA-3.2-3B (Meta, 2024b), a lightweight 3B model for efficiency-critical settings, and LLaMA-3.18B (Meta, 2024a), a larger 8B model offering stronger performance with higher compute cost.

Table 5. Accuracy of FlexOlmo and MetaMoE with and without relevance-weighted (RW) DPP. NLP

RW DPP

Implementation Details. For each client, we form a proxy dataset by first selecting a candidate pool of n = 3000 Alpaca samples most similar to its private data, then choosing m = 500 proxies via relevance-weighted DPP with the cosine similarity kernel κ(xi , xj ) = cos(zi , zj ), where zi is x’s embedding extracted from M0 . Client models are initialized from the seed model M0 . For proxy-aligned expert training, we finetune the model’s FFN sublayers on both private and proxy data using LoRA (Hu et al., 2022) (rank 16, α = 32) for 10 epochs with AdamW (Loshchilov & Hutter, 2019) (learning rate 0.0001, batch size 32, no weight decay, constant schedule with 200 warm-up steps). For router training, we adopt top-1 routing and finetune for 1 epoch with AdamW (learning rate 0.0001, batch size 32).

CV

LLaMA-3.2-3B

LLaMA-3.1-8B

ViT-B/32

ViT-B/16

lm lexO

o

✗ ✓

72.50 73.35

77.46 79.78

92.92 93.20

93.53 94.38

E

✗ ✓

73.60 74.42

80.32 81.59

94.12 94.52

95.39 96.24

F

aMo

Met

Visualization of Selected Proxy Samples. In Figure 2, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize the proxy samples selected by three different selection strategies: random sampling, similarity-based selection used in FlexOlmo, and our MetaMoE based on relevance-weighted DPP. As can be seen from Figure 2(a), random selection fails to capture either relevance or diversity, often yielding samples that are poorly aligned with the private data distribution. FlexOlmo improves alignment by selecting samples highly similar to the private domain, but the resulting proxies lack diversity: many chosen samples cluster in a narrow region of the data space, providing limited coverage of the right side of the private-data manifold (Figure 2(b)). In contrast, our MetaMoE explicitly balances relevance and diversity through the relevance-weighted DPP kernel. As shown in Figure 2(c), the resulting proxy set not only anchors closely to the private domain but also forms a diverse cover of the data space. This richer proxy distribution allows the router to discriminate across heterogeneous con-

Results. Tables 3 and 4 report the testing accuracy of NLP tasks with LLaMA-3.2-3B and LLaMA-3.1-8B as the seed models, respectively. Across both backbones, MetaMoE consistently outperforms all privacy-preserving baselines. Unlike BTM, which ensembles outputs without parameter sharing, or ModelSoup, which only averages parameters, MetaMoE preserves expert specialization within a unified MoE model and enables router-based coordination. Compared with BTX and FlexOlmo, which also use proxy data but lack relevance-diverse alignment mechanisms, Meta8

Title Suppressed Due to Excessive Size Table 6. Average accuracy of MetaMoE with and without contextaware (CA) router across NLP and CV tasks. NLP

CA Router ✗ ✓

primarily provide coordination signals for router learning, rather than effective supervision for expert training.

CV

LLaMA-3.2-3B

LLaMA-3.1-8B

ViT-B/32

ViT-B/16

72.62 74.42

79.41 81.59

93.92 94.52

95.94 96.24

In contrast, MetaMoE combines private data for expert specialization with proxy data for alignment and routing, achieving significantly higher accuracy. These results demonstrate that private-domain data are essential for learning strong experts, while proxy data play a complementary role by facilitating effective router learning.

Table 7. Average accuracy of MetaMoE with and without ProxyAligned (PA) Expert Training across NLP and CV tasks. NLP

PA Training ✗ ✓

CV

LLaMA-3.2-3B

LLaMA-3.1-8B

ViT-B/32

ViT-B/16

72.71 74.42

80.99 81.59

92.98 94.52

94.09 96.24

Train solely on proxy data

4.6. Robustness to the Choice of Public Dataset Our main NLP experiments use Alpaca (Taori et al., 2023) as the public dataset D0 . To verify that MetaMoE does not depend on a specific public corpus, we conduct an additional experiment using OpenOrca (Lian et al., 2023) as an alternative public dataset. OpenOrca is a large-scale collection of augmented FLAN data (Longpre et al., 2023) with GPTgenerated responses, differing substantially from Alpaca in both scale and construction methodology. All other settings (seed models, client datasets, hyperparameters) remain identical to those in Section 4.2.

MetaMoE

As shown in Table 8, MetaMoE consistently outperforms all baselines when using OpenOrca, confirming that a generalpurpose public dataset suffices without careful domainspecific curation. Combined with the results using Alpaca and the ImageNet experiment on CV tasks (Section 4.1), these results demonstrate that MetaMoE is robust across diverse public dataset choices in both modalities.

Figure 3. Comparison of MetaMoE with training solely on proxy data in the CV setting with CLIP ViT-B/16.

texts. Consequently, our MetaMoE provides a stronger and more aligned proxy of the unavailable private data, which translates into consistent performance gains (Table 5).

Table 8. Accuracy of NLP tasks using OpenOrca as the public dataset with LLaMA-3.2-3B as the seed model.

4.4. Ablation Study We examine the effectiveness of the context-aware router (Section 3.4). As shown in Table 6, incorporating sequencelevel context consistently yields higher accuracy across all evaluated datasets and backbones. These gains demonstrate that incorporating sequence-level context into the router enables more reliable expert assignment.

ModelSoup BTM BTX FlexOlmo MetaMoE

We assess the effectiveness of proxy-aligned expert training (Section 3.3) by comparing experts trained only on private data with those additionally exposed to client-specific proxy data. Experiments are conducted on both CV and NLP tasks. As shown in Table 7, incorporating proxy data yields consistent gains, suggesting proxy-aligned training enables more effective expert collaboration.

CSQA

CosmosQA

SocialIQA

Average

73.71 74.61 71.66 73.63 75.18

75.24 75.44 72.90 73.97 77.12

71.75 68.17 69.40 70.73 72.06

73.57 72.74 71.32 72.78 74.79

5. Conclusion We proposed MetaMoE, a privacy-preserving framework for unifying independently trained experts into a single Mixtureof-Experts model. By leveraging diversity-aware proxy selection for router learning and proxy-aligned expert training, MetaMoE enables effective expert coordination without sharing private data. We further provide formal privacy guarantees showing that the shared routing vectors have bounded per-sample sensitivity and do not reveal recoverable domainlevel information about private data. Experiments on computer vision and natural language processing benchmarks show consistent improvements over recent methods, underscoring the importance of diversity-aware proxies for privacy-preserving MoE unification.

4.5. Effect of Expert Training without Private Data To isolate the role of private data, we conduct an ablation on CV tasks with CLIP ViT-B/16 in which experts are trained solely on proxy samples and never observe client private data. As shown in Figure 3, training experts exclusively on proxy data results in substantial accuracy degradation across all domains. This confirms that proxy samples alone are insufficient for learning domain-specific expertise and 9

Title Suppressed Due to Excessive Size

Acknowledgements

Feng, W., Hao, C., Zhang, Y., Han, Y., and Wang, H. Mixture-of-LoRAs: An efficient multitask tuning for large language models. Preprint arXiv:2403.03432, 2024.

The research work described in this paper was conducted in the JC STEM Lab of Machine Learning and Symbolic Reasoning funded by The Hong Kong Jockey Club Charities Trust.

Gillenwater, J., Kulesza, A., and Taskar, B. Near-optimal MAP inference for determinantal point processes. In Neural Information Processing Systems, 2012.

Impact Statement

Han, I., Kambadur, P., Park, K., and Shin, J. Faster greedy MAP inference for determinantal point processes. In International Conference on Machine Learning, 2017.

This paper proposes a framework for privacy-preserving model unification. While MetaMoE enforces data residency and provides formal guarantees on the shared routing vectors, deploying it in high-stakes domains (e.g., healthcare, finance) may require additional safeguards such as formal differential privacy mechanisms or external audits. We encourage practitioners to evaluate the privacy properties of all communicated artifacts in the context of their specific regulatory requirements.

Helber, P., Bischke, B., Dengel, A., and Borth, D. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. Horn, R. A. and Johnson, C. R. Matrix Analysis. Cambridge University Press, 1985.

References

Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.

Chen, S., Jiang, W., Lin, B., Kwok, J., and Zhang, Y. RouterDC: Query-based router by dual contrastive learning for assembling large language models. In Neural Information Processing Systems, 2024.

Huang, L., Le Bras, R., Bhagavatula, C., and Choi, Y. CosmosQA: Machine reading comprehension with contextual commonsense reasoning. In Conference on Empirical Methods in Natural Language Processing, 2019.

Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Annual Meeting of the Association for Computational Linguistics, 2024.

Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.

Darzi, E., Dubost, F., Sijtsema, N. M., and van Ooijen, P. M. Exploring adversarial attacks in federated learning for medical imaging. IEEE Transactions on Industrial Informatics, 2024.

Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 1991. Jiang, W. and Pan, S. J. MetaDefense: Defending finetuning based jailbreak attack before and during generation. In Neural Information Processing Systems, 2026.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.

Jiang, W., Zhang, Y., and Kwok, J. Effective structuredprompting by meta-learning and representitive verbalizer. In International Conference on Machine Learning, 2023.

Dwork, C. and Roth, A. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 2014.

Jiang, W., Shi, H., Yu, L., Liu, Z., Zhang, Y., Li, Z., and Kwok, J. Forward-backward reasoning in large language models for mathematical verification. In Findings of the Association for Computational Linguistics, 2024.

Fan, D., Messmer, B., Doikov, N., and Jaggi, M. On-device collaborative language modeling via a mixture of generalists and specialists. In International Conference on Machine Learning, 2025.

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D’Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu,

Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 2022. 10

Title Suppressed Due to Excessive Size

J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konecný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Qi, H., Ramage, D., Raskar, R., Raykova, M., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramèr, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 2021.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 2017. Meta. Introducing Llama 3.1: Our most capable models to date. Preprint, 2024a. Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. Preprint, 2024b.

Kulesza, A. and Taskar, B. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 2012.

Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.

Lavancier, F., Møller, J., and Rubak, E. Determinantal point process models and statistical inference. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2015.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Neural Information Processing Systems, 2022.

Li, M., Gururangan, S., Dettmers, T., Lewis, M., Althoff, T., Smith, N. A., and Zettlemoyer, L. Branch-Train-Merge: Embarrassingly parallel training of expert language models. Preprint arXiv:2208.03306, 2022. Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020.

Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.

Li, T., Jiang, W., Liu, F., Huang, X., and Kwok, J. T. Learning scalable model soup on a single GPU: An efficient subspace training strategy. In European Conference on Computer Vision, 2024.

Qwen. Qwen2.5: A party of foundation models. Technical Report, 2024. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.

Li, Y., Meng, F., Zhang, M., Zhu, S., Wang, S., and Xu, M. LoRASuite: Efficient LoRA adaptation across large language model upgrades. In Neural Information Processing Systems, 2025. Lian, W., Goodson, B., Pentland, E., Cook, A., Vong, C., and Teknium. OpenOrca: An open dataset of GPT augmented FLAN reasoning traces, 2023.

Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., and Cord, M. Rewarded Soups: Towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Neural Information Processing Systems, 2023.

Lin, B., Jiang, W., Xu, Y., Chen, H., and Chen, Y.-C. PARM: Multi-objective test-time alignment via preference-aware autoregressive reward model. In International Conference on Machine Learning, 2025.

Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. In Neural Information Processing Systems, 2021.

Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, A. Designing data and methods for effective instruction tuning. Preprint arXiv:2301.13688, 2023.

Roller, S., Sukhbaatar, S., Szlam, A., and Weston, J. E. Hash layers for large sparse models. In Neural Information Processing Systems, 2021.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.

Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. SocialIQA: Commonsense reasoning about social interactions. In Conference on Empirical Methods in Natural Language Processing, 2019.

Macchi, O. The coincidence approach to stochastic point processes. Advances in Applied Probability, 1975. 11

Title Suppressed Due to Excessive Size

Schölkopf, B., Herbrich, R., and Smola, A. J. A generalized representer theorem. In International Conference on Computational Learning Theory, 2001.

models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.

Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. TIES-Merging: Resolving interference when merging models. In Neural Information Processing Systems, 2023. Yang, E., Shen, L., Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. Model merging in LLMs, MLLMs, and beyond: Methods, theories, applications and opportunities. Preprint arXiv:2408.07666, 2024.

Shi, W., Bhagia, A., Farhat, K., Muennighoff, N., Walsh, P., Morrison, J., Schwenk, D., Longpre, S., Poznanski, J., Ettinger, A., Liu, D., Li, M., Groeneveld, D., Lewis, M., tau Yih, W., Soldaini, L., Lo, K., Smith, N. A., Zettlemoyer, L., Koh, P. W., Hajishirzi, H., Farhadi, A., and Min, S. FlexOlmo: Open language models for flexible data use. In Neural Information Processing Systems, 2025.

Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., and Gao, Y. A survey on federated learning. Knowledge-Based Systems, 2021. Zhang, X., Kang, Y., Chen, K., Fan, L., and Yang, Q. Trading off privacy, utility, and efficiency in federated learning. ACM Transactions on Intelligent Systems and Technology, 2023.

Sukhbaatar, S., Golovneva, O., Sharma, V., Xu, H., Lin, X. V., Rozière, B., Kahn, J., Li, D., tau Yih, W., Weston, J., and Li, X. Branch-Train-MiX: Mixing expert LLMs into a mixture-of-experts LLM. In Conference on Language Modeling, 2024. Talmor, A., Herzig, J., Lourie, N., and Berant, J. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In North American Chapter of the Association for Computational Linguistics, 2019. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An instruction-following LLaMA model. Technical Report, 2023. Van der Maaten, L. and Hinton, G. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008. Wang, H., Sreenivasan, K., Rajput, S., Vishwakarma, H., Agarwal, S., Sohn, J.-y., Lee, K., and Papailiopoulos, D. Attack of the tails: Yes, you really can backdoor federated learning. In Neural Information Processing Systems, 2020. Wei, K., Li, J., Ding, M., Ma, C., Yang, H. H., Farokhi, F., Jin, S., Quek, T. Q., and Poor, H. V. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 2020. Wei, Y., Fu, S., Jiang, W., Zhang, Z., Zeng, Z., Wu, Q., Kwok, J. T., and Zhang, Y. GITA: Graph to visual and textual integration for vision-language graph reasoning. In Neural Information Processing Systems, 2024. Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model Soups: averaging weights of multiple fine-tuned 12

Title Suppressed Due to Excessive Size

A. Computation of Relevance Score Following FlexOlmo (Shi et al., 2025), we compute the relevance score g(x, Dp ) of a public sample x ∈ D0 with respect to a client dataset Dp by training a binary classifier to distinguish Dp from D0 . Specifically, we construct a training set by labeling samples from Dp as positive and randomly drawing 10K samples from D0 as negative. We append a classification head to the last hidden layer of the seed model, and finetune this classifier to distinguish whether a sample comes from the client dataset Dp (positive) or the public dataset D0 (negative). After training, we apply the classifier to every x ∈ D0 . The predicted probability that x belongs to Dp is used as the relevance score: g(x, Dp ) = P[classifier predicts x ∈ Dp ].

(10)

This score quantifies the extent to which each public sample is representative of the private domain. In practice, we rank all x ∈ D0 by g(x, Dp ) and retain the top-n candidates for subsequent proxy selection.

B. Cholesky Updates for Efficient Inference In this appendix, we show how to make greedy DPP MAP inference computationally efficient for large-scale proxy selection. The main challenge is the repeated evaluation of determinants for growing kernel submatrices. A naive implementation recomputes a full factorization for each of the n candidates at every greedy step, incurring O(nm3 ) time per iteration when the current subset has size m. We address this bottleneck by maintaining a Cholesky factorization of the current kernel submatrix and updating it incrementally as new elements are considered. With this strategy, scoring all n candidates in a greedy iteration requires only O(nm) time. The derivation below details this update mechanism and explains its implications for scalability. e S . At each step of Naive Computation. Consider a candidate subset S ⊆ {1, . . . , n} with associated kernel submatrix L greedy MAP inference, one must compute e S∪{x} ) det(L for a new candidate x ∈ / S. If performed directly, this requires recomputing the determinant of an (|S| + 1) × (|S| + 1) matrix, incurring O(m3 ) time per evaluation. Over all n candidates and m greedy steps, the total cost for each greedy step scales as O(nm3 ), which is infeasible when both n and m are large. Cholesky Factorization. To avoid redundant recomputation, we exploit the Cholesky decomposition (Horn & Johnson, 1985). Suppose the current kernel submatrix admits a decomposition e S = PP⊤ , L where P is a lower-triangular matrix of size |S| × |S|. The determinant is then easily obtained as eS ) = det(L

|S| Y

P2ii ,

i=1

so that the computational burden shifts from determinant computation to maintaining P. Incremental Update.

When a new element x is considered, the augmented kernel matrix can be written in block form: # " eS L k e LS∪{x} = ⊤ e , (11) k Lxx

where k contains the similarities between x and items in S. Instead of recomputing a full factorization, we extend P by one row and column:   P 0 P′ = ⊤ , y σ e S∪{x} . Expanding both sides and equating with (11) gives so that P′ P′⊤ = L Py = k,

e xx − ∥y∥2 . σ2 = L 2 13

Title Suppressed Due to Excessive Size

Thus, the update reduces to solving a triangular system (for y) and computing a residual variance (for σ 2 ). Both steps are efficient: solving a triangular system costs O(m), and computing a norm is linear in m as well. Hence, each iteration of greedy MAP inference only requires a total time of O(nm) for searching over the n candidates. This makes the approach scalable to large public datasets, while still retaining the DPP’s balance between relevance and diversity.

C. Computer Vision Datasets Figure 4 presents randomly sampled images from the three client datasets used in the CV experiments: Pets (Parkhi et al., 2012), Flowers (Nilsback & Zisserman, 2008), and EuroSAT (Helber et al., 2019). These examples illustrate the visual diversity across domains, ranging from fine-grained object recognition of dog and cat breeds (Pets), to natural scene categorization of flower species (Flowers), and remote sensing imagery for land-use classification (EuroSAT). Such heterogeneity highlights the challenge of unifying domain-specialized experts into a single model while preserving privacy and ensuring robust multi-domain generalization. We adopt ImageNet (Deng et al., 2009) as the public dataset from which proxy samples are drawn. Figure 5 shows randomly sampled examples from ImageNet.

(a) Pets. (b) Flowers. (c) EuroSAT. Figure 4. Sample images from the three client domains: Pets, Flowers, and EuroSAT.

Figure 5. Sample images from ImageNet.

D. Natural Language Processing Datasets The client-side NLP datasets comprise CommonsenseQA (Talmor et al., 2019), CosmosQA (Huang et al., 2019), and SocialIQA (Sap et al., 2019). These cover complementary reasoning skills: CommonsenseQA requires grounding abstract questions in everyday knowledge; CosmosQA emphasizes multi-sentence comprehension with causal and temporal reasoning; and SocialIQA targets social motivations and reactions in human interactions. Examples 1–3 show two representative 14

Title Suppressed Due to Excessive Size

samples from CommonsenseQA, CosmosQA, and SocialIQA, respectively. Such diversity highlights the challenge of integrating domain-specialized experts in NLP while ensuring broad generalization across reasoning styles and task formats. As the public corpus D0 , we use Alpaca (Taori et al., 2023), an open-domain instruction–response dataset ranging from factual queries to reasoning and generation. Example 4 illustrates three instances from Alpaca. Example 1: Samples from CommonsenseQA (Talmor et al., 2019) Question: The fox walked from the city into the forest, what was it looking for? (A) pretty flowers (B) hen house (C) natural habitat (D) storybook (E) dense forest Answer: (C) natural habitat Question: To learn one must have the right book, to work efficiently what must one have? (A) improve yourself (B) become knowledgeable (C) have tools (D) persistence (E) have more knowledge Answer: (C) have tools

Example 2: Samples from CosmosQA (Huang et al., 2019) Let me be clear, everything is good. Having said that, I ’ve been a little preoccupied lately, as Leslie threw out her back on Sunday. It’s been an interesting, and marriage-solidifying 36 hours. Sunday afternoon, around 2 pm, Leslie and I were hanging in the library, sans kids (they were with their father), listening to music and talking. Question: How did Leslie’s throwing out of her back help to solidify your marriage? (A) Leslie realized that I am really good at taking care of people and wants to keep me around for that reason (B) Leslie saw that if she ever hurt herself she would need my help to heal and realized that she needs me (C) None of the above choices (D) We were able to spend time together just talking and that helped us to connect better Answer: (D) We were able to spend time together just talking and that helped us to connect better But the thing is, I forgave and forgot a while ago. I ’m not bitter anymore. I just genuinely don’t want to go back to that church. I couldn’t get the balls to tell her that ... Question: Why wouldn’t I have the balls to tell her that? (A) I am not a man and I can’t tell her those things to her face (B) I am afraid of that church and I would feel less manly if I told her (C) None of the above choices (D) It is her favorite church and I don’t want to hurt her feelings Answer: (D) It is her favorite church and I don’t want to hurt her feelings

Example 3: Samples from SocialIQA (Sap et al., 2019) During their yearbook class, Alex took an old photo of herself and Quinn, and crossed out Quinn’s face on the photograph. Quinn saw her do this. Question: How would Quinn feel as a result? (A) Like Alex misses hanging out with her (B) Like Alex betrayed her (C) be friends again Answer: (B) Like Alex betrayed her Tracy was teaching Kendall to dance so Tracy raised Kendall’s hips into the right position for the next move. Question: How would you describe Tracy? (A) a bad teacher (B) hopeful that this doesn’t make Kendall uncomfortable (C) a patient teacher Answer: (B) hopeful that this doesn’t make Kendall uncomfortable

15

Title Suppressed Due to Excessive Size

Example 4: Samples from Alpaca (Taori et al., 2023) Question: Use the given data to calculate the median. Input: [2, 3, 7, 8, 10] Answer: The median is 7. Question: Analyze the given text for its tone. Input: The world has been greatly impacted by the COVID-19 pandemic and it has drastically changed our lives. Answer: The tone of the text is one of concern and recognition of the dramatic effects of the COVID-19 pandemic on the world. It is an acknowledgement of the upheaval that the virus has caused and that our lives have been drastically changed. Question: Explain the concept of artificial intelligence in simple terms. Answer: Artificial Intelligence (AI) is the ability of machines and computers to make decisions and complete tasks similar to what humans can do. AI can learn from data, recognize patterns, and make predictions. AI technology can automate tedious and complex processes, helping to reduce human effort and errors, and enable more accurate and efficient decisions. AI can be used in a wide range of applications, from robotics and autonomous vehicles to healthcare. It is increasingly becoming an integral part of everyday life.

E. Additional Experiments E.1. Computation Overhead of Relevance-Weighted DPP Both FlexOlmo and MetaMoE require computing embeddings for all public samples, which dominates the proxy-selection cost. The additional work unique to MetaMoE is the greedy MAP inference for relevance-weighted DPP, implemented with efficient Cholesky updates (see Appendix B). Table 9 shows the measured running time for NLP and CV setups. Relevance-weighted DPP adds only 0.2–0.3 minutes (3–5%) per client, while expert finetuning takes over 10 GPU-hours, confirming that the diversity-aware selection introduces only marginal overhead relative to similarity-only baselines while delivering more representative proxy sets. Table 9. Proxy-selection running time (minutes).

NLP Similarity-based selection Relevance-weighted DPP

CV

LLaMA-3.2-3B

LLaMA-3.1-8B

ViT-B/32

ViT-B/16

1.91 2.17

4.21 4.54

6.41 6.61

15.70 15.91

E.2. Comparison with Federated Learning Methods: CoMiGS (Fan et al., 2025) and MoA (Feng et al., 2024) Federated learning (FL) methods differ fundamentally from MetaMoE: they require exchanging large model states every round. This leads to substantial bandwidth and memory overhead and creates instability when client data are heterogeneous because divergent local updates must be averaged. MetaMoE eliminates synchronization entirely—each client fine-tunes its expert locally (on private plus proxy data), and only a single exchange of frozen expert weights occurs before router training. Comparison with CoMiGS (Fan et al., 2025). CoMiGS (Fan et al., 2025) adopts a federated learning paradigm that requires repeated synchronization among clients—periodically exchanging model parameters for joint optimization. This approach incurs high communication and memory costs and often becomes unstable under heterogeneous client data, leading to degraded performance. In contrast, MetaMoE avoid exchanging model parameters entirely. Each client independently fine-tunes its expert on private and proxy data, and all experts are unified once through MoE integration. This merge-aftertraining design achieves better scalability, eliminates communication overhead, and remains stable under heterogeneous data. Comparison with Mixture-of-LoRAs (MoA) (Feng et al., 2024). MoA also seeks to unify multiple LoRA experts but assumes direct access to all client data for router training, which violates privacy constraints. In the experiments reported here, MoA is reimplemented by training its router only on public data to maintain expert coordination without violating privacy constraints, enabling a fair comparison under the same privacy-preserving setting. Empirical results. MetaMoE is compared with CoMiGS and MoA under the same LoRA configurations. As shown in Table 10, MetaMoE consistently outperforms both methods across all benchmarks, demonstrating that MetaMoE’s unification offers better performance. 16

Title Suppressed Due to Excessive Size Table 10. Comparison with CoMiGS and MoA on NLP tasks (LLaMA-3.2-3B).

CoMiGS (Fan et al., 2025) MoA (Feng et al., 2024) MetaMoE

CSQA

CosmosQA

SocialIQA

Average

72.32 71.09 74.94

71.46 74.64 76.05

71.19 70.11 72.26

71.65 71.95 74.42

E.3. Effect of Expert Training without Private Data To isolate the role of private data, we conduct an ablation where experts are trained solely on proxy samples and never observe client data. We evaluate this proxy-only baseline in the CV setting. As summarized in Tables 11 and 12, training exclusively on proxies leads to large accuracy drops, confirming that proxies primarily provide coordination signals for router learning. MetaMoE, which blends private data (for domain expertise) with proxies (for alignment), substantially outperforms the proxy-only alternative, demonstrating the necessity of private-domain expertise. Table 11. Proxy-only baseline versus MetaMoE (CLIP ViT-B/32).

Train solely on proxy data MetaMoE

Pets

Flowers

EuroSAT

Average

78.60 91.91

42.63 93.67

12.98 97.98

44.74 94.52

Table 12. Proxy-only baseline versus MetaMoE (CLIP ViT-B/16).

Train solely on proxy data MetaMoE

Pets

Flowers

EuroSAT

Average

82.47 94.22

51.73 97.08

22.30 97.41

52.17 96.24

E.4. Privacy-Preserving Guarantees MetaMoE never exposes private data, and any similarity between selected proxy samples and private data does not constitute a privacy violation, as all proxy candidates are drawn from a public dataset that is already accessible to all parties. (1) Semantic similarity is not equivalent to private-data exposure. The relevance-weighted DPP method selects public samples that are representation-wise similar to the client domain. These proxies may resemble private data but are not derived from private samples, and all clients already have access to them. In privacy-preserving systems, leakage occurs only when non-public information becomes newly revealed. Selecting an already public sample—no matter how similar—does not expose any new private information. (2) Overlap with public data does not constitute private-data exposure. If a public sample coincidentally overlaps with one in a client’s dataset, revealing that sample still constitutes public-data exposure, not private-data exposure, since the content is already publicly available prior to any interaction with MetaMoE. This principle aligns with standard privacy frameworks such as the California Consumer Privacy Act (CCPA), which explicitly excludes “publicly available information” from the definition of personal data. Under this definition, the exposure of a public sample is not regarded as a violation of privacy. (3) MetaMoE introduces no new channels for private-data leakage. The proxy-selection process transmits only the IDs of selected public samples and the final expert weights—never private samples, gradients, or intermediate activations. Because all proxy candidates are drawn from a public dataset, the process does not disclose or allow inference about private data. E.5. Comparison with LoRASuite (Li et al., 2025) MetaMoE is a low-rank adaptation unification method, whereas LoRASuite (Li et al., 2025) is not, and the two address fundamentally different goals. MetaMoE aims to unify multiple domain-specialized LoRA experts trained on the same backbone into a privacy-preserving MoE—answering how to combine many LoRA experts without sharing client data. In contrast, LoRASuite focuses on LoRA migration, transferring a single LoRA adapter trained on backbone (A) to backbone 17

Title Suppressed Due to Excessive Size

(B) after the backbone is upgraded, with the goal of expert adaptation rather than expert unification. Furthermore, LoRASuite requires client data to align activations and performs poorly without it, while MetaMoE assumes no private data access and relies entirely on public proxies for supervision. Given these fundamental differences in objectives and data requirements, a direct empirical comparison would be inappropriate, as it would force LoRASuite into a multi-expert unification setting it was never designed for. E.6. Computational Cost Analysis This section provides a detailed comparison of MetaMoE’s computational cost against existing methods, covering both unification time and inference speed across vision and language tasks. The results demonstrate that MetaMoE achieves strong accuracy improvements without adding significant overhead. Unification cost. The merging (unification) time of MetaMoE is nearly identical to FlexOlmo and BTX across all backbones, showing that the accuracy gains do not come from higher computational cost during unification. While BTM and ModelSoup appear faster (or cost-free), they avoid the coordination required for MoE merging, which explains their lower accuracy. The small extra cost in MetaMoE is therefore a modest and worthwhile trade-off for its significant accuracy improvement. Inference efficiency. MetaMoE maintains inference speeds comparable to other MoE unification methods, confirming that the context-aware router introduces minimal runtime overhead and scales efficiently across backbones. In contrast, BTM requires inference over all experts for every input, leading to approximately 3× slower inference despite its zero unification cost. Overall cost–performance balance. Across both CV and NLP tasks, MetaMoE consistently achieves state-of-the-art accuracy with comparable computational efficiency, as summarized in Tables 13 and 14. Table 13. Cost–performance comparison on CV tasks.

ViT-B/32 BTM ModelSoup BTX FlexOlmo MetaMoE

ViT-B/16

ACC

Unify Time (s)

Inference Speed (samples/s)

ACC

Unify Time (s)

Inference Speed (samples/s)

90.33 74.20 74.30 92.92 94.52

− 5.72 11.13 11.93 12.15

606 1813 1758 1767 1751

91.75 79.42 81.20 93.53 96.24

− 5.72 19.72 18.24 19.88

249 743 715 719 710

Table 14. Cost–performance comparison on NLP tasks.

LLaMA-3.2-3B BTM ModelSoup BTX FlexOlmo MetaMoE

LLaMA-3.1-8B

ACC

Unify Time (s)

Inference Speed (samples/s)

ACC

Unify Time (s)

Inference Speed (samples/s)

72.74 73.57 71.14 72.50 74.42

− 8.24 118.21 119.90 114.46

17.44 43.46 42.12 41.59 40.67

79.79 80.51 76.73 77.46 81.59

− 11.25 223.37 206.28 205.42

8.20 22.86 21.41 21.95 20.05

E.7. Robustness under Non-Overlapping Public Data In our main CV experiments (Section 4.1), the public dataset ImageNet may share semantic overlap with certain client domains (e.g., dog/cat breeds for Pets, flower species for Flowers). To evaluate MetaMoE under a more challenging setting where such overlap is eliminated, we remove all ImageNet categories that are semantically related to the three client domains, including dog and cat breeds (for Pets), flower species (for Flowers), and satellite-like or aerial imagery (for EuroSAT). All other settings remain identical to those in Section 4.1 with CLIP ViT-B/32. As shown in Table 15, MetaMoE remains the strongest method under zero domain overlap, achieving 93.78% average accuracy with only a 0.74-point drop compared to the full-overlap setting (94.52%, Table 1). In contrast, FlexOlmo degrades by 3.53 points (89.39% vs. 92.92%), confirming that relevance-weighted DPP is more robust to domain gap than relevanceonly proxy selection. Moreover, the NLP experiments in the main paper (Section 4.2) also reflect a non-overlapping regime: 18

Title Suppressed Due to Excessive Size

the public dataset Alpaca shares no domain overlap with the client datasets by construction, yet MetaMoE consistently outperforms all baselines (Tables 3–4). Table 15. Accuracy on CV tasks with CLIP ViT-B/32 under non-overlapping public data, where all ImageNet categories semantically related to client domains are removed.

ModelSoup BTM BTX FlexOlmo MetaMoE

Pets

Flowers

EuroSAT

Average

87.90 90.81 80.35 86.56 91.01

70.52 85.10 61.10 88.55 92.53

64.19 95.07 58.16 93.07 97.80

74.20 90.33 66.54 89.39 93.78

F. Privacy Analysis of Routing Vectors This section provides the complete derivations for the privacy guarantees stated in Section 3.6. We analyze the routing (l) vector ep shared by each client p and show that it reveals negligible private information, with formal guarantees that are independent of both the domain gap and the private dataset size. Notation. We fix a client p and a layer l, and drop the subscript p and superscript (l) when context is clear. Let {xi }N i=1 denote the private samples in Dp and {zj }m j=1 denote the proxy samples in D̂p , where N = |Dp | and m = |D̂p |. Let (1:l)

f (·) = Mp (·) denote the encoder (the first l layers of expert Mp ), and let B = maxx ∥f (x)∥2 denote the embedding norm bound. Define the mean private embedding and the mean proxy embedding as N

µpriv =

m

1 X f (xi ), N i=1

µproxy =

1 X f (zj ). m j=1

(12)

(l)

The routing vector ep (defined in (7)) can then be written as   m N X X 1 N m  e= f (zj ) = f (xi ) + µpriv + µproxy . N + m i=1 N + m N +m j=1

(13)

Adversary’s knowledge. We consider an honest-but-curious adversary (e.g., the central server) that can observe the routing vector e and knows the public quantities µproxy , m, and f . Crucially, the adversary does not know N , the private dataset size, as it is never communicated by the MetaMoE protocol. F.1. Per-Sample Sensitivity Bound We show that the influence of any single private sample on the routing vector is bounded by O(1/m), regardless of N and the domain gap. Proposition F.1 (Sensitivity bound). Let e be the routing vector defined in (13), and let e′ denote the routing vector obtained by replacing a single private sample xk with an arbitrary sample x′k . The ℓ2 sensitivity of e satisfies 2B 2B ≤ . N +m m

(14)

Proof. Replacing xk with x′k changes only the k-th term in the private sum, yielding
1 e − e′ = f (xk ) − f (x′k ) . N +m Taking the ℓ2 norm and applying the triangle inequality gives

(15)

∆2 (e) = max′ ∥e − e′ ∥2 ≤ xk ,xk

∥f (xk ) − f (x′k )∥2 ∥f (xk )∥2 + ∥f (x′k )∥2 2B ≤ ≤ . N +m N +m N +m Since N ≥ 0 implies N + m ≥ m, we obtain ∆2 (e) ≤ 2B/m. ∥e − e′ ∥2 =

19

(16)

Title Suppressed Due to Excessive Size

Implications. The bound ∆2 (e) ≤ 2B/m depends solely on the proxy set size m, which is a public hyperparameter. In our experiments, m = 500, so each individual private sample’s contribution is diluted among at least 500 embeddings, making its influence on e negligibly small. Furthermore, this bound is independent of both the private dataset size N and the domain gap between private and proxy data. When N is small (a challenging privacy scenario), the bound is already controlled by O(1/m); when N grows large, it additionally tightens to O(1/(N + m)). This sensitivity bound is stated in the same sense as the foundational framework of differential privacy (Dwork & Roth, 2014): low sensitivity implies that any individual private sample has a vanishingly small effect on the shared statistic, which is a necessary condition for strong privacy guarantees. F.2. Unrecoverability of Private-Data Statistics We show that the mean private embedding µpriv , the coarsest possible summary of the private data, cannot be recovered from the routing vector e. Proposition F.2 (Unrecoverability). Given access to the routing vector e, the mean proxy embedding µproxy , the proxy set size m, and the encoder f , an adversary cannot uniquely determine µpriv without knowledge of the private dataset size N . Proof. From (13), the adversary can compute (N + m) e − m µproxy = N µpriv .

(17)

(N + m) e − m µproxy . N

(18)

Isolating µpriv yields µpriv =

Since the adversary knows e, µproxy , and m, evaluating (18) requires knowing N , which appears both in the numerator and the denominator and cannot be canceled out. Because N is never communicated by the MetaMoE protocol, the adversary cannot evaluate (18) for the true value of N , and therefore cannot recover µpriv . Implications. The mean private embedding µpriv represents the coarsest possible form of domain-level information: it is a single vector that summarizes the entire private dataset’s representation in the encoder’s feature space. Since even this coarsest statistic is unrecoverable from e, finer-grained distributional properties of Dp (e.g., variance, class proportions, cluster structure, or individual samples) are a fortiori unrecoverable. This directly resolves the concern on whether the routing vector could leak domain-level or distributional information about the private data. F.3. Comparison with FlexOlmo We show that MetaMoE exposes strictly less private information than FlexOlmo (Shi et al., 2025) through the routing mechanism. FlexOlmo initializes its domain-informed router using per-expert routing embeddings computed as the meanP embedding N over private data alone (Section 3.3.2 of Shi et al. (2025)). Concretely, FlexOlmo shares the vector µpriv = N1 i=1 f (xi ), m which is the complete mean private embedding. In contrast, MetaMoE shares e = NN +m µpriv + N +m µproxy (13), where the private component µpriv is diluted by averaging with the publicly computable µproxy . This comparison establishes the following: (i) FlexOlmo directly reveals µpriv , giving an adversary full access to the mean private embedding. (ii) MetaMoE never reveals µpriv in isolation. The adversary observes only the diluted mixture e, from which µpriv is unrecoverable without N (Proposition F.2). (iii) Per-sample sensitivity is also stronger. FlexOlmo’s routing embedding has sensitivity 2B/N , which grows large when N is small. MetaMoE’s sensitivity is 2B/(N + m) ≤ 2B/m, which remains bounded even for small N . Therefore, MetaMoE provides a strictly stronger privacy guarantee than FlexOlmo in terms of both the recoverability of private-data statistics and the per-sample sensitivity. 20