arXiv:2606.15625v1 [cs.LG] 14 Jun 2026
Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts Yijun Lu†
Zihan Fang†
Pengpeng Qiao
Waseda University, Tokyo, Japan [email protected]
City University of Hong Kong, China [email protected]
Institute of Science Tokyo, Tokyo, Japan [email protected]
Zheng Lin∗
Jing Yang
Yuxin Zhang
The University of Hong Kong, China [email protected]
Universiti Malaya, Malaysia [email protected]
Fudan University, Shanghai, China [email protected]
Por Lip Yee
Zhe Chen
Jun Luo
Universiti Malaya, Malaysia [email protected]
Fudan University, Shanghai, China [email protected]
Nanyang Technological University [email protected]
Abstract—The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixtureof-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation. While federated learning (FL) emerges as the paradigm for privacy-preserving collaborative optimization, integrating MoE into FL under data heterogeneity may trigger conflicting expert optimizations. Client-specific data distributions forces same-indexed experts to optimize under inconsistent or even conflicting feature-label correlations. This mismatch induces destructive interference during aggregation, thus destabilizing the optimization trajectory and degrading model performance. To address this issue, we propose FC-MoE, a federated conflictaware framework for MoE fine-tuning. It employs an importanceaware weighting scheme to prioritize reliable local update and utilizes gradient consensus projection to suppress conflicting updates, ensuring a stable global optimization path. Moreover, a local knowledge retention mechanism further preserves specialized client expertise by re-anchoring domain-specific residuals. Extensive experiments demonstrate that FC-MoE accelerates convergence and enhances both global and local model performance in non-IID federated environments. Index Terms—Federated learning, mixture of experts, largescale language model, aggregation
I. I NTRODUCTION Large language models (LLMs), such as GPT [1], LLaMA [2], and DeepSeek [3], have achieved transformative success across academia and industry by scaling model parameters and training data, which enables the models to capture intricate linguistic patterns during pre-training and effectively generalize to diverse downstream tasks through finetuning [4]–[7]. However, continuous scaling of dense transformer models is increasingly hindered by the prohibitive computational cost required for large-scale pre-training and downstream adaptation [8]–[11]. To address this limitation, Mixtureof-Experts (MoE) architectures [12]–[15] have emerged as a scalable alternative. MoE employs a gating network to † Yijun Lu and Zihan Fang contributed equally to this work. ∗ denotes the corresponding author.
selectively route each token to a sparse top-k subset of specialized experts while leaving the remaining experts inactive. This sparse activation mechanism enables MoE models to retain most of their representational capacity while achieving performance comparable to large dense counterparts with significantly reduced computational overhead [13], [14], [16]– [18], making MoE-based LLM a promising foundation for efficient fine-tuning [19]. The real-world deployment of MoE-based LLMs demands massive volumes of data, which frequently contain sensitive details such as personal medical records and financial information [20], [21]. Because strict privacy regulations prohibit sharing these private datasets across clients, traditional centralized training paradigms face a critical bottleneck. Federated Learning (FL) has therefore emerged as a promising paradigm that enables collaborative model optimization without compromising data privacy [22]–[25]. Combining MoE’s sparse activation with the decentralized framework of FL [16], federated MoEbased LLM fine-tuning allows each client to locally activate and fine-tune only the sparse subset of experts relevant to its private datasets, and the updated experts are then aggregated on the server to construct an improved global MoE model, facilitating the integration of diverse knowledge across diverse clients. While the integration of MoE into FL frameworks holds significant promise for efficient LLM fine-tuning, it encounters critical aggregation challenges due to inherent data heterogeneity, where clients typically possess non-identically distributed (non-IID) data stemming from variations in local domains or task-specific requirements [26]–[28]. Given that the MoE routing mechanism dynamically directs tokens to experts based on client-specific data distributions, this heterogeneity causes same-indexed experts to be optimized under inconsistent or even conflicting feature–label correlations, leading to disparate expert activation and optimization patterns across clients. Consequently, updates for the same-indexed expert from various
Multiple Clients (e.g., Client i)
Central Server
1. Importance-aware Expert Weighting
2. Gradient Consensus Projection
Importance Score
Expert ...
Local Dataset
Expert Expert
MoE Architecture
Upload
f i,k ∆θ i,k
Construct Global Consensus Direction
Consensus Expert Update
Conflicting
Identify Conflicting Gradient Updates
Initialization 3. Local Knowledge Retention ...
...
...
Projection
Project onto Consensus Direction
...
... ...
...
... ...
... Filtered Global Expert Domain-Specific Parameters Features
Redistribute
Re-anchor Specialized Knowledge
Global Aggregation
Θ k(t+1)= Θ k(t) + ∑ c i,k ∆θ i,k
proj
Conflicting Consensus
Fig. 1: The overview of FC-MoE framework.
clients often become misaligned or even contradictory [29], inducing destructive interference during aggregation as existing solutions [16], [17], [28] typically assume all expert updates are mutually compatible. This interference destabilizes the optimization trajectory, degrading the convergence stability and the performance of the global MoE model. To tackle expert conflicts induced by non-IID data distributions, we propose a federated conflict-aware framework for MoE-based LLM fine-tuning, named FC-MoE. The framework aligns global consensus with local expert specialization by identifying conflicting updates and projecting their gradients toward a consensus direction during aggregation using an importance-aware weighting scheme, ensuring a stable and consistent optimization trajectory for the global MoE model. To prevent the loss of client-specific knowledge, we further introduce a local knowledge retention strategy that allows each client to re-anchor the specialized knowledge during local initialization, effectively mitigates expert conflicts without sacrificing local model performance.
II. M ETHOD A. Overall Architecture of FC-MoE As shown in Fig. 1, the proposed FC-MoE comprises three key modules: importance-aware expert weighting, gradient consensus projection, and local knowledge retention. Each client first computes importance scores for its active experts through importance-aware expert weighting and transmits these scores to the server along with local parameter updates. The server then constructs a global consensus direction based on the received expert weights and projects conflicting expert gradients onto this direction to suppress destructive interference. Finally, the resulting global consensus is redistributed to each client, where local knowledge retention is applied to reincorporate the filtered domain-specific features into the local model initialization for the next training round.
B. Importance-aware Expert Weighting Existing aggregation schemes typically assume that all local updates are mutually compatible and weight them solely based on client dataset size. Such indiscriminate averaging can induce destructive interference as misaligned or contradictory local experts across clients are allowed to negatively impact the global model. To mitigate this interference and guide a stable global aggregation, we introduce a dynamic weighting scheme that explicitly quantifies the importance of each local expert update. In the MoE architecture, an expert’s activation frequency reflects its statistical reliability with respect to the client’s specific data distribution, as frequently activated experts tend to produce more representative gradients. Activation frequency captures usage patterns but fails to reflect the extent of knowledge acquisition. Therefore, we further incorporate the gradient magnitude (i.e., the norm of the parameter update) as a measure of how much the expert has adapted to the local data, providing a direct geometric characterization of its learning intensity. Specifically, after each local training, client i calculates the parameter update for each active expert k as local local ∆θi,k = θi,k − θkglobal , where θi,k and θkglobal are the local and global expert parameters. The importance score ci,k for expert k is then defined as ci,k = fi,k · ∥∆θi,k ∥,
(1)
where fi,k represents the total activation count of expert k under the local gating network on client i’s local dataset Di , prioritizing experts that are heavily utilized by local data. The term ∥∆θi,k ∥ is the norm of the expert’s parameter update, capturing the intensity of local adaptation. By calculating the product of these two factors, this importance-aware expert weight ensures that the global aggregation is driven by experts that are both highly relevant and significantly optimized, providing a reliable indicator to mitigate expert conflicts during global aggregation. Client i then uploads the tuple (∆θi,k , ci,k ) to the central server.
Client Side
C. Gradient Consensus Projection
∆θ̄k =
N X
ci,k ∆θi,k , PN j=1 cj,k i=1
(2)
where Sk denotes the set of clients that activated expert k in the current round. Based on this reference anchor, the server quantifies the geometric interference between each local update ∆θi,k and the global consensus ∆θ̄k , where a conflict is defined by a negative cosine similarity cos(∆θi,k , ∆θ̄k ) < 0. To mitigate expert conflicts without completely discarding the client’s local learning, we project the conflicting local update ∆θi,k onto the normal plane of the consensus direction ∆θ̄k as ( ∆θi,k ·∆θ̄k ∆θi,k − ∥∆ ∆θ̄k , if cos(∆θi,k , ∆θ̄k ) < 0 proj θ̄k ∥2 ∆θi,k = ∆θi,k , otherwise (3) Finally, the server aggregates the projected updates using the normalized contribution scores to update the global expert parameters, ensuring that the global MoE model follows a stable and consistently improving optimization trajectory: (t+1) (t) Θk = Θk +
N X
ci,k proj ∆θi,k . PN j=1 cj,k i=1
(4)
D. Local Knowledge Retention While the gradient consensus projection facilitates global convergence by mitigating conflicting components, it may inevitably strip away certain domain-specific features required for optimal local performance. To prevent the degradation of local expert specialization, FC-MoE enables each client to isolate its specialized gradient residuals locally and strategically re-incorporate them into its local optimization in the next training round. The overall workflow is illustrated in Fig. 2. Specifically, upon receiving the aggregated global model Θ(t+1) from the server at the beginning of round t + 1, each client locally derives a filtered residual for each expert k. To
Server Side
Receive Global Model Θ (t) Round t (Init & Training)
1
Round t+1
Standard federated aggregation schemes (e.g., FedAvg) perform direct averaging of local updates, which results in destructive interference for MoE-based LLM under non-IID data distributions. To address this, we shift the aggregation from blind averaging to consensus-guided geometric projection. As gradient updates of local parameters mathematically represent the optimization trajectories of client-specific objectives, they can be used as a geometric indicator to detect conflicting expert optimizations. By establishing a global consensus trajectory as a reference anchor, the server can explicitly quantify the geometric compatibility of each local update, encouraging the model to converge toward a generalized global optimum rather than collapsing under conflicting local biases. Specifically, the server calculates a global consensus direction ∆θ̄k for each expert k, which represents the dominant optimization direction driven by the majority of the relevant global data distribution. This consensus is calculated by the importance-weighted aggregation of local updates:
Local Initialization θ i,klocal
Collect Updates
N
Local Knowledge Retention
Compute Consensus ( ∑ , ∆θ i,k )
Client Upload ( ∆θ i,k , c i,k )
proj
Local MoE Training
Resolve Conflicts ∆θ i,k
Compute Local Updates ∆θ i,k
Update Global Model ( ∑ , Θ k(t+1) ) Redistribute
Receive Redistributed MoE Θ (t+1)
Θ (t+1)
......
Redistribute Global MoE
Fig. 2: The overall workflow of conflict-aware federated MoEbased LLM fine-tuning. extract its personalized domain knowledge, the client projects its previous local update ∆θi,k onto global expert parameters: (t+1)
Ri,k =
∆θi,k · Θk
(t+1)
(t+1) 2 ∥Θk ∥
Θk
(5)
Before starting the next round local training, client i explicitly anchors this localized knowledge by incorporating a scaled version of the residual back into its local expert parameters: (t+1)
local θi,k = Θk
+ λRi,k
(6)
where λ serves as a retention coefficient that controls the intensity of local specialization. III. E XPERIMENT AND R ESULTS A. Experimental Setup Our evaluation employs the Switch Transformer [14] as the MoE-based LLM backbone, which represents a classic scalable MoE design, utilizing a Top-1 routing that assigns tokens to one of 16 experts per layer. To evaluate model performance, we report the test accuracy for the global model on widely used datasets, spanning from basic semantic classification (AGNews [30]) to complex knowledge reasoning (MMLU [31]). The performance of FC-MoE is compared against several representative FL frameworks, including FedAvg [32], PFLMoE [17], and FedMoE [16]. The distributed system comprises a central server and 10 synchronized clients, with computations accelerated by NVIDIA GeForce RTX 4090 GPUs. For local fine-tuning, each client executes one epoch per communication round with a learning rate of 1 × 10−4 , with a total of 25 communication rounds. The key hyper-parameters in FC-MoE is set to λ = 0.5. To simulate real-world data heterogeneity in federated learning, we partition the datasets using a Dirichletbased distribution scheme [17]. By setting the concentration parameter α = 0.1, we induce significant label skew across the client-specific data distributions, providing a challenging non-IID setting for evaluating our proposed conflict-aware aggregation framework.
AGNews α = 0.1
Method
α = 0.5
α = 1.0
accuracy
precision
recall
accuracy
precision
recall
accuracy
precision
recall
FedAvg
0.7740
0.7940
0.7568
0.8866
0.8921
0.8739
0.8978
0.9089
0.8907
PFL-MoE
0.7994
0.8223
0.7911
0.9041
0.9128
0.9036
0.9109
0.9170
0.9106
FedMoE
0.8137
0.8374
0.8056
0.9103
0.9192
0.9120
0.9164
0.9204
0.9155
FC-MoE
0.8359
0.8617
0.8360
0.9194
0.9246
0.9194
0.9224
0.9230
0.9223
MMLU α = 0.1
Method
α = 0.5
α = 1.0
accuracy
precision
recall
accuracy
precision
recall
accuracy
precision
recall
FedAvg
0.3008
0.3870
0.3002
0.3482
0.3854
0.3478
0.4012
0.4002
0.4010
PFL-MoE
0.3325
0.3541
0.3301
0.3789
0.3947
0.3740
0.4305
0.4242
0.4275
FedMoE
0.3506
0.3699
0.3500
0.3921
0.4063
0.3919
0.4463
0.4412
0.4460
FC-MoE
0.3764
0.4017
0.3760
0.4169
0.4372
0.4168
0.4695
0.4650
0.4694
21
1
14
7
0
FedAvg
PFL-MoE
FedMoE
FC-MoE
AGNews MMLU
FedAvg
(a) Convergence
PFL-MoE
FedMoE
local acc
test acc
local acc
a)
0.7849
0.5574
0.3306
0.2992
b)
0.8068
0.6108
0.3478
0.3225
c)
0.8264
0.6459
0.3616
0.3428
FC-MoE
0.8359
0.6732
0.3764
0.3559
FC-MoE
(b) Local accuracy
Fig. 3: The convergence and local accuracy across clients under non-IID data distributions with α = 0.1.
TABLE II: The test and local accuracy for FC-MoE under non-IID data distributions with α = 0.1.
B. Experimental Results and Analysis
0.4
0.9
Test accuracy
Table I summarizes the test accuracy across varying levels of data heterogeneity, where methods are evaluated under three non-IID configurations (α ∈ {0.1, 0.5, 1.0}). FC-MoE exhibits consistent superiority over all baselines on both the AGNews and MMLU benchmarks, achieving a test accuracy of 0.8359 on AGNews and 0.3764 on MMLU in the most challenging non-IID scenario (α = 0.1). Moreover, the performance gap between FC-MoE and existing methods widens as the data distribution becomes increasingly skewed. This observation confirms that our conflict-aware aggregation effectively mitigates the destructive interference caused by divergent local expert optimizations, leading to a more robust and generalized global model. Fig. 3a illustrates the convergence rate for various benchmarks on the AGNews and MMLU datasets. Compared to other baselines, FC-MoE exhibits significantly faster convergence, reaching the target performance in fewer communication rounds. This acceleration is attributed to our gradient consensus projection. By ensuring aggregated updates align with the global consensus, FC-MoE maintains a more stable and efficient path toward the global optimum, effectively
MMLU
test acc
0.4
0.1
AGNews
Method
0.7
Test accuracy
AGNews MMLU
Local Accuracy
Converged time (epoch)
TABLE I: The test accuracy on the AGNews and MMLU datasets under non-IID data distributions.
0.7 FC-MoE c) b) c)
0.5
0.3
0
5
10
15
Epoch
(a) AGNews
20
25
0.3 FC-MoE c) b) a)
0.2
0.1
0
5
10
15
20
Epoch
(b) MMLU
Fig. 4: The test accuracy of FC-MoE and its variants on AGNews and MMLU datasets. mitigating the optimization oscillations induced by gradient interference in heterogeneous environments. To evaluate the model’s capacity to preserve client-specific knowledge, we analyze the statistics of local accuracy on clients in Fig. 3b. The results reveal that FC-MoE not only achieves a higher mean accuracy but also exhibits substantially lower variance across clients, demonstrating superior local representational fidelity. This success is primarily due to the local knowledge retention strategy, which allows clients to re-anchor domain-specific residuals during local initialization, ensuring global alignment without compromising local expertise.
25
C. Ablation Study We conduct the ablation study to quantify the individual contributions of our designed components, with results visualized in Table II and Fig. 4, where FC-MoE is compared against three degraded variants: a) without gradient consensus projection, b) without local knowledge retention, and c) without importance-aware expert weighting. The results demonstrate that the removal of any single component leads to noticeable performance drops and increased oscillation in the test accuracy. The comparison confirms that the gradient consensus projection is the primary driver of optimization stability, while the importance-aware expert weighting ensures the consensus direction is driven by more reliable local updates. Further supported by local knowledge retention, FC-MoE effectively balances global convergence and local specialization in heterogeneous environments. IV. C ONCLUSION In this paper, we have proposed a novel conflict-aware aggregation framework, named FC-MoE , designed to tackle the inherent expert conflicts in federated MoE-based LLM fine-tuning under non-IID data distributions. FC-MoE explicitly quantifies the reliability of local updates through an importance-aware weighting scheme and geometrically aligns conflicting experts via a gradient consensus projection. Furthermore, a local knowledge retention mechanism is introduced to strategically anchor the filtered domain-specific residuals, preventing the degradation of client personalization. Extensive experiments across diverse benchmarks demonstrate the superiority of FC-MoE , which accelerates global model convergence and enhances generalization without sacrificing local expert specialization. R EFERENCES [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, Mar. 2023. [2] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, Feb. 2023. [3] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, Feb. 2025. [4] Y. Chen, Y. He, J. Yang, D. Zhang, Z. Yuan, M. A. Khan, others, and L. Y. Por, “Empower: Evolutionary medical prompt optimization with reinforcement learning,” IEEE Journal of Biomedical and Health Informatics, 2025. [5] Z. Lin, Y. Zhang, Z. Chen, Z. Fang, X. Chen, P. Vepakomma, W. Ni, J. Luo, and Y. Gao, “HSplitLoRA: A Heterogeneous Split ParameterEfficient Fine-Tuning Framework for Large Language Models,” IEEE Transactions on Mobile Computing, 2025. [6] Z. Fang, Z. Lin, Z. Chen, X. Chen, Y. Gao, and Y. Fang, “Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models,” IEEE Trans. Mobile Comput., 2025. [7] Y. Zhang, M. Hu, Z. Lin, X. Fan, F. Xie, Z. Fang, J. Yang, W. Zhu, Z. Chen, C. Lv et al., “Hera: Learning long-horizon coordination for device-cloud collaborative llm agents,” arXiv preprint arXiv:2605.24598, 2026. [8] S. He, D. Dong, L. Ding, and A. Li, “Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques,” arXiv preprint arXiv:2406.02500, Mar. 2025.
[9] Z. Du, S. Li, Y. Wu, X. Jiang, J. Sun, Q. Zheng, Y. Wu, A. Li, H. Li, and Y. Chen, “SIDA-MOE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models,” Proc. MLSys, vol. 6, pp. 224–238, Apr. 2024. [10] Z. Lin, Y. Zhang, Z. Chen, Z. Fang, C. Wu, X. Chen, Y. Gao, and J. Luo, “Leo-split: A semi-supervised split learning framework over leo satellite networks,” IEEE Transactions on Mobile Computing, 2025. [11] S. Wang, Z. Chen, B. Li, K. He, M. Zhang, and J. Wang, “Scaling laws across model architectures: A comparative analysis of dense and moe models in large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 5583–5595. [12] B. Lin, Z. Tang, Y. Ye, J. Huang, J. Zhang, Y. Pang, P. Jin, M. Ning, J. Luo, and L. Yuan, “Moe-llava: Mixture of experts for large visionlanguage models,” IEEE Transactions on Multimedia, 2026. [13] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu et al., “Deepseekmoe: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models,” arXiv preprint arXiv:2401.06066, Jan. 2024. [14] W. Fedus, B. Zoph, and N. Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, Apr. 2022. [15] Z. Fang, Z. Lin, S. Hu, Y. Ma, Y. Tao, Y. Deng, X. Chen, and Y. Fang, “Hfedmoe: Resource-aware heterogeneous federated learning with mixture-of-experts,” arXiv preprint arXiv:2601.00583, 2026. [16] H. Mei, D. Cai, A. Zhou, S. Wang, and M. Xu, “Fedmoe: Personalized federated learning via heterogeneous mixture of experts,” arXiv preprint arXiv:2408.11304, 2024. [17] B. Guo, Y. Mei, D. Xiao, and W. Wu, “PFL-MoE: Personalized Federated Learning Based on Mixture of Experts,” in Proc. APWebWAIM. Springer, Aug. 2021, pp. 480–486. [18] Z. Fang, Q. Wang, H. An, Z. Lin, Y. Deng, X. Chen, and Y. Fang, “Aggregation alignment for federated learning with mixture-of-experts under data heterogeneity,” arXiv preprint arXiv:2603.21276, 2026. [19] B. Liu and J. Yang, “Cmos: Customizing model structures for personalized federated learning,” in ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2025. [20] W. Ge, V. Govindarajan, J. Yang, M. Ayadi, Z. A. Shaikh, L. Li, L. Y. Por et al., “Optimized federated learning framework with regnetz and swin-transformer for multimodal pancreatic cancer detection,” Scientific Reports, 2025. [21] A. A. Khan, X. Zhang, F. Hajjej, J. Yang, C. S. Ku, and L. Y. Por, “Asmf: Ambient social media forensics chain of custody with an intelligent digital investigation process using federated learning,” Heliyon, vol. 10, no. 1, 2024. [22] S. Latif, J. Ahmad, W. Al Malwi, F. Asiri, N. Alnazzawi, J. Yang, and T. R. Gadekallu, “Mitigating model poisoning and tampering in consumer iot with hmac in split federated learning,” IEEE Transactions on Consumer Electronics, 2025. [23] Z. Lin, Z. Chen, Z. Fang, X. Chen, X. Wang, and Y. Gao, “Fedsn: A federated learning framework over heterogeneous leo satellite networks,” IEEE Transactions on Mobile Computing, vol. 24, no. 3, pp. 1293–1307, 2024. [24] M. Hu, J. Zhang, X. Wang, S. Liu, and Z. Lin, “Accelerating Federated Learning with Model Segmentation for Edge Networks,” IEEE Trans. Green Commun. Netw., 2024. [25] Y. Zhang, Z. Lin, Z. Chen, Z. Fang, W. Zhu, X. Chen, J. Zhao, and Y. Gao, “Satfed: A resource-efficient leo satellite-assisted heterogeneous federated learning framework,” Engineering, 2024. [26] H. Zhu, J. Xu, S. Liu, and Y. Jin, “Federated learning on non-iid data: A survey,” Neurocomputing, vol. 465, pp. 371–390, 2021. [27] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al., “Advances and open problems in federated learning,” Foundations and trends® in machine learning, vol. 14, no. 1–2, pp. 1–210, 2021. [28] L. Xie, T. Luan, W. Cai, G. Yan, Z. Chen, N. Xi, Y. Fang, Q. Shen, Z. Wu, and J. Yuan, “dflmoe: Decentralized federated learning via mixture of experts for medical data analysis,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 10 203– 10 213. [29] Y. Feng, Y.-a. Geng, Y. Zhu, Z. Han, X. Yu, K. Xue, H. Luo, M. Sun, G. Zhang, and M. Song, “PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning,” in Proc. WWW, Apr. 2025, pp. 134–146.
[30] X. Zhang, J. Zhao, and Y. LeCun, “Character-Level Convolutional Networks for Text Classification,” Proc. NeurIPS, vol. 28, Dec. 2015. [31] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring Massive Multitask Language Understanding,” arXiv preprint arXiv:2009.03300, Jan. 2021. [32] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proc. AISTATS, vol. 54. PMLR, Apr. 2017, pp. 1273–1282.