DIFFERENTIALLY PRIVATE CLUSTERED FEDERATED LEARNING WITH PRIVACY-PRESERVING INITIALIZATION AND NORMALITY-DRIVEN AGGREGATION Jie Xu* , Haaris Mehmood* , Rogier Van Dalen† , Karthikeyan Saravanan* , and Mete Ozay*
arXiv:2604.20596v1 [cs.LG] 22 Apr 2026
*
Samsung R&D Institute UK (SRUK), † Samsung AI Centre Cambridge
ABSTRACT Federated learning (FL) enables training of a global model while keeping raw data on end-devices. Despite this, FL has shown to leak private user information and thus in practice, it is often coupled with methods such as differential privacy (DP) and secure vector sum to provide formal privacy guarantees to its participants. In realistic cross-device deployments, the data are highly heterogeneous, so vanilla federated learning converges slowly and generalizes poorly. Clustered federated learning (CFL) mitigates this by segregating users into clusters, leading to lower intra-cluster data heterogeneity. Nevertheless, coupling CFL with DP remains challenging: the injected DP noise makes individual client updates excessively noisy, and the server is unable to initialize cluster centroids with the less noisy aggregated updates. To address this challenge, we propose PINA, a two-stage framework that first lets each client fine-tune a lightweight low-rank adaptation (LoRA) adapter and privately share a compressed sketch of the update. The server leverages these sketches to construct robust cluster centroids. In the second stage, PINA introduces a normality-driven aggregation mechanism that improves convergence and robustness. Our method retains the benefits of clustered FL while providing formal privacy guarantees against an untrusted server. Extensive evaluations show that our proposed method outperforms state-of-the-art DP-FL algorithms by an average of 2.9% in accuracy for privacy budgets (ϵ ∈ {2, 8}). Index Terms— Clustered Federated Learning, Differential Privacy, Data Heterogeneity, Non-IID Data 1. INTRODUCTION Federated learning (FL) enables a distributed group of edge devices to collaboratively train a shared model while keeping raw user data on-device [1]. Despite this, the exchanged gradients or model updates can reveal statistical fingerprints that compromise user privacy [2]. Differential privacy (DP) [3] protects against such inferences by injecting calibrated randomness into the updates, yielding a rigorous privacy guarantee. Local DP (LDP) [4] requires client updates to be privatized at the edge and therefore protects against a malicious server. However, the higher privacy benefits of LDP leads to noisier updates and drop in model performance. The alternative solution is to add noise to the sum of contributions which is known as central DP (CDP) [3], and leverage secure sum protocols [5, 6] to ensure that the server only has access to the aggregated updates and is prevented from viewing individual contributions. A second fundamental challenge in FL is data heterogeneity: client data are often non-independent and identically distributed (non-IID) [7]. Under such conditions, standard FL algorithms such
as FedAvg [1] can converge slowly and yield suboptimal performance [7]. The situation deteriorates further when DP is imposed: the added DP noise amplifies the impact of data skew, exacerbating the performance degradation caused by non-IID data [8, 9, 10]. Clustered federated learning (CFL) has been recently proposed to address data heterogeneity in FL [11, 12]. Instead of fitting a single global model, CFL simultaneously learns a set of cluster-specific models and dynamically assigns clients to the cluster whose data distribution best matches their local statistics. Clients that share a similar distribution form a cluster, making the data within each cluster less non-IID and enabling each cluster to converge faster and to a higher accuracy. Existing CFL algorithms, such as IFCA [11], rely on privileged server data that resembles users’ data to initialize cluster models. An alternative approach employs random restarts to initialize cluster models arbitrarily and repeat training until a stable clustering structure emerges. However, both strategies are problematic in a DP setting. The former requires access to non-public data at the server, which compromises privacy and undermines practicality. The latter consumes additional privacy budget, resulting in noisier client updates. Moreover, a naı̈ve application of LDP in CFL introduces excessive noise, which distorts the intrinsic cluster structure and severely degrades model performance. On the other hand, simply applying CDP to CFL requires the server to be fully trusted, which is impractical in adversarial settings. To protect against an untrusted server, secure sum protocols such as SecAgg [5] are often deployed to ensure the server only has access to the aggregated contributions. However, such mechanisms prevent the server from accessing individual client updates, which are necessary for initializing cluster models in the first place. In this paper, we introduce Clustered Federated Learning with Privacy-preserving Initialization and Normality-driven Aggregation (PINA), a novel CFL framework that ensures DP without requiring privileged server data or random restarts. PINA uses privacypreserving sketches of client updates to construct robust initial cluster prototypes, enabling dynamic client assignment to the nearest cluster. In subsequent training rounds, PINA employs a normalitydriven aggregation mechanism that adaptively mitigates the impact of imbalanced client contributions. We believe that these advancements brings clustered FL closer to practical adoption, providing realistic privacy guarantees. The contributions of this work are summarized as follows: • We propose PINA, a clustered FL algorithm that jointly addresses data heterogeneity and privacy, requiring neither privileged server data nor random restarts. • We devise a privacy-preserving initialization scheme that produces accurate cluster prototypes directly from privatized client sketches. • We introduce a normality-driven aggregation step that restores the magnitude of the aggregated updates, improving robustness
©2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
against imbalanced client participation. • Our extensive experiments demonstrate that PINA consistently outperforms existing DP-FL methods on non-IID data by an average of 2.9% in test accuracy, particularly in more realistic, naturally non-IID environments.
server
client
initial model
clients
model updates
train
...
initialize
filter
cluster models
2. PRELIMINARY
clustering noisy updates
cluster centers
filtered updates
LDP
2.1. Federated Learning (FL) Overview of FL: At the start of each communication round t, a global model W t is provided by the server and a randomly sampled user set Kt is constructed. Each user k ∈ Kt trains the model locally to obtain Wkt and shares the model difference ∆tk = Wkt − W t back to thePserver. The server aggregates the updates W t+1 = W t + |K1t | k∈Kt ∆tk , and then proceeds to the next round.
Fig. 1. Stage 1 of PINA. Global cluster models are initialized in a privacy-preserving manner. ′
weights W0 ∈ Rd×d and introduces new trainable low-rank decomposition matrices B and A as follows: W0 + ∆W = W0 + BA,
(4) ′
2.2. Differential Privacy Differential privacy (DP) provides a formal definition to quantify the amount of private information an algorithm leaks regarding its input data. Formally, DP is defined as follows: Definition (Differential Privacy [3]) A randomized mechanism M : D → R satisfies (ϵ,δ)-DP if for any pair of adjacent datasets D and D′ , and any subset of outputs S ⊆ R, we have Pr[M(D) ∈ S] ≤ exp(ϵ) · Pr[M(D′ ) ∈ S] + δ,
(1)
where (ϵ,δ) is known as the privacy budget. Following [2], we use Gaussian mechanism [13] with noise N (0, Iσ 2 ) to achieve DP where σ = zS. Here, S denotes a predefined threshold for clipping client updates and z denotes the noise multiplier. We compute z using moments accountant [14] with Rényi Differential Privacy (RDP) [15] for a tight composition bound. For any α ∈ (1, ∞) and ϵ > 0, a randomized mechanism M satisfies (α, ϵ′ )-RDP if for all neighboring datasets D and D′ , we have: Definition (Rényi Differential Privacy [15]) For any α ∈ (1, ∞) and ϵ > 0, a randomized mechanism M satisfies (α, ϵ′ )-RDP if for all neighboring datasets D and D′ , we have α D(x) 1 Dα (M(D)||M(D′ )) ≜ log E ≤ ϵ′ . (2) α−1 D′ (x) To convert (α, ϵ′ )-RDP back to the standard (ϵ, δ)-DP framework, we adopt the established conversion method outlined in [16]. Specifically, a randomised mechanism M that satisfies (α, ϵ′ )-RDP also satisfies (ϵ, δ)-DP with ϵ = ϵ′ + log(
α−1 log α + log δ )− , α α−1
(3)
for any 0 < δ < 1. For CDP with secure sum, we additionally combine Gaussian mechanism with privacy amplification via sampling [17] by σ = zS , achieving a significantly reduced noise level compared to LDP. |Kt |
where B ∈ Rd×r is initialized to zeros, A ∈ Rr×d follows random Gaussian initialization and r ≪ min(d, d′ ). This effectively reduces the number of trainable parameters by an order of O(r/ min(d, d′ )). 2.4. Clustered Federated Learning Clustered FL algorithms group clients with similar distributions together. Thereby, clients within the same cluster suffer less from data heterogeneity and train via FL more effectively. Several works propose clustered FL methods, including CFL [12], IFCA[11] and PACFL[19], which use techniques such as cosine similarity, empirical loss and singular value decomposition to assign clients to clusters and train models. Clients typically respond with their cluster ID and trained model. To our knowledge, [20] and [21] are the only existing works that add DP to clustered FL. However, [20] uses LDP to privatize user updates throughout the entire training process which significantly degrades the model’s utility, making it infeasible for training large models [22]. Meanwhile, [21] provides sample-level privacy instead of user-level privacy which is a weaker form of privacy protection than the latter. It’s also been pointed out that the loss-based clustering in [21] breaks the sample-level privacy guarantees by leaking more information than allowed. In contrast, works including [23, 24] add secure aggregation to clustered FL without incorporating DP constraints, relying on a fully trusted server. 3. OUR METHOD: PINA 3.1. Overview Our proposed method PINA consists of two stages: (1) Cluster Model Initialization and (2) Clustered Model Training. In (1), we privately initialize cluster models from user updates; and in (2) we perform cluster identification and model training in a federated setting, privately updating global cluster models. The workflow of PINA is outlined in Algorithm 1. In this work, we use pre-trained transformer-based models, wellsuited for cross-device FL due to their ability to leverage large-scale server data for pre-training and efficient client-side fine-tuning, enabling robust performance on heterogeneous data [25, 22, 26, 27].
2.3. LoRA Low-Rank Adaptation (LoRA) [18] is a parameter-efficient finetuning (PEFT) method for transformer-based pre-trained models. Instead of training the entire weight matrix, it freezes the pre-trained
3.2. Cluster model initialization For a number of initial FL rounds T in , clients Kt ⊆ K sampled at round t ≤ T in apply LoRA with r = 1 to the same pre-trained
Algorithm 1 PINA 1: S ERVER 2: parameters: #rounds (initialization): T in , #rounds (training): T tr , a set of users: K, sampling rate: q ∈ (0, 1], noise multiplier: z, clipping threshold: S in , S, number of clusters: C 3: for each round t = 1, 2, . . . , T in + T tr do 4: Sample a subset Kt ⊆ K of users uniformly at random 5: with probability q. 6: if t ≤ T in 7: Server ← T RAIN(t, C, W0 , z, S in ) ▷ Equations 5 8: else 9: W t+1 ← W t + |K1t | S ECURE S UM DP 10: {T RAIN(t, C, W, z, S)}k∈Kt , z, S ▷ Scaling 11: 12: L OCAL DP(∆k , z, S) 13: σ =z·S 14: return N (0, σ 2 I) + C LIP(∆k ) 15: 16: S ECURE S UM DP({∆k }k∈K , z, S)
server
client
selected model
... clients
cluster models
estimate cluster identity train
normality-based scaling
model updates
secure sum
Fig. 2. Stage 2 of PINA with normality-based update scaling. ▷ Equations 7
17: σ =z·S P ▷ Equations 7 18: return N (0, σ 2 I) + k∈K C LIP(∆k ) 19: 20: T RAIN(t, C, W, z, S) 21: parameters: No. of epochs: E, batch size: β, lr: η 22: if t ≤ T in : i = 0 else: i = Fk (W ) ▷ Equations 6 23: Wi+ ← Wi 24: for each local epoch e = 1, 2, . . . , E do 25: B ← (split local data into batches of size β) 26: for batch b ∈ B do 27: Wi+ ← Wi+ − η▽Fk (Wi+ ) 28: if t ≤ T in 29: return L OCAL DP FILTER Wi+ − Wi , z, S 30: else 31: return W + − W
weights as in Equation 4, freeze all weights except for the value projection matrix of the last attention layer denoted by Wvl , and train on local data. This leads to the number of trainable parameters for the initialization stage being equal to the hidden size h. Each sampled client then applies LDP to the two largest positive/negative values from the model updates both in terms of absolute value, with all the other values converted to zeros. When applying LDP, we set the clipping threshold S in for stage 1 by: v !2 r u r u (S)2 n in t = S, (5) S = n h h where n is the number of non-zero values and S is the clipping threshold for stage 2. These vectors with four non-zero values are then shared directly with the server, which runs a clustering algorithm (e.g. k-means) to obtain initial cluster models to be trained in the second stage. Here, we minimize the number of non-zero values transmitted to the server since the server has to have access to individual model updates to perform clustering. LDP is therefore required to privatized the shared updates, which will be too noisy to give meaningful clustering results with a large number of non-zero values being shared. However, updates to four parameters provide limited information. It is therefore also necessary to only train a small number of param-
eters for this step to maximize the ratio of the number of updates received by the server to the total number of trainable parameters. 3.3. Cluster model training After initializing global cluster models, server sends the latest cluster models to each sampled client at the start of each communication round for training. As in [11], clients perform cluster identification based on training loss and train the selected cluster model on local data. Let Z = {z1 , . . . , zn } be the samples held by client k ∈ K. We define the empirical loss F associated with client k as follows: 1 X f (W ; z), (6) Fk (W ) = |Z| z∈Z where f (W ; z) is the loss function associated with sample z. Clients then clip model updates to a predefined threshold S by: S C LIP(∆k ) = ∆k · min 1, , (7) ∥∆k ∥2 and share the clipped updates with the server. The updates are then aggregated and noised before being applied to the global cluster models. The aggregation and noise addition can be implemented via a secure sum protocol such as SecAgg [5], where each client adds noise to their clipped updates known as distributed DP (DDP) [28], ensuring the aggregated updates have the correct noise [29]. The server receives only the noisy, aggregated updates, with no access to individual client contributions, providing privacy guarantees even against an untrusted server. As secure aggregation is orthogonal to our core contribution, we abstract it (highlighted in pink in Algorithm 1) as a black-box layer. After receiving the aggregated noisy updates through secure sum, for a number of communication rounds T no , the server normalizes the magnitude of each aggregate ∆i to the one with the smallest ℓ2 norm by: ∆i = ∆i ·
∥∆î ∥2 , ∥∆i ∥2
(8)
where i ∈ [[1, C]] and î = argminj∈[[1,C]] ∥∆j ∥2 with [[1, C]] = {1, . . . , C}. This is done to stabilize early training of cluster models. After round T in + T no , for the remainder of training, we alternatively scale the aggregated model updates to each global cluster model based on its normality, estimated via the Shapiro-Wilk test statistic [30] as follows: ( ∆i · P wi wj , if wi < 0.99 j∈[[1,C]] , (9) ∆i = ∆i · 0, otherwise
Table 1. Results and comparisons on rotated CIFAR-10, rotated FMNIST and FEMNIST. We boldface the best accuracy. Rotated CIFAR-10
Rotated FMNIST
FEMNIST
ϵ=∞
ϵ=2
ϵ=8
ϵ=∞
ϵ=2
ϵ=8
ϵ=∞
ϵ=2
ϵ=8
FedAvg FedProx (µ = 0) FedProx (µ = 1) FedNova IFCA
92.1±0.10 92.3±0.28 92.2±0.26 90.3±0.10 94.3±0.20
90.2±0.28 90.4±0.20 90.3±0.20 89.7±0.41 34.4±2.26
90.5±0.15 90.6±0.20 90.6±0.10 89.9±0.23 73.8±2.54
87.8±0.11 87.8±0.10 87.9±0.10 88.8±0.45 89.4±0.46
86.0±0.05 86.1±0.10 86.0±0.20 86.9±1.21 21.4±1.15
86.2±0.05 86.2±0.10 86.2±0.05 87.0±0.97 64.3±3.20
81.1±0.20 80.8±0.11 81.0±0.05 80.4±0.37 81.7±1.19
79.2±0.20 79.2±0.05 79.4±0.10 79.5±0.15 4.6±0.25
79.4±0.11 79.3±0.11 79.5±0.20 80.0±0.60 5.5±0.37
PINA (Ours)
93.8±0.30
92.6±0.25
92.9±0.15
89.9±0.10
88.9±0.28
89.0±0.23
83.7±0.10
82.3±0.30
82.5±0.11
4. EXPERIMENTS Experimental settings: We use privacy budget of ϵ ∈ {2, 8} which are commonly used in existing works [8, 9] and δ = |K|11.1 [2]. Following [2, 31, 26], we simulate a cohort size of 10k with a smaller cohort size to achieve a more realistic signal-to-noise ratio which represents industry scale more closely. We use rotated CIFAR-10 (C = 2), rotated FMNIST (C = 4) and FEMNIST (C = 2) for our experiments. Following [11], we generate the first two by applying the same rotation (0, 180 degrees for CIFAR-10 and 0, 90, 180, 270 degrees for FMNIST) to all images of a client. We set the total number of clients to 5,000 for CIFAR-10/FMNIST and 2,840 for FEMNIST. We use standard data augmentation (e.g. random cropping and horizontal flipping) to increase the size of the training set by 5x for CIFAR-10 and FMNIST. For all experiments, clients train for E = 10 local epochs with a batch size of β = 50 and learning rate of η = 0.01. We compare PINA with state-of-the-art (SOTA) methods including FedAvg [1], FedProx [32], FedNova [33] and IFCA [11]. For IFCA, we set the clipping threshold to the same value as our method and apply LDP for ϵ ∈ {2, 8}. It was shown in [11] that IFCA could be combined with personalized FL to improve performance even further. In this paper, we do not consider personalized FL. We exclude SCAFFOLD [34] from our experiments since it is designed for the cross-silo setting [35] and we only focus on the more challenging cross-device setting in this work. Client dropouts are not considered in our experiments. We use a 22M-parameter ViT-Small model pretrained on ImageNet-21k and publicly available via Hugging Face. We assume all clients are provided with this pre-trained model at the start of training, consistent with standard FL practices. Comparing with existing DP-FL methods: Table 1 shows results for different methods with non-private FL (ϵ = ∞) and DP-FL (ϵ ∈ {2, 8}). The results demonstrate that PINA consistently outperforms SOTA methods across all three datasets. In particular, for ϵ = 2, PINA achieves up to 2.9%, 2.9% and 3.1% improvements on rotated CIFAR-10, rotated FMNIST and FEMNIST, respectively.
For ϵ = 8, the improvements reach up to 3.0%, 2.8% and 3.2% on the same datasets. Notably, the improvements are most pronounced on the naturally non-IID dataset FEMNIST, highlighting the ability of our proposed method to handle real-world data heterogeneity. Overall, PINA shows an average accuracy improvement of 2.9% over SOTA methods for ϵ ∈ {2, 8} on non-IID data, validating its effectiveness in learning on heterogeneous user data while preserving user privacy. 0.93
1.0 0.8 0.6 0.4 0.2 0.0 0
20
40
IFCA( = ) IFCA+LDP( =2) PINA( =2) PINA+RI( =2) PINA+RI( =8) 60 80 100
round
(a) Clustering accuracy
accuracy
where wi denotes the Shapiro-Wilk test statistic for the ith cluster’s aggregated updates. Since each client shares updates to all cluster models, including the ones that are not selected and trained, clusters chosen by only a few clients experience significantly reduced update magnitudes after aggregation, leading to slow convergence. To address this, we propose this novel scaling mechanism, which restores the intended ℓ2 norm of the updates to global cluster models. This improves both the robustness and fairness of the proposed framework. To avoid applying amplified noise to a global cluster model with zero contributing clients, we zero out the update whenever the test statistic reaches the threshold of 0.99.
accuracy
Method
0.92 0.91 0.90 60
70
PINA( =2) PINA( =8) PINA+RI( =2) PINA+RI( =8) 80 90 100
round
(b) Classification accuracy
Fig. 3. Comparison of clustering and classification accuracy on CIFAR-10. PINA+RI denotes our proposed method with random initialization. Results are averaged over three runs. Ablation: Fig. 3a illustrates the clustering accuracy of IFCA and our proposed method under varying levels of added noise. As evident from the results, our method achieves optimal clustering with a stringent privacy budget of ϵ = 2. In contrast, incorporating LDP into IFCA throughout the entire training process has a detrimental effect on the clustering structure. Notably, our method attains optimal clustering even with random initialization when the privacy budget is relaxed to ϵ = 8, demonstrating its robustness. Furthermore, as shown in Fig 3b, our novel initialization mechanism enables the algorithm to converge to the optimal clustering structure significantly faster than random initialization, resulting in accelerated convergence and improved classification accuracy. 5. CONCLUSION In this work, we propose PINA, a privacy-preserving clustered FL framework that effectively mitigates data heterogeneity in DP-FL. By combining privatized client sketches for robust initialization and a normality-driven aggregation mechanism that accounts for imbalanced contributions, PINA achieves superior performance on nonIID data without requiring server-side privileged data or random restarts. Extensive experiments show that PINA consistently outperforms SOTA DP-FL methods under non-IID settings, achieving an average improvement of 2.9% in test accuracy.
6. REFERENCES [1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al., “Communication-efficient learning of deep networks from decentralized data,” in AISTATS, 2017. [2] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang, “Learning differentially private recurrent language models,” ICLR, 2018. [3] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography, 2006. [4] Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith, “What can we learn privately?,” SIAM Journal on Computing, 2011. [5] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, et al., “Practical secure aggregation for privacypreserving machine learning,” in CCS, 2017. [6] Yucheng Fu and Tianhao Wang, “Benchmarking secure sampling protocols for differential privacy,” in CCS, 2024. [7] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith, “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, 2020. [8] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, et al., “Federated learning with differential privacy: Algorithms and performance analysis,” IEEE TIFS, 2020. [9] Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut, “Differentially private federated learning on heterogeneous data,” in AISTATS, 2022. [10] Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao, “{PrivateFL}: Accurate, differentially private federated learning via personalized data transformation,” in USENIX Security, 2023. [11] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran, “An efficient framework for clustered federated learning,” NeurIPS, 2020. [12] Felix Sattler, Klaus-Robert Müller, and Wojciech Samek, “Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints,” IEEE TNNLS, 2020. [13] Cynthia Dwork, Aaron Roth, et al., “The algorithmic foundations of differential privacy,” Foundations and Trends® in Theoretical Computer Science, 2014. [14] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang, “Deep learning with differential privacy,” in CCS, 2016. [15] Ilya Mironov, “Rényi differential privacy,” in 2017 IEEE 30th computer security foundations symposium (CSF), 2017. [16] Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato, “Hypothesis testing interpretations and Renyi differential privacy,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2020. [17] Borja Balle, Gilles Barthe, and Marco Gaboardi, “Privacy amplification by subsampling: Tight analyses via couplings and divergences,” in NeurIPS, 2018. [18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, et al., “LoRA: Low-rank adaptation of large language models,” in ICLR, 2022.
[19] Saeed Vahidian, Mahdi Morafah, Weijia Wang, Vyacheslav Kungurtsev, Chen Chen, et al., “Efficient distribution similarity identification in clustered federated learning via principal angles between client data subspaces,” in AAAI, 2023. [20] Zaobo He, Lintao Wang, and Zhipeng Cai, “Clustered federated learning with adaptive local differential privacy on heterogeneous IoT data,” IEEE IoT, 2024. [21] Saber Malekmohammadi, Afaf Taik, and Golnoosh Farnadi, “Mitigating disparate impact of differential privacy in federated learning through robust clustering,” arXiv, 2024. [22] Jae Ro, Theresa Breiner, Lara McConnaughey, Mingqing Chen, Ananda Suresh, et al., “Scaling language model size in cross-device federated learning,” in FL4NLP, 2022. [23] Hasin Us Sami and Başak Güler, “Secure aggregation for clustered federated learning,” in ISIT, 2023. [24] Yulin Zhao, Zhiguo Wan, Zhangshuang Guan, et al., “Clusterguard: Secure clustered aggregation for federated learning with robustness,” Cryptology ePrint Archive, 2024. [25] Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang, “Federated learning from pre-trained models: A contrastive learning approach,” NeurIPS, 2022. [26] Jie Xu, Karthikeyan Saravanan, Rogier van Dalen, Haaris Mehmood, David Tuckey, and Mete Ozay, “DP-DyLoRA: Fine-tuning transformer-based models on-device under differentially private federated learning using dynamic low-rank adaptation,” arXiv preprint arXiv:2405.06368, 2024. [27] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, et al., “Rethinking architecture design for tackling data heterogeneity in federated learning,” in CVPR, 2022. [28] Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou, “A hybrid approach to privacy-preserving federated learning,” in AISec, 2019. [29] Slawomir Goryczka and Li Xiong, “A comprehensive comparison of multiparty secure additions with differential privacy,” IEEE Transactions on Dependable and Secure Computing, 2017. [30] Samuel Sanford Shapiro and Martin B Wilk, “An analysis of variance test for normality (complete samples),” Biometrika, 1965. [31] Congzheng Song, Filip Granqvist, and Kunal Talwar, “FLAIR: Federated learning annotated image repository,” NeurIPS, 2022. [32] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith, “Federated optimization in heterogeneous networks,” MLSys, 2020. [33] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” NeurIPS, 2020. [34] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, et al., “Scaffold: Stochastic controlled averaging for federated learning,” in ICML, 2020. [35] Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, et al., “Breaking the centralized barrier for cross-device federated learning,” NeurIPS, 2021.