Federated Deep Clustering Networks for High-Dimensional and Heterogeneous Data

Federated Deep Clustering Networks for High-Dimensional and Heterogeneous Data Morris Stallmann, Charalampos S. Kouzinopoulos, Marcin Pietrasik, Anna Wilbik

arXiv:2609.21829v1 [cs.LG] 18 Sep 2026

Department of Advanced Computing Sciences (DACS) Maastricht University Maastricht, Netherlands {m.stallmann, charis.kouzinopoulos, marcin.pietrasik, anna.wilbik}@maastrichtuniversity.nl

Abstract—Clustering high-dimensional data is a fundamental task in unsupervised machine learning with applications to a variety of domains. In the centralized data scenario, this task is commonly solved using deep clustering methods that utilize deep neural network architectures to learn clusteringfriendly latent space representations. In Federated Learning, where data is distributed between clients and is private, deep clustering methods are less explored. In particular, recently introduced federated deep clustering methods, despite showing very promising performance, still fall short in reliably providing good performance if data across clients are non-identicallyindependently distributed. In this work, we introduce a generalization of Deep Clustering Networks to the federated scenario, named FedDCN, that simultaneously optimizes a reconstruction loss and a clustering loss. To ensure robustness and latent space alignment in non-identically-independently distributed data scenarios, FedDCN generates synthetic data augmentations, and its learning objective includes a geometric regularization for latent space alignment. Through experimental evaluation, the effectiveness of the approach under IID and non-IID assumptions is demonstrated, and future research directions are identified. Index Terms—federated learning, federated clustering, deep clustering, unsupervised machine learning, representation learning, data heterogeneity, data augmentation

I. I NTRODUCTION Clustering is the process of partitioning objects in a data space such that objects in the same partition are similar to each other and different from objects in other partitions. It helps to organize data in the absence of labels and has been applied to various use cases, such as fraud detection [1], image segmentation [2], acoustic scene identification [3], video understanding [4], as well as to a number of application areas including logistics, manufacturing, energy, and healthcare [5]. In many real-world deployments, data are inherently decentralized and remain distributed across multiple clients. Aggregating such data at a central repository is often impractical due to communication and storage constraints as well as privacy considerations. Federated Learning (FL) addresses these challenges by collaborative model training over decentralized datasets while keeping raw data local to each client. Participating clients iteratively communicate model updates, thus mitigating privacy risks and other constraints associated with data centralization. Consequently, it is not surprising that federated clustering has attracted increasing attention and a number of methods

have recently been proposed and extended to the federated setting (see Section II). Classic approaches such as federated versions of k-means or fuzzy c-means have been successfully generalized to the federated setting and show robust behavior even under the assumption of non-IID data. However, these “classical” methods struggle with high-dimensional data [6], [7]. In the non-federated setting, deep clustering methods are widely recognized as state-of-the-art when clustering highdimensional data [8], [9]. Deep clustering methods combine deep neural network (DNN) architectures for low-dimensional and clustering-friendly latent representation learning with grouping mechanisms to identify meaningful data partitions. To enable clustering of high-dimensional data in the FL setting, a number of recent works introduce federated deep clustering protocols. Despite very promising results, many of these pioneering approaches fail to reliably produce highquality data partitions under the assumption of data heterogeneity across clients (see Section II-B). Statistical data heterogeneity, or non-IIDness, refers to situations where the local datasets follow different distributions, and remains one of the key challenges in FL (see Section II-C). In this work, we specifically study the impact of label skew and refer to it as non-IIDness throughout the paper. This work aims to address one of the main challenges in federated deep clustering and move towards greater robustness against client data heterogeneity. To this end, we propose a federated protocol for learning Deep Clustering Networks (DCN) [10], a well-known deep clustering architecture and optimization scheme in the centralized case that was introduced as an improvement over Deep Embedded Clustering (DEC) networks. DEC is the base for state-of-the-art federated deep clustering methods that this work aims to improve through the introduction of Federated Deep Clustering Networks (FedDCN). To instill non-IID robustness, we propose a batch augmentation technique in combination with a geometric regularizer that complements the network’s autoencoder architecture and is well grounded in existing research on aligning local and global distributions. The main contribution of this paper is a federated deep clustering framework that: 1) is the first to generalize DCN to the federated setting. 2) does not assume access to a centralized dataset for pretraining, unlike other approaches, thereby being more

widely applicable in privacy-sensitive contexts. 3) achieves state-of-the-art performance on benchmark datasets and robustness against data heterogeneity through synthetic data augmentation and a geometryaware loss function. Section II introduces related works; our proposed method is described in Section III; its experimental evaluation is described and discussed in Section IV before concluding with final remarks in Section V. II. R ELATED W ORK This section provides an overview of deep clustering and prior work on federated deep clustering, and discusses strategies for handling data heterogeneity in FL to clearly position our work within the existing literature. A. Non-Federated Deep Clustering Deep clustering methods utilize the representation learning capabilities of DNNs and learn clustering-friendly (lowdimensional) latent spaces from high-dimensional data spaces to address the shortcomings of traditional clustering methods. A taxonomy based on network architecture and optimization scheme is suggested in [8] to define four broad groups of deep clustering methods. The methods in the first group use DNNs to directly output latent representations and cluster assignments (DNN-based clustering). The second group comprises methods that utilize autoencoders (AE) to learn a latent representation (AE-based clustering), and the clustering itself does not necessarily have to be performed by a DNN. Our proposed method falls into this group. One of the fundamental methods in this category is DEC, introduced in [11]. It is one of the first algorithms to explicitly optimize latent space representation mapping so that a clustering loss is minimized using stochastic gradient descent. Later, DCN was introduced in [10] as an improvement of DEC. DCNs train AEs and a k-means model to simultaneously minimize reconstruction loss and k-means clustering loss. Methods that fall into the third group utilize Generative Adversarial Networks (GANs), where the adversarial training is augmented with a clustering loss. In CatGAN [12], for example, the discriminator has to decide whether a sample is real or fake and has to estimate its cluster assignment. Lastly, the fourth group consists of Graph Neural Network (GNN)-based methods that use GNNs to map graph nodes to a latent space where the clustering is performed subsequently. For an exhaustive overview, refer to recent surveys [8], [9]. AE-based methods have an advantage over DNN-based methods in the federated setting: the encoder-decoder architecture can be utilized to address non-IIDness through the generation of synthetic data, as described in Section III below. GAN-based methods may have the same advantage, but are generally considered to be harder to train [8]. We did not consider GNN-based methods, since the input of this work is not graphs.

B. Federated Deep Clustering Recently, a number of federated deep clustering methods have been introduced, of which the majority being DNN- or AE-based. One of the first AE-based federated deep clustering methods is a federated version of DEC (F-DEC) [13]. FDEC uses convolutional autoencoders in combination with a clustering layer and optimizes a weighted sum of KullbackLeibler (KL) divergence (as clustering loss) and reconstruction loss. The approach assumes the availability of a centralized dataset for pretraining and achieves strong performance (accuracy and normalized mutual information, see Equation (8)) on IID datasets. However, the performance decreases by up to 50% on the MNIST dataset as non-IIDness increases. Similarly to F-DEC, FDEC, as proposed in [14], minimizes KL divergence, but additionally applies k-means clustering in the latent space. Moreover, the method uses stacked autoencoders and does not require access to a centralized dataset. In their experimental evaluation, the authors observe strong results, but also decreased reliability in non-IID scenarios (15% accuracy on the MNIST dataset), despite using a robust cluster center aggregation strategy. In the decentralized FL setting, the work in [15] introduces an AE-based method, where clients communicate directly with each other. While the authors demonstrate good performance in the IID scenario, the non-IID scenario was not part of the experimental evaluation. In [16], [17], DNN-based federated deep clustering methods were introduced. Both methods are based on contrastive loss functions and show promising results in IID data scenarios, while the experiments also uncover the impact of non-IIDness (a drop of 25%-29% on the clustering performance). C. Handling Non-IID Data in Federated Learning A key challenge in FL and federated deep clustering is data heterogeneity between clients [18]–[20] (also see previous Section II-B). In FL, data heterogeneity, or non-IIDness, refers to the divergence of distributions between the local and private data of clients. If not mitigated, this can lead to convergence or model degradation issues. The work in [19] categorizes approaches to handle nonIIDness into three broad categories: data-, algorithm-, and system-based approaches. Data-based approaches address nonIIDness at the most fundamental level and aim to harmonize local data distributions through data sharing or data augmentation. While data sharing approaches, such as in [21], are highly effective, they typically assume access to a centralized (IID) dataset, which may be unavailable in privacy-preserving settings. As an alternative, it has been suggested to apply data augmentation techniques to approximate IID distributions rather than sharing raw data instances, including approaches that use synthetic data creation [22], [23]. Algorithm-based approaches aim to address the non-IID problem through the learning algorithm. For example, methods such as FedProx [24] introduce a regularization term in the local learning objective that penalizes deviation from the weights of the global model.

A. Local Deep Clustering and Synthetic Data Creation To create a synthetic IID reference dataset, we adapt the DCN model and its AE architecture. Each client learns a DCN using its local dataset (Section III-A1) and shares the local latent cluster centers as well as local decoders with the central server. The central server constructs a synthetic dataset by sampling around the latent space cluster centers and mapping the samples back to the observed space (Section III-A2). 1) Local Deep Clustering Networks - Pretraining and Alternating Optimization: The proposed DCN consists of a stacked AE and a clustering module that operates in the latent space. Training the DCN involves optimizing the following objective: min

wc ,[klc ]K l=1

Fig. 1. Illustration of the three stages in FedDCN training: synthetic data creation, pretraining, federated training.

Lc

with Lc = αLcrecon + λLcclust + βLcgeom ,

Methods that fall into the system-based category alter the federated system. Most notably, clustered FL falls into this category. In clustered FL approaches, such as IFCA [25], clients with compatible data are grouped and learn one model per group instead of a single model for all clients. Deep clustering methods solve two fundamental tasks without label supervision: representation learning and input space partitioning. Robust partitioning methods in non-deep clustering have been introduced that are able to deal with nonIID data. [26] and [27] combine local k-means or fuzzy cmeans with central server-side k-means averaging and demonstrate the robustness of these approaches to non-IID data. However, learning representations in unsupervised FL remains a challenge, especially under data heterogeneity [28], [29]. As a solution, the authors of [29] introduce dictionary and alignment modules that depend on access to a shared dataset. Our approach to address non-IIDness is a hybrid of dataand algorithm-based methods.

III. P ROPOSED M ETHOD The primary contribution of this work is a federated clustering method that performs well on IID and non-IID highdimensional data, without requiring direct access to raw data instances, thereby protecting data privacy. Previous studies have shown that AE-based federated deep clustering shows promising performance on high-dimensional IID data, but still struggles with non-IID data (see Section II-B). To address non-IIDness during the federated training process, data-based approaches have been shown to be very effective, especially in federated unsupervised representation space learning (see Section II-C). The AE architecture of DCNs allows us to create a synthetic dataset that approximates the global data distribution without having direct access to the raw data instances. This motivates the design of our three-stage federated deep clustering framework illustrated in Figure 1.

(1)

Lcrecon = Lcclust = Lcgeom =

Nc X

||xi − g(f (xi ))||22 ,

i=0 Nc K X X l=1 i=0 Nc X

1xi assigned to kl ||f (xi ) − kl ||22

||f (xi ) − umap(xi )||22 ,

(2) (3)

(4)

(5)

i=0

where wc are the weights of client c’s AE network, [k c ]K l=1 are client c’s latent space cluster centers, K is the number of clusters, and Nc = |X c | is the number of data points of client c. The AE’s encoder layers are stacked on top of each other, a bottleneck layer is added, and the encoder layers are mirrored to define the decoder layers. Each layer applies batch normalization and uses ReLu as the activation function. Note that we introduce a geometric loss term Lcgeom that the original DCN formulation does not contain. This is to prevent distortions in the latent space (Section III-A3). The optimization algorithm is similar to the one introduced by [10]. First, the network is pretrained with λ = 0 (that is, ignoring the clustering loss) for Tpre epochs using the Adam optimizer [30]. Second, an alternating optimization algorithm minimizes Lc . The algorithm alternates between two steps: updating the network parameters wc with fixed cluster centers [k c ]K l=1 and updating cluster centers and assignments with fixed wc (see Algorithm 1 for an overview). After local DCN optimization, each client computes the standard deviations of the distances between each cluster center and the points assigned to it. Every client c = 1, . . . , C shares standard deviations σlc of all clusters l = 1, . . . , K with the central server, where these standard deviations inform the latent space sampling (Section III-A2). 2) Synthetic Data Creation - Sample and Decode: Given all clients’ latent space cluster centers, decoders, and per-cluster standard deviations, the central server constructs the synthetic dataset to be used for pretraining and latent space alignment. Given a client c, a cluster center klc , and the corresponding

Algorithm 1 Local DCN optimization // Pretraining 1: for t = 1, . . . , Tpre do c 2: wtc ← Adam(wt−1 ) // minimizing (2) with λ = 0 3: end for 4: // DCN training 5: for t = 1, . . . , T do c 6: wtc ← Adam(wt−1 ) // minimizing (2) with λ ≥ 0 c K 7: [k ]l=1 ← kmeans(fwtc (X c )) 8: Update cluster assignments 9: end for standard deviation σlc , the central server draws Nlsynth latent space samples from an isotropic Gaussian centered at kl with covariance Σcl = σlc ∗ I, where I is the identity matrix. Then, the synthetic dataset is created by decoding the latent samples using client c’s decoder gc : X synth =

C [ K [

{gc (xlatent ) : xlatent ∼ N (klc , Σcl )}. (6)

c=1 l=1

Note that assuming isotropic Gaussians is a simplification that likely introduces inaccuracies in the latent space sampling, but also protects clients’ data privacy as it reveals relatively little information about clients’ latent space distributions. In our experiments, we fix the synthetic dataset size to 5000 samples. Each client’s decoder is used to generate a number of samples proportional to its local dataset size. The number of samples per cluster center Nlsynth is derived from the number of samples within a radius of one standard deviation of the cluster center and is proportional to it. That way, we prioritize each of the clients’ most cohesive centers and deprioritize less cohesive centers, indicating a weaker data partition. 3) Geometry Regularization: Injecting geometric information into the loss function serves two main purposes. Firstly, it can prevent latent space distortions [31]. Since we sample in the latent space within a radius of one standard deviation around cluster centers, these distortions may lead to imbalanced or non-IID synthetic data (thereby defeating their purpose of providing alignment in the non-IID scenario). Secondly, the geometric loss introduces latent feature space alignment between clients during the federated training process (Section III-C). The central server aligns the local representation spaces by injecting the same geometric information into every client’s local learning objective. It shares synthetic datapoints with their geometric embeddings, and the clients include the geometry regularization (Equation (5)) in the local learning objectives. This is similar to aligning feature spaces as in [29], without accessing raw data instances. To inject geometric information, we follow the approach of [32] and introduce a regularization term Lcgeom (Equation (5)) in the loss function. For each feature vector x, its geometric embedding umap(x) is computed, and large distances between the learned and geometric embeddings are penalized. UMAP (Uniform Manifold Approximation and Projection) is

a manifold learning technique that maps data from a highdimensional space to a low-dimensional space while preserving the (assumed) geometric structure and pairwise distances [33]. Thus, umap : Rp → Rd denotes the function that maps a high-dimensional feature vector x ∈ X c to its geometric d-dimensional embedding. B. Initialization and Pre-Training The central server initializes the global model by pretraining it on the synthetic dataset X synth and its geometric embeddings X geom = {umap(x) : x ∈ X synth }. After creating X geom , Algorithm 1 is applied to minimize Equation (2). At the end of this phase, the central server shares global model weights w0 , cluster centers [kl ]K l=1 , and datasets X synth and X geom with the clients. C. Federated Simultaneous Optimization of Representations and Clustering In the final stage, the clients train the DCN PCcollaboratively 1 c L . Similarly to local and minimize: minw,[kl ]K c=1 Nc l=1 DCN optimization, federated optimization alternates between updating global autoencoder weights wt with fixed cluster centers [kl ]K l=1 and updating cluster centers with fixed autoencoder weights for a predefined number of epochs Tf ed . In the AE weights update phase in epoch t, each client c first optimizes Equation (2) locally using the Adam optimizer. Note that each batch of the clients’ local dataset is augmented with a batch of the same size from the shared synthetic dataset X synth . After the local optimization, the resulting local AE weights wtc are shared with the central server, which applies federated to deriveP the new global AE weights wt ← PC Nc averaging C c , where N = w t c=1 N c=1 Nc is the total number of datapoints. Then, the central server shares the updated wt . Using the updated global weights, each client c maps its data into the new latent space, where it applies kmeans to identify new local cluster centers [klc ]K l=1 . After collecting all local latent cluster centers, the central server runs kmeans on them to derive the new global latent space cluster centers [kl ]K l=1 and shares them with the clients. The federated training process is outlined in Algorithm 2. IV. E XPERIMENTAL E VALUATION This section evaluates our proposed method with respect to the following criteria: effectiveness in comparison to related federated deep clustering methods, impact of non-IID data, impact of an increasing number of clients (or, decreasing the number of local datapoints as the dataset is fixed), and sensitivity to initialization and hyperparameter choice. The following subsections describe the experimental design before describing and discussing the results. Code to replicate the experiments is available on GitHub: https://github.com/ stallmo/fed-dcn.

Algorithm 2 FedDCN training protocol 1: Local pretraining according to Algorithm 1. 2: Create X synth according to Equation (6). 3: Create X geom synth 4: w0 , [kl ]K // Initialize global l=1 ← Algorithm 1 with X

model AE weights and cluster centers on the central server synth 5: broadcast(w0 , [kl ]K , X geom ) l=1 , X 6: for t = 1, . . . , Tf ed do 7: weights ← {} 8: for client ∈ {1, . . . , C} do 9: wtc ← client.LocalAeU pdate() 10: weights ← weights ∪ {wtc } 11: end for PC 12: wt ← c=1 NNc wtc 13: broadcast(wt ) 14: localCenters ← {} 15: for client ∈ {1, . . . , C} do 16: clientCenters ← client.LocalClustering() 17: localCenters ← localCenters ∪ {clientCenters} 18: end for 19: [kl ]K l=1 ← kmeans(localCenters) 20: broadcast([kl ]K l=1 ) 21: end for

A. Data In the centralized setting, DCNs were developed as an enhancement of DECs and, hence, our primary goal is to compare FedDCN with the federated versions of DECs, namely F-DEC and FDEC (see Section II-B). In our experiments, the proposed method is trained on the same image datasets as F-DEC and FDEC: MNIST, FashionMNIST, and USPS, and evaluated on the standard test splits to be able to compare our results with F-DEC and FDEC. To create federated IID and non-IID scenarios, we follow the standard protocol of [19], [34]. Specifically, the training data are distributed across clients according to the Dirichlet distribution, based on class labels with concentration parameters αdirichlet = 1000.0 (approximate IID) and αdirichlet = 0.5 (non-IID). All metrics are calculated on the unseen test dataset. B. Evaluation Metrics The related deep clustering methods F-DEC and FDEC are evaluated using clustering accuracy (ACC) and normalized mutual information (NMI), which we also include in our evaluation: P maxπ k conf usion[k, π(k)] ACC = , (7) N I(Y, C) N M I(Y, C) = 1 . (8) [H(Y ) + H(C)] 2 where conf usion is the confusion matrix of the assignment of clusters and the ground truth labels and π is a one-to-one mapping between the assignment of the cluster and the labels,

TABLE I H YPERPARAMETERS SEARCH SPACE . Hyperparameter AE layer dimensions Bottleneck dimension d β pre during pretraining β f ed during federated training λ α

Search Space { (128), (256, 128), (512, 256), (512, 256, 128), (1024, 512, 256, 128, 64) } [3, 16] [0.0001, 1.0] [0.0001, 1.0] [0.0001, 1.0] [0.1, 10.0]

N is the total number of points in the test set. NMI measures the agreement between the set of ground truth assignments Y and the cluster assignments C and normalizes it by the mean entropy of the sets. I(·, ·) denotes the mutual information score and H(·) is the entropy [35]. Since the calculation of these external cluster validation metrics relies on the availability of ground truth labels (which may not be available in real-world clustering), we also include an internal cluster validation metric in our evaluation. Unlike validation metrics that require the calculation of pairwise distances, the Davies-Bouldin index (DBI) canP easily be calcuK 1 R , where lated in a federated setting [27]: DBI = K PN i=1 i Si +Sj 1 Ri := maxi̸=j Rij , Rij = Mij , Si = N j=1 ||xj − ci || is PD the “cluster spread”, and Mij = ( k=1 ||ci [k] − cj [k]||) is the “center separation”. Intuitively, good clusters i are cohesive (low spread Si ) and well separated (high Mij ∀j). Hence, the lower the DBI, the better the clustering. Note that our results are optimized for accuracy to facilitate comparison with existing works. In practice, ground truth label information may not be available, and optimization for internal validation metrics like the DBI may be more appropriate. In additional experiments, we observe that the DBI can be improved at the expense of lower ACC and N M I. C. Experimental Protocol The experimental evaluation follows three steps per data scenario (dataset and its distribution across clients). First, we search for the hyperparameters that give the best performance in terms of accuracy according to Equation (7) on the train set, then calculate all metrics in the five-client scenario, and lastly calculate the 20-client scenario metrics. First, given a data scenario, we apply hyperparameter optimization (HO) for 50 rounds. The HO is implemented using Optuna [36]. After this step, the optimal hyperparameters are fixed. Second, with the optimal hyperparameters, we repeat the same experiment five times with different seeds due to the sensitivity of the clustering algorithms to initialization. We report both the best and the average results. All reported metrics are calculated on the test set. Lastly, the number of clients is increased from five to 20, as in the experimental evaluation of F-DEC and FDEC. Since the dataset and its size are fixed, this leads to every client having fewer data to learn from, potentially leading to weaker AEs for synthetic data creation and a negative impact on clustering performance. The best and average results are reported.

TABLE II B EST RESULTS ON THE MNIST, FASHION -MNIST, AND USPS DATASETS ( AVG . OVER FIVE RUNS USING THE SAME HYPERPARAMETERS IN PARANTHESES ). *R ESULTS ARE OBTAINED USING A DIFFERENT LATENT SPACE SAMPLING STRATEGY DURING SYNTHETIC DATA CREATION DUE TO THE SMALL DATASET SIZE .

MNIST Clients 5

Scenario IID Non-IID

20

IID Non-IID

Fashion-MNIST Clients Scenario 5

IID Non-IID

20

IID Non-IID

USPS Clients 5

Scenario IID Non-IID

20

IID Non-IID

Accuracy 0.8026 (0.7643 ±0.0525) 0.7799 (0.7328 ±0.0515) 0.7160 (0.6838 ±0.0206) 0.7636 (0.7029 ±0.0475)

NMI 0.7592 (0.7445 ±0.0144) 0.7922 (0.7693 ±0.0221) 0.6772 (0.6666 ±0.0103) 0.7442 (0.6981 ±0.0423)

DBI (latent space) 0.6915 (0.7395 ±0.0391) 0.9905 (1.0758 ±0.0826) 0.7462 (0.7929 ±0.0396) 1.066 (1.1555 ±0.0925)

Accuracy 0.6427 (0.5866 ±0.0443) 0.5885 (0.5650 ±0.0253) 0.6028 (0.5849 ±0.0117) 0.5561 (0.5241 ±0.0318)

NMI 0.6491 (0.6164 ±0.0183) 0.6103 (0.6022 ±0.0090) 0.6193 (0.6003 ±0.0182) 0.5956 (0.5720 ±0.0269)

DBI (latent space) 0.1422 (0.3504±0.1597) 0.6816 (0.8389 ±0.1238) 0.3049 (0.4184 ±0.116) 0.8395 (0.8853 ±0.0452)

Accuracy 0.7848 (0.7521 ±0.0448) 0.6956 (0.6239 ±0.0577) 0.7653* (0.6994 ±0.0597) 0.7803* (0.7159 ±0.0372)

NMI 0.7472 (0.7323 ±0.0164) 0.7364 (0.7100 ±0.0280) 0.7036* (0.6825 ±0.0095) 0.7076* (0.6925 ±0.0124)

DBI (latent space) 0.3812 (0.576 ±0.1546) 0.5257 (0.6256 ±0.1066) 0.3177* (0.3930 ±0.0579) 0.8780* (0.9764±0.0634)

D. Results This section presents and discusses the experimental results, organized by the main research questions. All results of our experiments can be found in Table II, a comparison with FDEC and FDEC can be found in Table III, and the optimal hyperparameters in terms of accuracy in Table IV. 1) Comparison with related deep clustering methods: The proposed method is compared with F-DEC and FDEC. We report the best results of our method and compare them to the results as reported in the original evaluations of F-DEC and FDEC, as can be seen in Table III. We observe similar performance in most scenarios. In particular, in the IID scenarios, FedDCN’s performance matches the reported performance of both F-DEC and FDEC in most cases. In the non-IID scenarios, a notable performance difference can be observed in the MNIST experiments. While F-DEC and FDEC are negatively influenced by data heterogeneity, FedDCN achieves even better performance than in the IID scenario. In the Fashion-MNIST experiments, the accuracy and NMI of FedDCN drop slightly more than those of FDEC. In absolute terms, the performance is still almost identical. On the USPS dataset, both FDEC and FedDCN are not strongly affected by data heterogeneity. However, FDEC achieves a better NMI score in both scenarios. F-DEC is outperformed by both FDEC and FedDCN by a larger margin. In summary, the methods have different strengths and weaknesses. On the one hand, the proposed method FedDCN is the only one that demonstrates strong non-IID robustness

TABLE III P ERFORMANCE COMPARISON OF F ED DCN ( OURS ) AND THE RESULTS AS REPORTED BY F-DEC [13] AND FDEC [14] IN FEDERATED IID AND NON -IID SCENARIOS WITH 20 CLIENTS . F-DEC FDEC MNIST IID ACC 0.74 NMI 0.65 0.68 MNIST Non-IID ACC 0.57 0.63 NMI 0.48 0.58 USPS IID ACC 0.79 NMI 0.82 USPS Non-IID ACC 0.68 0.78 NMI 0.54 0.79 Fashion-MNIST IID ACC 0.59 NMI 0.61 Fashion-MNIST Non-IID ACC 0.58 NMI 0.60

FedDCN 0.72 0.68 0.76 0.74 0.77 0.70 0.78 0.71 0.60 0.62 0.56 0.60

across all three benchmark datasets. This can be explained by the different approaches to handle data heterogeneity. While FDEC relies on robust cluster center aggregation, FedDCN also applies synthetic data augmentation and a geometry-aware loss term, leading to improved robustness. On the other hand, FedDCN achieves worse NMI on some datasets like USPS. This can be attributed to the geometry loss term. In particular, the optimal geometric loss weight during pretraining is the highest in the USPS experiments (see Table IV), potentially enforcing umap embedding alignment too strongly. Moreover, the hyperparameter search optimized accuracy and not NMI, possibly leading to suboptimal NMI performance. 2) Impact of Non-IID Data: In the MNIST dataset experiments with five clients, the best accuracy drops from 0.80 to 0.78. On the contrary, the NMI increases with the introduction of non-IIDness. With 20 clients, accuracy and NMI both increase in the non-IID scenario. On the FashionMNIST dataset, a negative impact of non-IIDness can be observed in experiments with five and 20 clients in both metrics. With five clients, the accuracy drops from 0.64 to 0.59 and with 20 clients from 0.60 to 0.56 while the NMI is less impacted. Lastly, the experiments with the USPS dataset show an impact on accuracy only in experiments with five clients. The NMI is not affected in experiments with five or 20 clients. Furthermore, the accuracy even increases in the non-IID experiments with 20 clients. Notably, the DBI increases (indicating poorer data partitions) in all cases where data heterogeneity is introduced. We attribute that to the fact that we optimize for an unrelated metric and expect the effect to be weaker when optimizing for DBI. Overall, the impact of non-IIDness is small, and sometimes even positive, which suggests that the combination of synthetic data augmentation and geometry-aware regularization can help addressing non-IID issues in federated deep clustering. 3) Increasing Number of Clients: In our experiments, we observe a drop in clustering performance as the number of

clients increases, similar to the observations in [13] and [14]. On the one hand, the accuracy decreases sharply from 0.8 to 0.72 in the MNIST IID scenario when increasing the number of clients from five to 20. On the other hand, accuracy decreases only from 0.78 to 0.76 in the MNIST non-IID experiments. In the Fashion-MNIST scenarios, the impact on accuracy is between −5% and −7% relative performance. Similarly, the impact on NMI is between −12% (MNIST IID) and −3% (Fashion-MNIST non-IID). In the USPS experiments, only a light impact is observed. The accuracy and DBI improve in the IID and regresses in the non-IID scenarios while the NMI slightly decreases in both. Note that we had to adjust the data sampling strategy in the synthetic data creation step after stage 1 in the USPS experiments with 20 clients. The low number of data points per client and latent space sampling strategy (Section III-A2) led to multiple zero cluster counts and a misaligned synthetic dataset. Rather than deciding the number of points to sample per cluster center based on the number of points within a radius of one standard deviation around the centers, the number of points is set to the total number of points assigned to the cluster center. We also observe a positive effect of that new sampling strategy on non-IIDness performance with five clients in the USPS experiments but leave a more principled investigation for future work. 4) Sensitivity to Initialization: Clustering algorithms, such as k-means are sensitive to cluster center initialization and the randomness it introduces [37]. FedDCN initializes k-means multiple times, both locally in stage 1 and globally in stage 2. Moreover, neural networks are also known to be susceptible to initialization. To study the influence of initialization in FedDCN, each optimal run is repeated five times with the same hyperparameters but different random seeds. Across all experiments, we observe a noticeable, but moderate effect of randomness. For example, in the USPS experiments with 20 clients, the best run (0.7653 accuracy) is significantly better than the average over five runs (0.6994). In other cases, such as the IID Fashion-MNIST experiments with 20 clients, the best performance (0.6028) is close to the average over the five runs (0.5849). The variability in NMI is consistently lower than the variability in accuracy while DBI varies the most, presumingly because it is completely unrelated to the hyperparameter optimization objective. In summary, it is advisable to repeat the training multiple times due to randomness during initialization and training. 5) Impact of Hyperparameters: As Table I shows, optimal hyperparameters vary widely between different data scenarios, highlighting the necessity of hyperparameter optimization. For example, the best geometric loss weight β during pretraining on the USPS dataset is 0.3711 in the IID scenario and 0.1970 in the non-IID scenario while it is ≤ 0.0304 in the remaining experiments. Similarly, the reconstruction error weight α and cluster loss weight λ vary by multiple orders of magnitude. Furthermore, an ablation study is conducted to understand the importance of including the geometric regularization term in the loss function. The experiments with five clients are

Fig. 2. Effect of removing the geometric loss on the NMI. TABLE IV O PTIMAL HYPERPARAMETERS ( OPTIMIZED FOR ACCURACY ). Data Scenario

AE layers

d

β pre

β f ed

λ

α

MNIST IID MNIST non-IID Fashion IID Fashion non-IID

[256, 128] [512, 256, 128] [512, 256, 128] [512, 256] [1024, 512, 256, 128, 64] [512, 256]

3 4 6 6

0.0037 0.0076 0.0131 0.0304

0.0204 0.0003 0.0003 0.0008

0.0002 0.0003 0.2814 0.0005

0.8273 2.8550 0.2137 0.3225

USPS IID USPS non-IID

Learn. rate 0.0086 0.0014 0.0008 0.0001

4

0.3711

0.0241

0.0470

0.1523

0.0002

5

0.1970

0.0017

0.0006

0.5828

0.0024

repeated with setting β pre = β f ed = 0 and keeping the remaining best hyperparameters fixed for every data scenario. Including the geometric loss has a positive impact on all metrics in all data scenarios with the exception of the DBI in the USPS IID and Fashion-MNIST IID scenarios. Whether this effect persists when optimizing for the DBI is an interesting follow-up research question. The positive effect on NMI (which is illustrated in Figure 2) is slightly stronger than on ACC. The ACC and NMI performance gains in all data scenarios let us conclude that the geometric loss is an important component of the method to align local feature spaces and avoid latent space distortions. However, further ablation studies will enhance understanding of the components’ interplay and the contribution of each loss term in different IID and non-IID scenarios. V. C ONCLUDING R EMARKS This work introduces FedDCN, a novel federated deep clustering method, that generalizes the concept of Deep Clustering Networks to the FL setting. The proposed method achieves comparable performance to other federated deep clustering techniques in the IID setting and favorable robustness against data heterogeneity. That robustness stems from a synthetic data augmentation protocol and the introduction of a geometryregularized loss term. Our experimental evaluation reveals sensitivity to hyperparameter choices and hints to improvements through different latent space sampling strategies. An ablation study shows the importance of including the geometric loss component and motivates an extended ablation study to better understand the impact of further components in FedDCN. Moreover, future work will focus on evaluation on additional datasets and explore the application to drift or anomaly detection, possibly using alternative AE architectures.

ACKNOWLEDGMENT The authors disclose the use of Claude Code to assist in the implementation of the experiments. They take full responsibility for the experimental evaluation and the content of the paper. R EFERENCES [1] D. de Roux, B. Perez, A. Moreno, M. d. P. Villamil, and C. Figueroa, “Tax Fraud Detection for Under-Reporting Declarations Using an Unsupervised Machine Learning Approach,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’18. New York, NY, USA: Association for Computing Machinery, Jul. 2018, pp. 215–222. [2] H. Mittal, A. C. Pandey, M. Saraswat, S. Kumar, R. Pal, and G. Modwel, “A comprehensive survey of image segmentation: clustering methods, performance parameters, and benchmark datasets,” Multimedia Tools and Applications, vol. 81, no. 24, pp. 35 001–35 026, Oct. 2022. [3] Y. Li, M. Liu, W. Wang, Y. Zhang, and Q. He, “Acoustic Scene Clustering Using Joint Optimization of Deep Embedding Learning and Clustering Iteration,” IEEE Transactions on Multimedia, vol. 22, no. 6, pp. 1385–1394, Jun. 2020. [4] B. Peng, J. Lei, H. Fu, C. Zhang, T.-S. Chua, and X. Li, “Unsupervised Video Action Clustering via Motion-Scene Interaction Constraint,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 1, pp. 131–144, Jan. 2020. [5] G. J. Oyewole and G. A. Thopil, “Data clustering: application and trends,” Artificial Intelligence Review, vol. 56, no. 7, pp. 6439–6475, Jul. 2023. [6] R. Winkler, F. Klawonn, and R. Kruse, “Fuzzy C-Means in High Dimensional Spaces:,” International Journal of Fuzzy System Applications, vol. 1, no. 1, pp. 1–16, Jan. 2011. [7] M. Steinbach, L. Ertöz, and V. Kumar, “The Challenges of Clustering High Dimensional Data,” in New Directions in Statistical Physics: Econophysics, Bioinformatics, and Pattern Recognition, L. T. Wille, Ed. Berlin, Heidelberg: Springer, 2004, pp. 273–309. [8] X. Wei, Z. Zhang, H. Huang, and Y. Zhou, “An overview on deep clustering,” Neurocomputing, vol. 590, p. 127761, Jul. 2024. [9] S. Zhou, H. Xu, Z. Zheng, J. Chen, Z. Li, J. Bu et al., “A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions,” ACM Computing Surveys, vol. 57, no. 3, pp. 1–38, Mar. 2025. [10] B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong, “Towards Kmeans-friendly Spaces: Simultaneous Deep Learning and Clustering,” in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, Aug. 2017, pp. 3861–3870. [11] J. Xie, R. Girshick, and A. Farhadi, “Unsupervised Deep Embedding for Clustering Analysis,” in Proceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, Jun. 2016, pp. 478–487. [12] J. T. Springenberg, “Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2016. [13] A. Mashhadi, J. Sterner, and J. Murray, “Deep Embedded Clustering of Urban Communities Using Federated Learning,” in 2021 International Joint Conference on Neural Networks (IJCNN), Jul. 2021, pp. 1–8. [14] X. Xu, H. Liao, and X. Yang, “Federated deep embedded clustering under privacy protection,” Applied Soft Computing, vol. 185, p. 113963, Dec. 2025. [15] N. Hasan, M. G. R. Alam, S. H. Ripon, P. H. Pham, and M. M. Hassan, “An autoencoder-based confederated clustering leveraging a robust model fusion strategy for federated unsupervised learning,” Information Fusion, vol. 115, p. 102751, Mar. 2025. [16] R. Miao and E. Koyuncu, “Contrastive and Non-Contrastive Strategies for Federated Self-Supervised Representation Learning and Deep Clustering,” IEEE Journal of Selected Topics in Signal Processing, vol. 18, no. 6, pp. 1070–1084, Sep. 2024.

[17] ——, “Federated Momentum Contrastive Clustering,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 4, pp. 1–19, Aug. 2024. [18] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji et al., “Advances and Open Problems in Federated Learning,” arXiv:1912.04977 [cs, stat], Mar. 2021, arXiv: 1912.04977. [19] H. Zhu, J. Xu, S. Liu, and Y. Jin, “Federated learning on non-IID data: A survey,” Neurocomputing, vol. 465, pp. 371–390, Nov. 2021. [20] T. Z. Sana, S. Abdulla, A. Nag, A. Das, M. M. Hassan, Z. Z. Fiza et al., “Advancing Federated Learning: A Systematic Literature Review of Methods, Challenges, and Applications,” IEEE Access, vol. 13, pp. 153 817–153 844, 2025. [21] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated Learning with Non-IID Data,” 2018, arXiv:1806.00582 [cs, stat]. [22] H. Chen and H. Vikalo, “Federated Learning in Non-IID Settings Aided by Differentially Private Synthetic Data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Jun. 2023, pp. 5027–5036. [23] Y. Cheng, L. Zhang, and A. Li, “GFL: Federated Learning on Non-IID Data via Privacy-Preserving Synthetic Data,” in 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom). Atlanta, GA, USA: IEEE, Mar. 2023, pp. 61–70. [24] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated Optimization in Heterogeneous Networks,” in Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze, Eds., vol. 2, 2020, pp. 429–450. [25] A. Ghosh, J. Chung, D. Yin, and K. Ramchandran, “An efficient framework for clustered federated learning,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020. [26] D. K. Dennis, T. Li, and V. Smith, “Heterogeneity for the Win: OneShot Federated Clustering,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, no. 139. PMLR, Feb. 2021, pp. 2611–2620. [27] M. Stallmann and A. Wilbik, “On a Framework for Federated Cluster Analysis,” Applied Sciences, vol. 12, no. 20, p. 10455, Jan. 2022. [28] B. van Berlo, A. Saeed, and T. Ozcelebi, “Towards federated unsupervised representation learning,” in Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking, ser. EdgeSys ’20. New York, NY, USA: Association for Computing Machinery, Apr. 2020, pp. 31–36. [29] F. Zhang, K. Kuang, L. Chen, Z. You, T. Shen, J. Xiao et al., “Federated unsupervised representation learning,” Frontiers of Information Technology & Electronic Engineering, vol. 24, no. 8, pp. 1181–1193, Aug. 2023. [30] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” 2014, version Number: 9. [31] P. Nazari, S. Damrich, and F. A. Hamprecht, “Geometric autoencoders: what you see is what you decode,” in Proceedings of the 40th International Conference on Machine Learning, ser. ICML’23. JMLR.org, 2023. [32] A. F. Duque, S. Morin, G. Wolf, and K. R. Moon, “Geometry Regularized Autoencoders,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7381–7394, Jun. 2023. [33] L. McInnes, J. Healy, N. Saul, and L. Großberger, “UMAP: Uniform Manifold Approximation and Projection,” Journal of Open Source Software, vol. 3, no. 29, p. 861, Sep. 2018. [34] Q. Li, Y. Diao, Q. Chen, and B. He, “Federated Learning on Non-IID Data Silos: An Experimental Study,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE). Kuala Lumpur, Malaysia: IEEE, May 2022, pp. 965–978. [35] K. P. Murphy, Probabilistic Machine Learning: An introduction. MIT Press, 2022. [36] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Anchorage AK USA: ACM, Jul. 2019, pp. 2623–2631. [37] S. Harris and R. C. De Amorim, “An Extensive Empirical Comparison of k -means Initialization Algorithms,” IEEE Access, vol. 10, pp. 58 752– 58 768, 2022.