arXiv:2604.28024v1 [cs.LG] 30 Apr 2026
FedHarmony: Harmonizing Heterogeneous Label Correlations in Federated Multi-Label Learning Zhiqiang Kou1,2 * Junxiang Wu1,2 * Wenke Huang3 Wenwen He3 Ming-Kun Xie4 Changwei Wang5 Yuheng Jia1,2† Di Jiang6 Yang Liu6 Xin Geng1,2 † Qiang Yang6 1 School of Computer Science and Engineering, Southeast University, Nanjing, China 2 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China 3 Wuhan University, China 4 RIKEN Center for Advanced Intelligence Project, Japan 5 Qilu University of Technology (Shandong Academy of Sciences), China 6 Academy for Artificial Intelligence, Hong Kong Polytechnic University, Hong Kong, China [email protected]
Abstract Federated Multi-Label Learning is a distributed paradigm where multiple clients possess heterogeneous multi-label data and perform collaborative learning under privacy constraints without sharing raw data. However, modeling label correlations under heterogeneous distributions remains challenging. Due to client-specific label spaces and varying co-occurrence patterns, correlations learned by individual clients inevitably deviate from the global structure, a phenomenon we term label correlation drift. To address this, we propose FedHarmony, a framework that harmonizes heterogeneous label correlations across clients. It introduces consensus correlation, capturing agreement among other clients and serving as a global teacher to correct biased local estimates. During aggregation, FedHarmony evaluates each client by both data size and correlation quality, assigning weights accordingly. Moreover, we develop an accelerated optimization algorithm for FedHarmony and theoretically establish faster convergence without sacrificing accuracy. Experiments on real-world federated multi-label datasets show that FedHarmony consistently outperforms state-of-the-art methods.
1. Introduction Multi-Label Learning (MLL) [49, 50] aims to predict multiple labels for each instance [10, 18, 19, 21, 42]. For example, an image may contain building, street, and person, and * Equal contribution. † Corresponding Author.
the goal is to identify all relevant labels for unseen samples. MLL has shown strong performance in various applications, including visual recognition [38], sentiment analysis [48], and medical image understanding [40]. A central challenge in MLL is modeling label correlations, such as common cooccurrence patterns [41, 51]. Recent methods use Graph Convolutional Networks [3, 51] and Transformer-based architectures [44, 46] to explicitly encode these label relationships, which significantly improves prediction performance. Driven by growing privacy and security concerns [1, 13, 14, 30], multi-label learning is increasingly conducted under the Federated Learning framework [29, 32, 33], as illustrated in Fig. 1(a), where each client holds its own multi-label dataset and Federated Multi-Label Learning (FedMLL) [8, 9, 22] refers to a distributed paradigm in which multiple clients with multi-label data perform collaborative learning under privacy constraints without sharing raw data. As discussed above,its key objective is to align and integrate the label correlations learned from distributed clients , enabling the central server to recover the overall label dependency structure from decentralized data [17, 20, 34, 37]. However, achieving this goal is non-trivial under heterogeneous data distributions [23, 47]. As shown in Fig. 1(a), we visualize the label co-occurrence matrices on the FLAIR dataset1 [36], showing that each client exhibits distinct cooccurrence frequencies. For example, the co-occurrence between the labels outdoor and equipment is very high in Client 1, while it is much lower in Client 2. Moreover, existing methods [24, 27, 28] weight clients by their training data size and average their updates accordingly, ignoring 1 This is a multi-labeled federated learning dataset.
Figure 1. Label co-occurrence patterns on the FLAIR dataset [36]. Different clients exhibit distinct label co-occurrence frequencies, which deviate from the global pattern representing the true data distribution.
the quality of the learned label correlations. As shown in Fig. 1(c), a client with a large amount of data may still learn poor correlation structures, yet it receives a disproportionately high aggregation weight, raising the question of whether this is desirable. To address these challenges, we propose FedHarmony, a federated multi-label learning framework that harmonizes heterogeneous label correlations across clients. In FedMLL, each client observes only a subset of the full label space, which inevitably leads to biased local correlation estimates [6]. We argue that no single client can fully capture the true label relationships; however, the correlations that are consistent across most clients are more likely to reflect the underlying global semantics. Based on this insight, we introduce the notion of consensus correlation, defined for each client as the collective agreement of all other clients. During local training, this consensus correlation serves as a global teacher: it provides structural guidance that continuously corrects the client’s biased local view, enabling its learned label relationships to gradually align with the global consensus. In the aggregation stage, FedHarmony further considers the quality of each client’s learned label structure. To improve training efficiency, we also develop an accelerated optimization algorithm that speeds up convergence without compromising predictive performance. Our main contributions are summarized as follows • We provide the first systematic study of label correlation drift in FedMLL, and we propose FedHarmony to explicitly address this issue through consensus-guided correlation modeling. • We develop an accelerated optimization algorithm for FedHarmony and theoretically show that it achieves faster convergence while preserving model accuracy. • Extensive experiments on multiple federated multi-label benchmarks demonstrate that FedHarmony consistently outperforms existing state-of-the-art methods.
2. FedHarmony Notation. We consider a federated multi-label learning system with K clients, indexed by k. Each client k owns a k private dataset Dk = {xi , yi }N i=1 , where Nk denotes the private data number for k-th client. The input space is X ⊂ Rd and the label space is Y = {0, 1}C , where C is the number of labels. For an instance x ∈ X , its label vector is y = (y1 , . . . , yC )⊤ ∈ {0, 1}C , where yc = 1 indicates that label c is relevant to x. We denote the global model parameter at the beginning of the t-th communication round as wt . The server then broadcasts wt to each client, initializing the local model as wkt ← wt . Each client performs local optimization on its private dataset and updates the model via stochastic gradient descent: wkt ← P (k) wkt − η i∈Bk ℓ wkt , ξi , where Bk denotes a mini-batch sampled from the local dataset Dk , ξ is the query instance, and η is the local learning rate. After local updates, each client uploads its parameter wkt to the server, which performs weighted PK aggregation to obtain the next global model: wt+1 = k=1 αk wkt , where αk is the aggregation weight for client k. Following [43], we use the binary cross-entropy loss for multi-label classification. Overview of FedHarmony. In federated multi-label learning, client-specific label spaces and heterogeneous co-occurrence patterns cause locally estimated label correlations to become inconsistent, leading to label correlation drift. FedHarmony aims to recover a consensus label structure that captures shared global semantics while correcting biased local correlations through consensusguided updates (Sec 2.1 and Sec 2.2). Moreover , clients differ in how well they learn these structures; thus, aggregation cannot rely solely on data size and must also account for learning quality to prevent unreliable correlations from dominating the global model (Sec 2.3).
∗(t)
2.1. Consensus Label Correlation Teacher In federated multi-label learning [8, 9, 22], data heterogeneity causes the locally learned label correlations to become biased [6]. We argue that no single client knows the true label relationships, and that correlations shared by the majority are more likely to reflect the truth [25]. Thus, we introduce a consensus correlation, representing the collective agreement of all clients except the target one. During client-side training, we use the consensus label correlation as a teacher to prevent the locally learned correlations from drifting away from the global structure. Next, we define the consensus label correlation: At communication round t, client k obtains prek diction scores for its local dataset Dk = {xi }N i=1 (t) (t) using its current model fk (·; θk ): Fk = (t) (t) ⊤ Nk ×C fk (x1 ; θk ), . . . , fk (xNk ; θk ) ∈ [0, 1] . These scores serve as soft estimates of label occurrence. From them, we compute the empirical marginal and joint probabilities: N
N
k 1 X (t) (t) F F ′, Nk i=1 k,ic k,ic (1) (t) where p̂k,c estimates the marginal probability that label c
(t)
p̂k,c =
k 1 X (t) F , Nk i=1 k,ic
(t)
p̂k,cc′ =
(t)
is predicted to appear on client k, and p̂k,cc′ estimates the joint probability that labels c and c′ are predicted to appear simultaneously. To capture their dependency strength, we compute a phi-style correlation coefficient: (t)
(t) (t)
p̂k,cc′ − p̂k,c p̂k,c′ (t) Rk,cc′ = q , (t) (t) (t) (t) p̂k,c (1 − p̂k,c ) p̂k,c′ (1 − p̂k,c′ ) + ε
(2)
where ε > 0 is a small constant for numerical stability. For the k-th client, the local label correlation matrix is defined (t) C (t) as Rk = Rk,cc′ c,c′ =1 ∈ RC×C . For client k at round t, the consensus label correlation is defined as the leave-one-out population consensus constructed from the label–label correlation statistics of all other clients: (t)
{Rj }j̸=k | {z }
A
∗(t)
t − −− → Rexp,k ∈ C.
(3)
all clients except k
local label–label correlation and Rexp,i the expert consensus (Sec. §2.1). The correction is imposed via a correlationalignment loss (t) ∗(t) Lalign = λ Ψ Ri , Rexp,i , i,t
(4)
where Ψ(·, ·) is a fixed distance/divergence in correlation space (kept consistent throughout).
Figure 2. Label correlation matrix on the VOC and FLAIR datasets. Each matrix visualizes the empirical co-occurrence probability between every pair of labels. Labels listed on the horizontal and vertical axes correspond to the semantic categories in each dataset, and brighter colors indicate higher co-occurrence probability.
However, not all labels are mutually correlated: each label interacts with only a small subset of others. As shown in Fig. 2, the label correlation matrix is sparse and approximately block-structured [2]. If we partition it into g smaller (approximately) rank-r submatrices and align correlations only within clusters [31], we can both improve optimization efficiency and incur negligible information loss. We now provide theory from these two perspectives and state two theorems with their proofs. Let R ∈ SC be a client correlation and R∗exp the (roundwise) expert consensus. Define the weighted Frobenius loss f (R) = ∥Γ◦(R − R∗exp )∥2F with weights Γ = (γcc′ ) ≥ 0. C Let {Sg }G : g=1 be a label partition, and let U = {X ∈ S 2 supp(X) ⊆ ∪g Sg } denote the block-diagonal subspace, with orthogonal projector PU . Define γcc′ > 0, γin := (c,c′min )∈∪g Sg2 (5) ′ ∈ [0, ρ γin ), ρ < 1, max γ cc γout := ′ 2 (c,c )∈∪ / g Sg
2.2. Consensus-Guided Correction
here, γin and γout denote the in-cluster and cross-cluster correlation weights, respectively. Decompose the consensus as ( B := PU R∗exp , (6) E := R∗exp − B,
We correct the local correlation by aligning it to the round(t) wise expert consensus. For client i at round t, let Ri be its
here, B denotes the in-cluster part of R∗exp , while E contains all cross-cluster correlations.
(t)
Here, {Rj }j̸=k are the uploaded correlation matrices from all clients except k, and At (·) denotes the consolidation operator that aggregates them into a consensus structure.
Theorem 2.1. Consider gradient descent with stepsize 2 η ≤ 1/(2 maxc,c′ γcc ′ ) on (i) the full objective ffull (R) = ∗ 2 ∥Γ ◦ (R − Rexp )∥F and (ii) the block-restricted objective fblk (R) = ∥Γ◦(PU R − B)∥2F . From the same initialization R(0) , after T steps, ( (T ) 2 T fblk Rblk ≤ (1 − 2η γin ) fblk R(0) , (7) (T ) 2 ffull Rfull ≤ (1 − 2η γout )T ffull R(0) . Since γin ≫ γout , block-wise alignment enjoys a strictly faster linear convergence rate. Proof. Vectorize under the Frobenius inner product. Both objectives are quadratic with diagonal Hessians: Hfull = 2 2 Thus 2 diag(γcc ′ ) and Hblk = 2 diag(γcc′ 1(c,c′ )∈∪g S 2 ). g 2 the strong convexity constants satisfy µfull = 2 min γcc′ ≤ 2 2 2 2γout and µblk = 2 minin γcc ′ ≥ 2γin . With η ≤ 1/L, standard results for strongly convex quadratics yield the linear factors (1 − ηµ)T . (T )
Theorem 2.2. Let Rblk be the iterate produced by minimizing fblk . Then (T ) (T ) ffull Rblk = fblk Rblk + ∥ Γout ◦ E ∥2F , {z } | | {z } in-block alignment error
(T )
By Theorem 2.1, restricting correction to in-cluster pairs increases curvature (from γout to γin ) and yields a faster linear rate. By Theorem 2.2, ignoring cross-cluster entries incurs at most ∥Γout ◦ E∥2F additional loss, i.e., negligible when the consensus is near block-diagonal or cross-cluster weights are small. Motivated by Theorems 2.1 and 2.2, we correct only the high-correlation subset of label pairs in the full label space. Concretely, we partition labels using the ∗(t) 2 expert correlation Rexp,i to obtain g clusters {Sg }G g=1 , so that strongly correlated labels fall into the same group and alignment focuses on dense, high-signal subspaces. Given {Sg }G g=1 , Eq. (4) is replaced by the within-cluster objective G X (t) ∗(t) Ψ Ri [Sg , Sg ], Rexp,i [Sg , Sg ] .
2 See Algorithm 1 for clustering details.
2.3. Correlation-Aware Aggregation We aggregate client updates by jointly considering (i) data quantity and (ii) the learning quality of each client’s labelstructure modeling. Early rounds emphasize quantity since local correlations are unreliable; later rounds increasingly favor structural quality. For client i at round t, define its block-wise structural discrepancy (t)
si
Proof. Since Rblk ∈ U and B = PU R∗ , the supports of (T ) Γ◦(Rblk −B) (in-block) and Γ◦E (cross-block) are disjoint and orthogonal under the Frobenius inner product. Hence (T ) (T ) ∥Γ◦ (Rblk − R∗ )∥2F = ∥Γ◦ (Rblk − B)∥2F + ∥Γ◦ E∥2F , and the second term equals ∥Γout ◦ E∥2F .
g=1
∗(t)
Require: Expert correlation Rexp ∈ RC×C , number of clusters G ∗(t) ∗(t) 1: Affinity: S ← 12 |Rexp |+|Rexp |⊤ ; set diag(S) ← 0 2: Laplacian: D ← diag(S1), L ← D−1/2 (D − −1/2 S)D 3: Embedding: take U ∈ RC×G as eigenvectors of L for the G smallest eigenvalues 4: Row normalization: Uc,: ← Uc,: /∥Uc,: ∥2 for all c 5: Clustering: run k-means on {Uc,: }C c=1 to obtain clusters {Sg }G g=1 6: return {Sg }G g=1
upper bound
where Γout keeps only cross-block weights. In particular, if E = 0 or γout is small, the second term is at most 2 O(γout ∥E∥2F ) and negligible.
Lalign =λ i,t
Algorithm 1 Spectral clustering on expert correlation
=
G X (t) ∗(t) Ψ Ri [Sg , Sg ], Rexp,i [Sg , Sg ] ,
(9)
g=1
and map it to a quality score (larger is better) via a monotone (t) (t) decreasing transform, e.g. qi = exp − γ si , γ> (t) 0. Normalize counts and qualities: n̄i = Pninj , q̄i = j
(t)
q P i (t) . j qj
Let α(t) ∈ [0, 1] decrease over rounds to shift the
aggregation from quantity-driven to structure-driven, e.g. α(t) = max 0, 1 − t/T0 , with a user-chosen transition horizon T0 . The per-round aggregation weight is X (t) (t) (t) wi = α(t) n̄i + 1−α(t) q̄i , wi = 1. (10) i
The server aggregates model parameters as θ (t+1) = P (t) (t+1) . When t is small, α(t) ≈ 1 and the rule rei wi θ i duces to quantity-weighted averaging (FedAvg-like), which stabilizes early training. As t grows, α(t) ↓ 0 and clients (t) with better correlation alignment (larger qi ) dominate aggregation.
3. Experiment 3.1. Experimental Setup
(8)
Datasets. Following Liu et al. [27], we evaluate on three multi-label datasets: FLAIR [36], MS-COCO [26], and PASCAL VOC [7]. FLAIR contains 429,078 images from 51,414 Flickr users with a two-level label space (17 coarse and
Table 1. Overall comparisons on three multi-label benchmarks under non-IID settings. Rows for F ED H ARMONY are lightly shaded; bold numbers denote the best within each dataset block.
Method mAP (%) O mAP (%) CP (%) CR (%) CF1 (%) OP (%) OR (%) OF1 (%) FedAvg 35.4 70.6 34.1 27.9 30.7 70.2 56.0 62.3 FedCurv 35.4 71.6 34.9 25.5 29.5 72.7 53.2 61.4 FedProx 39.6 75.7 40.2 30.0 34.3 75.7 58.2 65.8 FedNova 30.3 55.6 12.3 11.0 11.6 56.5 35.9 43.9 FLAIR FedLGT 36.9 72.8 39.2 25.0 30.5 75.5 51.4 61.1 SphereFed 35.2 71.9 35.1 22.5 27.5 77.2 48.3 59.4 FedRDN 35.5 71.4 35.0 26.3 30.0 72.3 54.1 61.9 F ED H ARMONY 51.0 84.0 49.3 43.6 46.3 79.0 71.5 75.1 FedAvg 63.4 73.9 63.2 54.9 58.7 69.3 64.4 66.7 FedCurv 63.3 73.3 64.0 53.7 58.4 68.6 64.4 66.4 FedProx 64.0 74.1 62.7 56.2 59.3 68.5 65.8 67.1 FedNova 4.3 4.9 2.5 49.0 4.8 4.2 57.8 7.9 COCO-80 FedLGT 64.5 74.4 61.4 57.5 59.4 67.5 66.8 67.2 SphereFed 63.0 72.7 61.8 55.2 58.3 65.9 65.3 65.6 FedRDN 63.4 73.6 64.0 54.5 58.9 69.1 64.4 66.6 F ED H ARMONY 71.4 79.9 73.5 58.6 65.2 78.6 67.6 72.7 FedAvg 75.7 69.2 81.8 54.5 65.4 61.6 63.9 62.7 FedCurv 72.8 56.9 75.5 50.7 60.7 62.4 57.5 59.9 FedProx 75.5 68.9 82.6 53.2 64.7 67.1 61.4 64.1 FedNova 18.1 36.6 4.7 1.9 2.7 94.9 11.0 19.7 VOC2007 FedLGT 78.1 70.8 77.7 61.0 68.3 60.2 68.0 63.9 SphereFed 75.2 59.4 81.5 56.8 66.9 58.3 65.8 61.8 FedRDN 78.3 72.2 79.9 57.2 66.6 70.6 65.0 67.7 F ED H ARMONY 86.9 89.1 88.5 71.2 78.9 88.5 78.8 83.4
Figure 3. Qualitative comparison. For each image, we show the predictions of FedAvg ( ) and Ours ( ), alongside the ground-truth labels ( ). Our method suppresses spurious labels (e.g., ’bird’ on COCO horse) and recovers missing semantics (e.g., equipment, material, structure), demonstrating better correlation-aware recognition across scenes.
1,628 fine labels), naturally exhibiting quantity and labeldistribution skew. MS-COCO provides approximately 330K images annotated with 80 object categories in the stan-
dard multi-label formulation. PASCAL VOC offers 20 categories; we report results on VOC2007 (9,963 images) and VOC2012 train/val (11,540 images). To convert MS-COCO
and PASCAL VOC into heterogeneous federated datasets, we induce non-IIDness along three axes. The full protocol (Dirichlet concentration, client counts, transform sets, and seeds) is detailed in the Appendix. Counterparts. We compare FedHarmony with representative federated learning methods from three major paradigms: (i) optimization-driven aggregation methods, including FedAvg [28], FedProx [24], and FedNova [39]; (ii) curvature- or geometry-aware optimization methods, such as FedCurv [35] and SphereFed [4]; and (iii) taskor feature-aware baselines, including FedLGT for federated multi-label classification [27] and FedRDN for mitigating distribution skew through feature-level augmentation [45]. All methods are implemented under identical experimental settings, sharing the same backbone, optimizer, and communication budget to ensure a fair comparison. Implementation Details. Unless otherwise stated, we follow the settings of Huang et al. [11, 12], Liu et al. [27]. All methods use a ViT-B/16 backbone [5] with a Cway sigmoid classification head. Local training runs for 5 epochs per round using Adam [16] with a learning rate of 10−4 and batch size 16, and we set the total communication rounds to T = 50. To account for quantity skew (e.g., in FLAIR), we adopt non-uniform client sampling [24], where the probability of selecting client i is proportional to its local data size. All methods share identical backbones, optimizers, schedules, and client partitions to ensure fairness. Experiments are implemented in PyTorch and trained on eight NVIDIA RTX 4090 GPUs. Additional details are provided in the Appendix. Evaluation Metrics.We evaluate multi-label prediction performance using eight widely adopted metrics [51]. Specifically, we report mean Average Precision (mAP, ↑), overall mean Average Precision (O-mAP, ↑), class-wise precision (CP, ↑), class-wise recall (CR, ↑), class-wise F1 score (CF1, ↑), overall precision (OP, ↑), overall recall (OR, ↑), and overall F1 score (OF1, ↑). Higher values indicate better performance for all metrics.
3.2. Comprehensive Comparative Analysis Table 1 and Fig. 3 present both quantitative and qualitative comparisons under heterogeneous non-IID settings across COCO-80, VOC2007, and FLAIR. FedHarmony achieves the best performance on all datasets and metrics, with a notable gain of more than 10 mAP on FLAIR over the strongest baseline, and similarly strong margins on COCO80 and VOC2007. Qualitative results further show that FedAvg often yields spurious or incomplete predictions, whereas FedHarmony produces more consistent and semantically accurate label sets. From the results above, we obtain the following observations: • FedHarmony substantially outperforms FedAvg and
Table 2. Cumulative training time comparison on FLAIR and VOC2007 datasets across communication rounds. The proposed Block-Optimized (B-OPT) method notably accelerates convergence compared to the No-Block-Optimized (NB-OPT) variant. Round 1 2 3 4 5 6 7 8 9 10
B-OPT 03:59 07:55 12:05 16:09 20:14 24:17 28:18 32:17 36:21 40:22
Flair NB-OPT 05:14 10:15 15:43 21:06 27:14 33:31 39:53 46:26 51:58 56:19
VOC2007 B-OPT NB-OPT 01:32 02:29 02:36 04:03 03:42 05:39 04:49 07:17 05:55 08:56 07:05 10:34 08:14 12:08 09:29 13:51 10:32 15:29 11:43 17:09
FedProx, indicating that aligning cross-client correlation structures is more effective than parameter averaging for multi-label representation learning. • FedCurv and SphereFed show marginal improvements, suggesting that curvature correction is insufficient to address the semantic inconsistencies introduced by heterogeneous label dependencies. • Compared with FedLGT and FedRDN, FedHarmony remains stable under severe non-IID conditions, demonstrating that correlation-structure alignment provides a more robust and generalizable inductive bias. Overall, these results verify that harmonizing clientspecific label correlations leads to globally coherent and locally adaptive representations, yielding consistent performance gains across federated multi-label benchmarks. Table 3. Comparison of No-Block vs. Block-Optimized across three benchmarks. All metrics are reported in percentage (%).
Method mAP ↑ O mAP ↑ CP ↑ No-Block 71.2 79.8 73.8 COCO-80 Block-Optimized 72.4 80.8 71.9 No-Block 86.1 88.9 87.0 VOC2007 Block-Optimized 87.0 89.1 87.4 No-Block 47.0 81.9 45.6 FLAIR Block-Optimized 48.0 83.8 45.9
3.3. Impact of Block-wise Optimization We evaluate the effect of block-wise optimization by comparing the No-Block and Block-Optimized variants on COCO-80, VOC2007, and FLAIR (Table 3). The performance differences across all eight metrics remain minimal, typically within 0.3%–0.5%. To assess statistical significance, we conduct a Wilcoxon signed-rank test using
Figure 4. Qualitative comparison of learned label correlation structures on the FLAIR dataset.
Table 4. Wilcoxon signed-rank test results between No-Block and Block-Optimized on three benchmarks. All p-values > 0.05, indicating no significant differences.
Dataset
W
p-value
Result
COCO-80
7.0
0.382
tie (not significant)
VOC2007
0.0
0.148
tie (not significant)
FLAIR
2.0
0.547
tie (not significant)
paired metric values for each dataset. As shown in Table 4, all p-values exceed 0.05 (COCO-80: 0.382; VOC2007: 0.148; FLAIR: 0.547), indicating no significant difference between the two configurations. We further compare training efficiency (Table 2). The Block-Optimized variant substantially reduces cumulative training time, achieving a 28.3% reduction on FLAIR (56:19 → 40:22) and 31.7% on VOC2007 (17:09 → 11:43) by Round 10. Overall, block-wise optimization introduces no statistically significant performance degradation while providing clear efficiency gains, confirming that the proposed clustering strategy preserves accuracy and improves computational efficiency.
3.4. The remaining experiments Qualitative Analysis of Correlation Structure. We compare the label correlation matrices learned by FedHarmony and competing methods against the ground-truth (GT) matrix. As shown in Fig. 4, FedHarmony reconstructs a structure that closely matches the GT and accurately recovers subtle co-occurrence patterns, such as the moderate equipment–material relation (0.43 vs. GT 0.42). In contrast, methods such as FedRdn and FedNova produce sparse matrices that collapse most correlations toward zero, failing to capture the underlying semantic dependencies. These qualitative results support our central claim. Conventional aggregation strategies lose structural information and suffer from uncertain correlation aggregation, whereas FedHarmony,
Table 5. Ablation on COCO-80 and FLAIR benchmarks. Base = FedAvg. A = ECL; A+B = ECL + CAA. (Legend text slightly shortened for space). All metrics are reported in percentage (%). COCO-80
Metric mAP ↑ O mAP ↑ CP ↑ CR ↑ CF1 ↑ OP ↑ OR ↑ OF1 ↑
FLAIR
Base
+A
+A+B
Base
+A
+A+B
63.4 73.9 63.2 54.9 58.7 69.3 64.4 66.7
69.5 78.5 73.0 54.4 62.4 79.2 64.5 71.1
71.2 79.8 73.8 58.4 65.2 78.5 67.7 72.7
35.4 70.6 34.1 27.9 30.7 70.2 56.0 62.3
46.4 79.6 44.2 40.9 42.5 72.8 73.3 73.0
47.0 81.9 45.6 39.3 42.2 74.8 73.9 74.3
through its Correlation Expert and structure-driven aggregation, harmonizes heterogeneous client structures and recovers a globally faithful correlation matrix. Ablation Study. To verify the contribution of each component in our framework, we conduct ablations on three datasets. The primary results are summarized in Table 5 and the results for VOC2007 are deferred to the Appendix. Base = FedAvg; A = expert-guided correlation loss; B = correlation-aware aggregation; C = block-wise clustering. One factor is added at a time (Base → A → A+B → A+B+C); all other settings are fixed. From the Table 5, we can get the below conclusion • A vs. Base. Primary improvements on all datasets for mAP and O mAP; recall increases where non-IID is strongest. • A+B vs. A. Consistent gains in OR/OF1 across datasets; a minor precision drop (OP) on COCO. • A+B+C vs. A+B. Further increases in CR/OF1; effect largest on the sparsest dataset (FLAIR); slight CP/OP decrease on COCO. • Overall. A contributes most of the total mAP gain; B adds recall-oriented improvements via quality-weighted aggregation; C improves alignment conditioning. The full configuration (A+B+C) is best across metrics.
Figure 5. Performance analysis based on client configurations: (a, b) impact of the number of participating clients, and (c, d) impact of the total number of clients.
VOC2007, and FLAIR. As shown in Fig. 5(a,b), performance remains highly stable across different ratios, with variations typically within 1–2%. This indicates that the framework maintains consistent accuracy even when only a subset of clients joins each communication round. These results highlight the robustness of our correlation-aware aggregation and block-wise optimization, which effectively reduce performance fluctuations caused by client sampling and allow the global model to generalize reliably under heterogeneous and partially available client settings.
Figure 6. Robustness of FedHarmony under different label skew levels controlled by γ.
Robustness to Structural Heterogeneity. We assess robustness to label imbalance by varying the Dirichlet concentration parameter γ, which controls the FL-MLL sampling process. Smaller γ produces stronger non-IID partitions, where each client observes only a small subset of labels (details in Appendix A). Fig. 6 shows that as γ decreases from 1.0 to 0.25, FedHarmony maintains almost unchanged mAP, OF1, CP, and CR on COCO-80 and VOC2007, while all baselines degrade rapidly. This stability stems from aligning client-specific correlation structures toward a shared consensus rather than relying on raw sample frequencies, preventing over-represented labels from dominating aggregation. Overall, the flat curves across different γ values confirm that FedHarmony generalizes reliably under severe label skew and remains structurally robust across heterogeneous clients. Influence of the Number of Participating Clients. We evaluate the impact of client participation by testing three participation ratios (25%, 50%, 75%) on COCO-80,
Effect of the Number of Clients. We evaluate scalability by varying the number of participating clients from 1000 to 2500 on the COCO and VOC benchmarks while keeping the total training data fixed (Fig. 5(c,d)). Across both datasets, FedHarmony exhibits highly stable performance, with Overall Recall and Overall F1 fluctuating by only 1–2 points as the federation size increases. This indicates that the method consistently harmonizes label correlations even under increasingly fragmented client distributions. The slight performance drop at extremely large scales (e.g., 2500 clients on VOC) stems from heightened local data sparsity and stronger correlation heterogeneity [15]. Overall, the results demonstrate strong robustness to client scaling and confirm FedHarmony’s ability to maintain semantic consistency in large-scale federated multi-label learning.
4. Conclusion We propose FedHarmony, which explicitly addresses local correlation drift and enforces cross-client consistency through expert-guided correlation regularization and correlation-aware aggregation. To further enhance optimization stability, we develop a lightweight block-wise acceleration scheme that improves convergence efficiency while preserving accuracy and training stability. Extensive experiments with statistical significance validation on COCO-80, VOC2007, and FLAIR demonstrate that our method consistently achieves superior and robust performance under heterogeneous data distributions settings.
Acknowledgement This research was supported by the Jiangsu Science Foundation (BG2024036, BK20243012), the National Science Foundation of China (62125602, U24A20324, 92464301), the New Cornerstone Science Foundation through the XPLORER PRIZE, and the Fundamental Research Funds for the Central Universities (2242025K30024 supported by the National). This work was Natural Science Foundation of China under Grant U24A20322 and Grant 62576094. This research work is also supported by the Big Data Computing Center of Southeast University.
References [1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016. 1 [2] Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain. Sparse local embeddings for extreme multi-label classification. Advances in neural information processing systems, 28, 2015. 3 [3] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177– 5186, 2019. 1 [4] Xin Dong, Sai Qian Zhang, Ang Li, and HT Kung. Spherefed: Hyperspherical federated learning. In European Conference on Computer Vision, pages 165–184. Springer, 2022. 6 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 6 [6] Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, and Liang Liang. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 32(1):59–71, 2020. 2, 3 [7] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98–136, 2015. 4 [8] Bita Ghasemkhani, Ozlem Varliklar, Yunus Dogan, Semih Utku, Kokten Ulas Birant, and Derya Birant. Federated multi-label learning (fmll): Innovative method for classification tasks in animal science. Animals, 14(14):2021, 2024. 1, 3 [9] Kun Guo, Dangrun Chen, Qingqing Huang, Fuan Li, Chen Guo, Duanji Wu, Ximeng Liu, and Kai Chen. Privacypreserving multi-label propagation based on federated learning. IEEE Transactions on Network Science and Engineering, 11(1):886–899, 2023. 1, 3
[10] Sheng-Jun Huang and Zhi-Hua Zhou. Multi-label learning by exploiting label correlations locally. In Proceedings of the AAAI conference on artificial intelligence, pages 949– 955, 2012. 1 [11] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10133–10143, 2022. 6 [12] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16312–16322, 2023. 6 [13] Wenke Huang, Mang Ye, Zekun Shi, and Bo Du. Generalizable heterogeneous federated cross-correlation and instance similarity learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):712–728, 2024. 1 [14] Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairness: A survey and benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9387–9406, 2024. 1 [15] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020. 8 [16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. 6 [17] Zhiqiang Kou, Jing Wang, Yuheng Jia, and Xin Geng. Inaccurate label distribution learning. IEEE Transactions on Circuits and Systems for Video Technology, 34(10):10237– 10249, 2024. 1 [18] Zhiqiang Kou, Jing Wang, Jiawei Tang, Yuheng Jia, Boyu Shi, and Xin Geng. Exploiting multi-label correlation in label distribution learning. In Proceedings of the ThirtyThird International Joint Conference on Artificial Intelligence, IJCAI-24, pages 4326–4334, 2024. 1 [19] Zhiqiang Kou, Si Qin, Hailin Wang, Jing Wang, Mingkun Xie, Shuo Chen, Yuheng Jia, Tongliang Liu, Masashi Sugiyama, and Xin Geng. Label distribution learning with biased annotations assisted by multi-label learning. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025. 1 [20] Zhiqiang Kou, Jing Wang, Yuheng Jia, Biao Liu, and Xin Geng. Instance-dependent inaccurate label distribution learning. IEEE Transactions on Neural Networks and Learning Systems, 36(1):1425–1437, 2025. 1 [21] Zhiqiang Kou, Yucheng Xie, Hailin Wang, Jing Wang, Mingkun Xie, Shuo Chen, Yuheng Jia, Tongliang Liu, and Xin Geng. Rankmatch: A novel approach to semi-supervised label distribution learning leveraging rank correlation between labels. In Proceedings of the 39th Conference on Neural Information Processing Systems, 2025. 1 [22] Khalid Odartey Lamptey, Browne Judith Ayekai, and Salah Ud Din. Federated learning on multi-label evolving data streams. IEEE Internet of Things Journal, 2025. 1, 3
[23] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020. 1 [24] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020. 1, 6 [25] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems, 33:2351–2363, 2020. 3 [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 4 [27] I-Jieh Liu, Ci-Siang Lin, Fu-En Yang, and Yu-Chiang Frank Wang. Language-guided transformer for federated multilabel classification. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13882–13890, 2024. 1, 4, 6 [28] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. Pmlr, 2017. 1, 6 [29] Hongdao Meng, Yongjian Deng, Qiyu Zhong, Yipeng Wang, Zhen Yang, and Gengyu Lyu. Federated multi-view multilabel classification. IEEE Transactions on Big Data, 11(4): 2072–2084, 2024. 1 [30] Lei Meng, Zhuang Qi, Lei Wu, Xiaoyu Du, Zhaochuan Li, Lizhen Cui, and Xiangxu Meng. Improving global generalization and local personalization for federated learning. IEEE Transactions on Neural Networks and Learning Systems, 36(1):76–87, 2025. 1 [31] Gulisong Nasierding, Grigorios Tsoumakas, and Abbas Z Kouzani. Clustering based multi-label classification for image annotation and retrieval. In 2009 IEEE international conference on systems, man and cybernetics, pages 4514– 4519. IEEE, 2009. 3 [32] Xin Qi, Tao Xu, Chengrun Dang, Zhuang Qi, Lei Meng, and Han Yu. Federated learning in oncology: bridging artificial intelligence innovation and privacy protection. Information Fusion, 130:104154, 2026. 1 [33] Zhuang Qi, Lei Meng, Zitan Chen, Han Hu, Hui Lin, and Xiangxu Meng. Cross-silo prototypical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM International Conference on Multimedia, page 3099–3107, 2023. 1 [34] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-label classification. Machine learning, 85(3):333–359, 2011. 1 [35] Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, and Itai Zeitak. Overcoming forgetting in federated learning on non-iid data. arXiv preprint arXiv:1910.07796, 2019. 6 [36] Congzheng Song, Filip Granqvist, and Kunal Talwar. Flair: Federated learning annotated image repository. Advances
in Neural Information Processing Systems, 35:37792–37805, 2022. 1, 2, 4 [37] Grigorios Tsoumakas and Ioannis Vlahavas. Random klabelsets: An ensemble method for multilabel classification. In European conference on machine learning, pages 406– 417. Springer, 2007. 1 [38] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2285–2294, 2016. 1 [39] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020. 6 [40] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestxray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017. 1 [41] Jia-Hao Xiao, Ming-Kun Xie, Heng-Bo Fan, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Dualdecoupling learning and metric-adaptive thresholding for semi-supervised multi-label learning. In European Conference on Computer Vision, pages 437–454. Springer, 2024. 1 [42] Ming-Kun Xie, Jiahao Xiao, and Sheng-Jun Huang. Labelaware global consistency for multi-label learning with single positive labels. Advances in Neural Information Processing Systems, 35:18430–18441, 2022. 1 [43] Ming-Kun Xie, Jiahao Xiao, Hao-Zhe Liu, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Classdistribution-aware pseudo-labeling for semi-supervised multi-label learning. Advances in Neural Information Processing Systems, 36:25731–25747, 2023. 2 [44] Ming-Kun Xie, Jia-Hao Xiao, Pei Peng, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Counterfactual reasoning for multi-label image classification via patching-based training. arXiv preprint arXiv:2404.06287, 2024. 1 [45] Yunlu Yan, Huazhu Fu, Yuexiang Li, Jinheng Xie, Jun Ma, Guang Yang, and Lei Zhu. A simple data augmentation for feature distribution skewed federated learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25749–25758, 2025. 6 [46] Penghui Yang, Ming-Kun Xie, Chen-Chen Zong, Lei Feng, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Multi-label knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 17271–17280, 2023. 1 [47] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019. 1 [48] Jianfei Yu, Luis Marujo, Jing Jiang, Pradeep Karuturi, and William Brendel. Improving multi-label emotion classifica-
tion via sentiment classification with dual attention transfer network. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 1097–1102, 2018. 1 [49] Min-Ling Zhang and Lei Wu. Lift: Multi-label learning with label-specific features. IEEE transactions on pattern analysis and machine intelligence, 37(1):107–120, 2014. 1 [50] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering, 26(8):1819–1837, 2013. 1 [51] Yue Zhu, James T Kwok, and Zhi-Hua Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering, 30(6):1081– 1094, 2017. 1, 6