CALM: CLASS-WISE AGREEMENT AND LABEL-GATED DISAGREEMENT MODULATION FOR DECENTRALIZED FEDERATED LEARNING Yifan Ying
Qing Tian∗
Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL, USA
arXiv:2609.05884v1 [cs.LG] 5 Sep 2026
ABSTRACT Conventional federated learning relies on parameter averaging, which forces clients to be doubly homogeneous: all must run an identical architecture, and accuracy degrades when local data are non-IID. Decentralized federated distillation sidesteps both: each client runs its peers’ model snapshots as teachers on its own local data and distills from their soft predictions, with no server, no public data, and no shared architecture. Under severe non-IID skew, however, the trustworthiness of the aggregated teacher target is a matter of degree, yet existing pipelines make hard, all-or-nothing decisions: outlier teachers are discarded by threshold, and whatever target survives is trusted in full. We propose CALM, which replaces every hard decision with a smooth trust gate at three levels: per class, teachers are weighted by agreement with the peer consensus; per sample, distillation is scaled by the teachers’ divergence from that target; and a label gate scales it by how strongly the target supports the sample’s true label. None of this adds communication or auxiliary data. On CIFAR-10, SVHN, OrganAMNIST, and Google Speech Commands with heterogeneous client architectures under Dirichlet label skew, CALM consistently outperforms uniform and hard-filtered distillation and matches or exceeds competing heterogeneous-FL methods. Index Terms— Federated learning, knowledge distillation, nonIID data, model heterogeneity, ensemble disagreement 1. INTRODUCTION Federated learning (FL) trains models across clients without centralizing raw data [1, 2], but its canonical instrument, parameter averaging, imposes two forms of homogeneity that practical deployments rarely satisfy: all clients must share one architecture, and performance degrades sharply when local data are non-IID [3, 4, 5, 6]. Knowledge distillation (KD) escapes the architectural constraint by exchanging soft predictions rather than weights [7, 8, 9], and its fully decentralized variant removes the central server and public transfer set as well: each client shares only a model snapshot, instantiates its peers as local teachers, evaluates them on its own private data, and distills from their combined soft predictions [5, 10]. The appeal is concrete: picture a fleet of voice-assistant devices jointly learning spoken-command recognition, where a home hub with an AI accelerator serves a full-scale acoustic model, a budget smartphone can host only compact ones, raw voice recordings must never leave the device, and each device hears its own skewed slice of the command vocabulary, speakers, and acoustic conditions. Peer distillation fits exactly: any two models over the same label space can exchange knowledge as black boxes, and because each teacher is evaluated on the student’s own recordings, no audio ever leaves a device, yet ∗ Corresponding author: [email protected].
knowledge of commands a device rarely hears still reaches it from peers that hear them often. In this setting, the distillation target for every training sample is an aggregate of peer teacher predictions, and under heterogeneous, non-IID data its quality is sharply uneven: for a sample whose class most peers have barely seen, the teacher pool splinters into a few informed opinions and many guesses. Prior methods respond with hard decisions. Agreement filtering discards teachers that deviate from the peer consensus beyond a fixed threshold [11], echoing Byzantine-robust aggregation [12]; the surviving teachers are then averaged, and the resulting target is imposed on the student with full force on every sample. Each hard decision loses graded information: a teacher just past the cutoff is silenced entirely while one just inside speaks at full volume, and a target formed from a pool in open conflict teaches as loudly as one backed by unanimous consensus. We propose to make trust graded everywhere. Our method, CALM (Class-wise Agreement and Label-gated Disagreement Modulation), replaces the pipeline’s hard decisions with three smooth gates operating at three levels. At the teacher-class level, each teacher’s contribution to the target for class c is weighted by exp(−∆/τ ), where ∆ is its deviation from the per-class peer consensus, so that agreement is rewarded continuously where a hard filter would impose a cutoff and discard the graded evidence. At the sample level, following the intuition that ensemble disagreement proxies epistemic uncertainty [13], the KD loss is scaled by exp(−d(x)/σ), where d(x) is the teachers’ mean divergence from the aggregated target, so contested targets contribute less than consensual ones. Both signals, however, are label-free and can be fooled when correlated teachers confidently agree on the wrong class. The third gate therefore anchors trust in the one piece of ground truth the client holds: distillation is further gated by (q y )η , the probability the aggregated target assigns to the sample’s true label, so peer knowledge that contradicts local evidence is attenuated rather than absorbed. Our contributions are: (1) we identify the all-or-nothing treatment of teacher trust, both in target construction and in loss application, as a failure mode of decentralized federated KD under nonIID data; (2) we propose CALM, three composable smooth trust gates (soft class-wise agreement, disagreement-modulated loss, and a label-consistency gate) that add no communication, no auxiliary data, and no learned parameters; and (3) on CIFAR-10, SVHN, OrganAMNIST, and Google Speech Commands, under heterogeneous architectures and severe Dirichlet skew, CALM improves global accuracy over uniform and hard-filtered distillation and matches or exceeds competing methods, with each gate contributing and the combination strongest. Beyond accuracy, the weight λ(x, y) that CALM assigns to each sample also separates samples whose peer target is wrong from those whose target is right (Fig. 2), a built-in indicator of where peer supervision can be trusted.
Fig. 1: Overview of the CALM framework. Left: in a server-free, public-data-free setting, clients with heterogeneous architectures exchange model snapshots, never raw data, and a student obtains teacher logits by running each on its own local batch. Top middle (Gate 1): CALM builds a soft class-wise agreement target, downweighting each teacher smoothly as it deviates from the peer consensus and renormalizing onto the probability simplex. Bottom middle (Gate 2): teacher disagreement around the aggregated target becomes a per-sample trust factor that lowers distillation strength when the pool conflicts. Bottom right (Gate 3): the label-consistency gate reads how strongly the target supports the ground-truth label, attenuating distillation when the target contradicts local evidence. Top right: the final KD weight multiplies Gates 2 and 3; the heatmap shows λ = λdis · g varying smoothly from full-strength to weak distillation, with no threshold anywhere.
2. RELATION TO PRIOR WORK
3. METHOD
Heterogeneity in FL. Parameter-space methods mitigate non-IID drift by regularization or control variates, e.g., FedProx [14] and SCAFFOLD [15], but operate in a shared weight space and cannot span distinct architectures [4]. KD-based FL exchanges predictions instead: FedMD [9] aligns clients on a public proxy set, generatorbased methods synthesize the transfer set instead of collecting it [16], and ensemble distillation [8] fuses predictions on a central server that distills on its own public or generated data; all relocate the data dependency rather than remove it, or restore the central coordinator that FL set out to avoid. DFML [5] distills peer-to-peer with neither server nor public data, the regime we operate in. CRAD [11] shares it, censoring teachers per class, whereas CALM replaces that hard decision with graded weighting and adds two sample-level gates. Weighting teachers and samples. Existing methods weight teachers by data volume [8], by class-wise reliability [17], or per sample by confidence against the ground-truth label [18], or censor outliers outright, echoing Byzantine-robust aggregation [12]. These schemes decide which teachers shape the target and how strongly, yet even the label-aware among them assume co-located teachers and impose the fused target at full strength on every sample. CALM’s agreement weighting is soft rather than censoring, and its two gates modulate how strongly each sample’s fused target is imposed. Per-sample loss scaling has a long lineage—heteroscedastic attenuation [19], curriculum learning [20]—but unlike those unsupervised weightings, CALM also vets the target against the client’s own labels.
Fig. 1 summarizes the framework. We consider N clients with private datasets Di over a shared label space {1, . . . , C}. Data are non-IID: class proportions differ sharply across clients, and a client may hold few or no samples of some classes. Models are heterogeneous: client i runs an architecture fi suited to its hardware. There is no central server and no public dataset; the only interface any two clients share is the C-dimensional class posterior. At communication round r, client i receives its peers’ round(r−1) model snapshots {θjr−1 }j∈Ni , where the teacher set Ni ⊆ {1, . . . , N } \ {i} contains all peers or, under partial participation, the other clients active in the round. Client i runs each snapshot locally as a teacher on its own training batches, so raw data never leaves the client. For a labeled sample (x, y) ∈ Di the teacher soft predictions are qj = softmax fθr−1 (x)/T ∈ ∆C−1 , j ∈ Ni , (1) j
with distillation temperature T P [7]. The conventional pipeline averages these uniformly, q = |N1i | j qj [8], optionally after censoring outlier teachers, and trains the student on L = (1 − α) LCE + α T 2 KL(q ∥ softmax(zs /T )) ,
(2)
where zs = fi (x) and α ∈ [0, 1] balances local supervision against distillation. Every sample’s target is trusted equally in Eq. (2), and any filtering of teachers is binary. CALM removes both hard assumptions with three gates.
3.1. Gate 1: Soft Class-wise Agreement Target For each class c, we form the preliminary consensus (the uniform average of the conventional pipeline) and each teacher’s deviation from it, q̃ c =
1 X c qj , |Ni | j∈N
∆cj = qjc − q̃ c ,
(3)
3.4. Full Objective The final per-sample distillation weight multiplies the two samplelevel gates, λ(x, y) = λdis (x) · g(x, y), (9)
i
and convert deviation into a soft agreement weight, acj = exp − ∆cj /τ ,
with exponent η ≥ 0 controlling the gate’s strength (η=0 disables it). If the peer target supports the true class, distillation proceeds; if it contradicts the local evidence, the sample’s KD term is attenuated and learning falls back on cross-entropy supervision.
(4)
with agreement scale τ . The distillation target is the agreementweighted average, renormalized onto the simplex: P c c q j∈N aj qj qc = P i c , q ← PC . (5) c′ a j∈Ni j c′ =1 q This weighting subsumes hard agreement filtering without any hard cutoff: a teacher far from the consensus on class c is exponentially attenuated rather than removed, and no teacher is ever irrevocably excluded. As τ → 0 the target concentrates on the most consensual teachers. Because the weights are computed per class, a teacher can be trusted for the classes it knows and discounted elsewhere, which is precisely the structure non-IID data induces.
and the training loss over a batch B ⊂ Di is i 1 X h LCALM = (1 − α) LCE + α λ(x, y) LKD , |B|
where LKD = T 2 KL(q ∥ softmax(zs /T )) and q is the soft agreement target of Eq. (5). The three gates share one design principle: every trust decision in the pipeline, which teacher shapes the target for a class, how strongly a sample’s target teaches, and whether it squares with the local label, is a smooth monotone function of the evidence rather than a threshold. Each gate degrades gracefully toward the baseline as its scale loosens (τ → ∞, σ → ∞, η → 0), the gates compose by multiplication, and none adds communication, auxiliary data, or learned parameters. 4. EXPERIMENTS
3.2. Gate 2: Disagreement-Modulated Distillation Strength Gate 1’s target, however, carries only the teachers’ relative deviations: the weighted average is unchanged when all agreement weights shrink by a common factor, precisely what happens when the entire pool drifts far from consensus. A pool in open conflict thus yields as clean-looking a target as a unanimous one, i.e., the severity of disagreement is erased in aggregation. We therefore reinstate that erased information as a per-sample signal, measuring how contested the target is by the teachers’ mean divergence from it, d(x) =
1 X KL(qj ∥ q) , |Ni | j∈N
(6)
i
and convert it into a per-sample trust factor, λdis (x) = clip exp − d(x)/σ , λmin , 1 ,
(10)
(x,y)∈B
(7)
where σ sets the tolerance to disagreement and the floor λmin prevents disagreement alone from silencing a sample, so that even contested samples retain a trickle of peer knowledge. Computing d(x) reuses the qj already produced for aggregation, costing one divergence per teacher. Since the KD gradient is simply multiplied by λdis (x), this gate acts as a per-sample learning rate on borrowed knowledge: consensual targets teach at full strength, contested ones whisper. 3.3. Gate 3: Label-Consistency Gate Both gates above are label-free, and share a blind spot: teachers trained on similarly skewed shards can agree, confidently and unanimously, on the wrong class, in which case low disagreement certifies a bad target. The student, however, holds ground truth for its own training samples. We therefore read the aggregated target’s mass on the true label y as a direct check of its reliability and gate distillation by g(x, y) = (q y )η , (8)
4.1. Setup Datasets and partitioning. We evaluate on four benchmarks spanning three modalities: CIFAR-10 and SVHN, natural-image benchmarks; OrganAMNIST, an eleven-class abdominal-CT organclassification benchmark [21] representing privacy-sensitive medical settings; and Google Speech Commands [22], a spoken keywordclassification benchmark. We use a 10-class subset of Speech Commands v0.02; each waveform is resampled to 16 kHz and converted to a 32-bin mel spectrogram (nfft =512, hop 160). Each dataset is partitioned across all N clients by a Dirichlet distribution with concentration αDir =0.3 (severe label skew); the skew-sensitivity study additionally uses αDir ∈ {0.1, 0.5}. The main comparison reports both N =50 and N =100, the latter doubling as the scaling study; the gate-by-gate ablation uses N =50. Compared methods. We compare against FedMD [9], FedGD [16], MSFKD [17], and DFML [5] (decentralized and public-datafree, the regime we share). FedMD and MSFKD use 5,000 unlabeled public proxy samples: CIFAR-100 images for CIFAR-10, SVHNextra images for SVHN, PathMNIST images for OrganAMNIST, and 200 samples from each of 25 unused keywords for SpeechCommands; all are disjoint from client and test data. FedGD instead synthesizes 500 samples per round. CALM uses neither. Within our backbone we compare: (i) Uniform KD, Eq. (2) with uniform averaging; (ii) Hard filtering, per-class censoring of teachers with an adaptive median threshold; and (iii) full CALM. Federation and protocol. Clients run three structurally distinct models assigned round-robin, ResNet-18 (≈11M parameters), ResNet-18-Half (≈2.8M), and CNN-6 (≈0.8M), covering width and family heterogeneity. In each round, 10 uniformly drawn clients are active (modeling intermittent availability) and exchange snapshots peer-to-peer, so each teacher pool Ni is the other nine. We train for R=300 communication rounds with distillation weight α=0.7, temperature T =4, agreement scale τ =0.1, label-gate exponent η=0.5, disagreement scale σ=1, and floor λmin =0.05. All
Table 1: Comparison on CIFAR-10, SVHN, OrganAMNIST, and SpeechCommands with 50 total clients (left block) and 100 total clients (right block). Global accuracy (%); best results are in bold. 50 clients Method
CIFAR-10
100 clients
SVHN
OrganA
Speech
CIFAR-10
SVHN
OrganA
Speech
Competing heterogeneous-FL methods FedMD [9] 20.26 FedGD [16] 21.01 MSFKD [17] 24.05 DFML [5] 40.07
17.27 10.12 16.08 70.35
20.25 22.11 21.98 58.76
10.48 17.15 19.09 61.49
10.50 11.38 19.82 29.43
19.35 10.01 18.31 22.86
20.34 13.47 23.08 46.98
10.46 10.81 10.85 37.23
Our backbone, varying target trust Uniform KD 28.43 Hard agreement filtering 39.60 CALM (ours) 39.83
44.24 58.57 71.50
53.51 59.38 61.05
75.39 79.74 81.85
23.40 30.78 31.89
23.78 34.87 42.86
50.24 51.13 52.80
59.93 60.00 62.61
Table 2: Gate-by-gate ablation on SVHN with 50 total clients (10 active per round). Global accuracy (%). Variant Uniform KD Soft target only Soft target + disagreement CALM (ours)
Gate 1
Gate 2
Gate 3
Global Acc.
– ✓ ✓ ✓
– – ✓ ✓
– – – ✓
44.24 58.53 63.84 71.50
Table 3: Global accuracy (%) on CIFAR-10 with 100 total clients (10 active per round) under increasing label skew. Smaller αDir means more skew. Method Uniform KD Hard agreement filtering CALM (ours)
αDir = 0.1
αDir = 0.3
αDir = 0.5
20.36 25.07 26.21
23.40 30.78 31.89
27.59 33.85 35.46
methods share the same client schedules, data partitions, and random seeds. Checkpoints are selected on each client’s local validation split and evaluated on the held-out global test set. 4.2. Main Results Table 1 reports global accuracy. CALM is best on every dataset under our backbone. It also consistently outperforms hard agreement filtering, with especially large gains on SVHN, and exceeds the strongest external baseline, DFML, on seven of the eight settings, trailing by only 0.24 points on 50-client CIFAR-10. Gains hold across all three modalities, so the gates are modality-agnostic. Scaling to 100 clients (right block) shrinks each shard and lowers every method that relies on local data, but CALM stays first on all four benchmarks (SVHN: 42.86% vs. 23.78% for uniform averaging). 4.3. Analyses Component ablation. Table 2 builds CALM gate by gate on SVHN. The soft agreement target provides the largest single jump, lifting Uniform KD from 44.24% to 58.53%; disagreement modulation adds a further 5.3 points, and the label gate completes CALM at 71.50%. The class-level agreement weighting (Gate 1) repairs the target by downweighting the few teachers that stray from the consensus, while the two complementary sample-level gates (Gates 2 and 3) discount the samples whose targets no reweighting can repair. Dis-
Fig. 2: Analysis of CALM’s final KD weight λ(x, y) on SVHN (50 clients, 10 active per round), final round. The distributions of λ(x, y) for correct and incorrect aggregated targets separate clearly (AUROC 0.970, 95% client-bootstrap CI [0.962, 0.979]); histograms are normalized within group.
agreement modulation (Gate 2) damps contested, high-divergence targets, whereas the label gate (Gate 3) assigns systematically lower g(x, y) to agreed-but-wrong targets that low disagreement alone leaves at full weight, which is why it yields the larger sample-level gain (+7.7 vs. +5.3 points). Robustness to skew. Table 3 varies the Dirichlet concentration αDir ∈ {0.1, 0.3, 0.5} on CIFAR-10 (100 clients). CALM leads at every skew level, with the largest gain over uniform averaging at intermediate skew (+8.5 points at αDir =0.3); at extreme skew the margins narrow, as few teachers overlap each client’s classes and little transferable signal remains to reweight. Trust-score reliability. Fig. 2 evaluates the deployed weight λ(x, y) as a per-sample trust score: wrong aggregated targets receive systematically lower λ (AUROC 0.970), so peer supervision is strongest where the target is right, and even correct targets are admitted only partially (median 0.435). 5. CONCLUSION We introduced CALM for decentralized, public-data-free federated learning, replacing the pipeline’s hard trust decisions with three smooth gates: class-wise agreement weighting builds the target, per-sample disagreement modulates how strongly it teaches, and a label gate anchors both in local labels. Under severe non-IID skew, CALM outperforms uniform distillation and hard filtering, and matches or exceeds prior decentralized KD.
6. REFERENCES [1] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas, “Communicationefficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 2017, vol. 54 of Proceedings of Machine Learning Research, pp. 1273–1282. [2] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaı̈d Harchaoui, et al., “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, vol. 14, no. 1– 2, pp. 1–210, 2021. [3] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra, “Federated learning with Non-IID data,” arXiv preprint arXiv:1806.00582, 2018. [4] Boyu Fan, Siyang Jiang, Xiang Su, Sasu Tarkoma, and Pan Hui, “A survey on model-heterogeneous federated learning: Problems, methods, and prospects,” in Proceedings of the IEEE International Conference on Big Data (BigData), 2024, pp. 7725–7734. [5] Yasser H. Khalil, Amir Hossein Estiri, Mahdi Beitollahi, Nader Asadi, Sobhan Hemati, Xu Li, Guojun Zhang, and Xi Chen, “DFML: Decentralized federated mutual learning,” Transactions on Machine Learning Research, 2024. [6] Daniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, Aris Anagnostopoulos, Ioannis Chatzigiannakis, and Andrea Vitaletti, “A thorough assessment of the non-IID data impact in federated learning,” Journal of Industrial Information Integration, vol. 50, 2026, Art. no. 101052. [7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. [8] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi, “Ensemble distillation for robust model fusion in federated learning,” in Advances in Neural Information Processing Systems 33 (NeurIPS), 2020, pp. 2351–2363. [9] Daliang Li and Junpu Wang, “FedMD: Heterogeneous federated learning via model distillation,” arXiv preprint arXiv:1910.03581, 2019. [10] Eunjeong Jeong and Marios Kountouris, “Personalized decentralized federated learning with knowledge distillation,” in IEEE International Conference on Communications (ICC), 2023, pp. 1982–1987. [11] Baraa Bilbeisi, Mengchen Fan, Baocheng Geng, and Qing Tian, “CRAD: Class-wise reliability-aware distillation for decentralized heterogeneous federated learning,” arXiv preprint arXiv:2609.00446, 2026. [12] Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in Advances in Neural Information Processing Systems 30 (NIPS), 2017, pp. 119–129. [13] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in Neural Information Processing Systems 30 (NIPS), 2017, pp. 6402–6413.
[14] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith, “Federated optimization in heterogeneous networks,” in Proceedings of Machine Learning and Systems (MLSys), 2020, vol. 2, pp. 429–450. [15] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” in Proceedings of the International Conference on Machine Learning (ICML), 2020, pp. 5132–5143. [16] Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu, “TARGET: Federated class-continual learning via exemplarfree distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4759–4770. [17] Dong Wang, Naifu Zhang, Meixia Tao, and Xu Chen, “Knowledge selection and local updating optimization for federated knowledge distillation with heterogeneous models,” IEEE Journal of Selected Topics in Signal Processing, vol. 17, no. 1, pp. 82–97, 2023. [18] Hailin Zhang, Defang Chen, and Can Wang, “Confidenceaware multi-teacher knowledge distillation,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 4498–4502. [19] Alex Kendall and Yarin Gal, “What uncertainties do we need in bayesian deep learning for computer vision?,” in Advances in Neural Information Processing Systems 30 (NIPS), 2017, pp. 5574–5584. [20] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, “Curriculum learning,” in Proceedings of the International Conference on Machine Learning (ICML), 2009, pp. 41–48. [21] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni, “MedMNIST v2 – a large-scale lightweight benchmark for 2D and 3D biomedical image classification,” Scientific Data, vol. 10, no. 1, 2023, Art. no. 41. [22] Pete Warden, “Speech commands: limited-vocabulary speech recognition,” arXiv:1804.03209, 2018.
A dataset for arXiv preprint