Benchmarking Federated Learning & Knowledge Distillation for Point Cloud Classification Aizierjiang Aiersilan
arXiv:2607.01272v1 [cs.GR] 30 Jun 2026
University of Macau [email protected]
Abstract. Deploying 3D point cloud analysis in privacy-sensitive and resource-constrained settings faces two coupled barriers: data cannot be centralized for training, and the trained model must run on limited edge hardware. We present a multi-seed benchmark that jointly evaluates federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification. It spans thirteen FL algorithms and ten KD objectives, supporting their full 130-pair teacher–objective cross-product per dataset; every standardized configuration is repeated over three random seeds for 504 training runs in total, with the complete combined grid evaluated at multi-seed scale on the clinical dataset. We characterize federated degradation and the combined-pipeline pitfall on ModelNet40, then validate them on a real-world clinical craniosynostosis dataset of patient head shapes, where the privacy and edge-deployment stakes are concrete. We report three findings. First, under extreme non-independent and identically distributed (non-IID) label skew, standalone FL degrades sharply: on ModelNet40 the strongest method reaches only 76.32% against a 92.26% centralized reference, on the clinical data the best reaches 75.83% against 100%, and the four server-side optimizers collapse to near the chance level; the best algorithm differs by dataset, so none is universally robust. Second, distillation compresses the teacher into a student 74.51% smaller and roughly twice as fast at inference, with five of the seven objectives evaluated on ModelNet40 matching or surpassing the 92.44% teacher. Third, the combined pipeline exposes an evaluation pitfall: when distillation keeps a hard-label cross-entropy term on a labeled proxy split, a collapsed federated teacher at 8.50% paired with Logit-MSE still yields a 92.94% student. This 84.4-point gap reflects the proxy labels rather than the federated model, and the hard-label term reuses the very labels whose privacy motivated federation. Objectives without a hard-label term instead track teacher quality (r ≈ 0.99 on the clinical grid) and collapse when the teacher does. We therefore recommend evaluating FL-KD pipelines with label-free distillation, so that the reported accuracy reflects the federated teacher rather than the proxy. Keywords: Federated learning · Knowledge distillation · 3D point cloud · Model compression · Non-IID · Privacy-preserving learning · Medical imaging · Benchmark
Benchmark & Code: https://ezharjan.github.io/FLKD3DBenchmark
2
1
Aizierjiang Aiersilan
Introduction
3D point cloud analysis underpins applications including autonomous driving, robotics, augmented reality, and medical imaging, where sensor data are naturally represented as irregular point sets in Euclidean space. Architectures such as PointNet [34] and PointNet++ [35] achieve strong classification performance on standard benchmarks, yet their deployment in privacy-sensitive, resourceconstrained settings faces two challenges that have not been studied together in one benchmark. The first challenge is data privacy. In medical [25, 39, 44], industrial [9, 27, 32], and multi-institutional settings [21, 26, 53], raw sensor data cannot be centralized because of regulatory or competitive constraints. Federated learning (FL) [30] addresses this by coordinating model training across clients without exchanging raw data, using iterative local updates and global aggregation. Even so, gradient inversion attacks [6, 61] show that transmitted gradients can still leak private data, and the broader open problems in FL are surveyed in [15]. The statistical heterogeneity of data across clients, the non-IID problem [12, 14, 28], degrades convergence and final accuracy even for well-established aggregation algorithms. These effects are well documented for image classification, but their impact on hierarchical 3D feature extraction remains largely uncharacterized. The second challenge is computational efficiency at deployment. A full PointNet++ model is often unsuitable for resource-constrained edge platforms such as embedded robotic controllers or portable clinical scanners. Knowledge distillation (KD) [11] transfers the capacity of a large teacher into a compact student, producing a smaller and faster network that retains much of the teacher’s accuracy. Several distillation objectives exist for image recognition, including feature hints [40], attention transfer [57], and self-distillation [5], but their effectiveness for 3D point cloud models has not been systematically benchmarked. Combining FL and KD is appealing: FL trains a shared model without centralizing data, and KD then compresses that model for edge deployment. A recent survey [37] identifies the absence of domain-specific benchmarks as a critical gap. In practice, the two-stage pipeline is usually evaluated end to end, by reporting the compressed student’s accuracy, rather than by asking whether that accuracy actually reflects the federated teacher. To the best of our knowledge, no prior work provides a systematic, reproducible evaluation of FL and KD strategies for 3D point cloud classification across the full cross-product of federated teachers and distillation objectives, let alone on clinical 3D shape data; 3DFFL [17] addresses federated few-shot learning for point clouds but does not evaluate KD losses or the combined pipeline. We study two complementary datasets. On ModelNet40 [52], a 40-class benchmark, we characterize federated degradation and isolate the combinedpipeline pitfall against a strong centralized reference. We then validate that pitfall on a real-world clinical craniosynostosis dataset [43] of patient head shapes, where privacy mandates federation [39, 44] and edge compression is a deployment necessity; there the full multi-seed cross-product confirms that hard-label masking conceals federated failure on a privacy-critical task. We make four contributions:
FL and KD for Point Cloud Classification
3
– Benchmark design. The benchmark provides thirteen FL algorithms, a centralized baseline, and ten KD objectives, and supports the complete two-stage cross-product of all 130 teacher–objective pairs per dataset. Our standardized multi-seed evaluation comprises 504 training runs over three seeds: on ModelNet40 the centralized baseline and the thirteen FL teachers, and on the clinical dataset the centralized baseline, the thirteen FL teachers, the ten KD objectives, and the full 130-pair combined grid; the ModelNet40 distillation and combined results come from a separate focused round (Section 4.3) selected for its wide spread of teacher quality. All results are reported as mean and standard deviation. The framework is open-source and is built to be extended with further FL algorithms, KD objectives, and datasets through one shared interface. – Quantified degradation under label skew. On ModelNet40 the strongest FL method trails the centralized reference by about 16 points (FedNova at 76.32% against 92.26%), on the clinical data by about 24 points (FedProx at 75.83% against 100%), and the four server-side optimizers (FedAvgM, FedAdam, FedYogi, FedAdagrad) collapse to near the chance level on both. The method ranking is dataset-dependent, so no single algorithm is robust to label skew. – Effective compression through distillation. On ModelNet40 distillation transfers the teacher’s accuracy into a student that is 74.51% smaller and roughly twice as fast at inference (about 50% lower latency), with five of seven objectives matching or surpassing the 92.44% teacher; the clinical task exhibits the same near-ceiling compression. – A controlled diagnosis of the combined pipeline. The teacher-byobjective grid separates the distillation objectives by their reliance on hard labels. Objectives that keep a hard-label cross-entropy term recover the student to near-centralized accuracy regardless of teacher quality, so that on ModelNet40 a collapsed federated teacher at 8.50% paired with Logit-MSE still yields a 92.94% student, an 84.4-point gap driven by the proxy labels, not by transfer. The feature- and attention-based objectives whose cross-entropy weight is exactly zero instead track the teacher and collapse when it collapses; the full multi-seed cross-product on the clinical data confirms the same split across all thirteen teachers, with the four pure-transfer objectives at r ≈ 0.99. Hard-label distillation thus masks federated failure while reusing exactly the labels that federation was meant to keep private, and we accordingly recommend two controls (distilling with the cross-entropy term removed, or on an unlabeled proxy split) that distinguish genuine knowledge transfer from this artifact.
2
Related Work
2.1
3D Point Cloud Learning
Early volumetric methods discretized shapes onto occupancy grids and applied 3D convolutions [52], but their memory cost grows cubically with resolution.
4
Aizierjiang Aiersilan
PointNet [34] introduced a permutation-invariant architecture that processes raw point sets via shared multi-layer perceptrons and global max pooling. PointNet++ [35] extended this with hierarchical set abstraction layers that capture local geometry at progressively coarser scales through farthest point sampling and ball-query grouping. Later architectures incorporate graph-based aggregation as in DGCNN [49] and self-attention as in Point Transformer [60]. Further advances include kernel-point convolutions (KPConv) [45], random-sampling segmentation with RandLA-Net [13], the Point Cloud Transformer (PCT) [8], PointNeXt [36], and Point Transformer V2 [51]. Sarker et al. [42] survey deep learning for point cloud understanding. We use PointNet++ single-scale grouping (SSG) as the teacher backbone because its hierarchical, locality-sensitive feature extraction provides a demanding setting for federated aggregation under non-IID label skew. 2.2
Federated Learning
We benchmark thirteen FL algorithms spanning the major families of aggregation strategy. FedAvg [30] averages client models weighted by dataset size after local gradient descent. To counter client drift under heterogeneity, FedProx [22] adds a proximal term, SCAFFOLD [16] introduces client-level control variates that reduce gradient variance, FedNova [48] normalizes the accumulated local updates to remove the objective inconsistency caused by differing local-step counts, and FedDyn [1] adds a dynamic per-client regularizer that aligns local and global stationary points. A second family modifies the server-side update: FedAvgM applies server momentum [12], while FedAdam, FedYogi, and FedAdagrad apply adaptive server optimizers [38]. FedMedian [54] replaces averaging with coordinatewise median aggregation for robustness, and FedBN [23] keeps batch-normalization statistics client-local to mitigate feature shift. Two methods tailored to non-IID data are MOON [19], which adds a model-contrastive term aligning client and global representations, and Ditto [20], which learns a personalized model per client alongside the global one. The effects of non-IID partitions on several of these strategies are documented for image classification [12,14,28]; we characterize them for hierarchical 3D point cloud architectures, including the methods most often recommended for label skew. 2.3
Knowledge Distillation
We benchmark ten KD objectives spanning the major families of distillation loss. Hinton et al. [11] showed that a student can match the soft class-probability outputs of a larger teacher via temperature-scaled Kullback–Leibler divergence. FitNets [40] aligned intermediate layer representations through hint-based training. Zagoruyko and Komodakis [57] transferred attention maps derived from intermediate feature tensors. Born-Again Networks [5] showed that self-distillation can yield a student that surpasses its teacher. Our benchmark also evaluates four representative relational and decoupled objectives: contrastive representation distillation (CRD) [46], relational knowledge distillation (RKD) [33], similaritypreserving distillation (SP) [47], and decoupled knowledge distillation (DKD) [59],
FL and KD for Point Cloud Classification
5
alongside simpler logit-matching variants [2, 29]. Other objectives such as the comprehensive feature overhaul [10] and the label-smoothing reinterpretation of KD [56] are not evaluated here. A general survey is given by Gou et al. [7]. For 3D data, structured distillation for detection [58] and adversarial distillation [41] have been explored, but neither benchmarks multiple loss objectives nor connects KD with federated pre-training. Qin et al. [37] survey the combination of FL and KD and identify the absence of domain-specific evaluations as a key open problem. KD has been integrated into FL to mitigate non-IID degradation and catastrophic forgetting in clinical settings [18, 55], yet systematic benchmarking for 3D hierarchical models remains unexplored. A complementary line distills on unlabeled or server-side proxy data [24], the approach our diagnosis ultimately motivates. 2.4
Clinical Shape Analysis
Craniosynostosis is the premature fusion of cranial sutures, which alters head shape. Schaufelberger et al. [43] released a statistical shape model of patients, with instances for the principal suture-fusion types and a control group, giving a privacy-conscious source of patient-like head shapes. Because such scans are distributed across clinics and cannot be pooled freely [25, 39, 44], the setting motivates federated training and compact models for point-of-care devices.
3
Benchmark Design
3.1
Datasets and Non-IID Partitioning
We evaluate on two complementary datasets. ModelNet40 [52] provides point clouds across 40 shape categories; the standard pre-processed normal-resampled split provides 9,843 training and 2,468 test shapes. It serves as a widely used benchmark on which to measure federated degradation at scale. The craniosynostosis dataset [43] provides 3D head-surface instances sampled from a statistical shape model, with four classes (a healthy control group and the coronal, metopic, and sagittal suture-fusion pathologies) and 100 instances per class; we use a deterministic stratified 80/20 split, giving 320 training and 80 test shapes. Each shape is represented by 1,024 points with three spatial coordinates and no surface normals, an input tensor of shape 3 × 1,024. The two datasets are deliberately contrasting: a large-vocabulary collection where the centralized task is moderately hard, and a small four-class clinical collection where the centralized task is essentially solved. We report the primary empirical results on ModelNet40, the more demanding benchmark, and use the clinical collection as a corroborating study; its near-perfect centralized ceiling lets us read off teacher quality unambiguously. Beyond these two, the data loader is dataset-agnostic and currently supports ModelNet10 [52], OmniObject3D [50], YCB [3], and GazeboSim [4] under the same interface, so the protocol can be transferred to other point-cloud sources. To model federated deployment, the training set of each dataset is partitioned among five clients with a label-skew non-IID strategy: all training samples are
6
Aizierjiang Aiersilan
sorted by class label and the ordered sequence is divided into five consecutive equal slices. Each client therefore concentrates on a disjoint segment of the label space, reflecting the class imbalance that arises when different facilities collect data for different purposes. This is a deliberately severe, worst-case form of heterogeneity; we adopt it because it most directly isolates the failure modes the benchmark is designed to surface, and we return to its implications in Section 5. Multi-seed protocol. To make the comparison statistically interpretable rather than dependent on a single run, every configuration (baseline, FL, KD, and combined) is trained independently with three random seeds, {7, 42, 123}. These seeds control the model initialization and the stochastic elements of training, namely mini-batch ordering and point-cloud augmentation; the label-skew partition is deterministic and held fixed across seeds, so the reported variance isolates training-time randomness rather than partition luck. Unless stated otherwise, all reported numbers are the mean over the three seeds, and tables give the corresponding standard deviation. 3.2
Model Architectures
Teacher: PointNet++ SSG. The teacher follows the PointNet++ single-scale grouping architecture [35] on raw coordinates without surface normals, with three hierarchical set-abstraction levels followed by a two-layer classification head. It occupies 5.65 MB on ModelNet40 and 5.62 MB on the four-class clinical data, measured from its parameter and buffer tensors. Student: a compact PointNet++ SSG. The student, SmallPointNet2, keeps the teacher’s hierarchical structure with narrower set-abstraction and classification layers. It occupies 1.44 MB on ModelNet40 (a 74.51% reduction from the teacher) and 1.42 MB on the clinical data, and exposes a penultimate feature so that feature-based distillation has a signal. 3.3
Federated Learning Strategies
We evaluate thirteen FL aggregation algorithms on the teacher architecture, chosen to span the major mechanism families: drift correction, server-side optimization, robust aggregation, and personalization. All configurations share the same federated setting of five clients, five local epochs per round, and twenty communication rounds, with local training using the Adam optimizer at learning rate 0.001 and batch size 24. Each strategy is listed with its distinguishing mechanism alongside its accuracy in Table 1. 3.4
Knowledge Distillation Objectives
We evaluate ten distillation objectives, all transferring knowledge from the PointNet++ SSG teacher to the compact student, spanning the major families of KD loss. Logit matching is represented by Vanilla KD [11], Logit-MSE [2], Cosine [29], and DKD [59]; intermediate-feature and attention alignment by
FL and KD for Point Cloud Classification
7
Feature KD [40], Attention Transfer [57], and SP [47]; relational and contrastive transfer by RKD [33] and CRD [46]; and self-distillation by the Born-Again protocol [5]. All but DKD share a soft-target weight α=0.5, and the feature, attention, SP, and RKD objectives add an intermediate-representation weight β=0.5; the soft-target objectives use temperature T =2.0. For Self-Distillation, an initial compact student trained from the teacher then supervises the final reported student. Central to our combined-pipeline diagnosis is the weight on the hard-label cross-entropy term, LCE . Vanilla KD, Logit-MSE, Cosine, CRD, and SelfDistillation weight it by (1 − α)=0.5, and DKD adds it at full weight, so all six retain a direct supervised signal from the ground-truth labels. The feature, attention, SP, and RKD objectives weight it by (1 − α − β), which is exactly 0 at α=β=0.5, so these four are pure teacher-transfer objectives with no hard-label term.
3.5
Combined Pipeline
The two-stage combined pipeline first trains a federated teacher (PointNet++ SSG) for twenty rounds under the non-IID partition; the best checkpoint then supervises the compact student in a KD run on the centralized training set. Figure 1 illustrates the workflow. This setup mirrors deployments where a periodically aggregated federated model is compressed for edge hardware using a server-side proxy dataset. We run Stage 2 on the fully labeled centralized data precisely to test the pitfall it can create, so that the spread of teacher quality and the role of the hard-label term can both be measured directly. The pipeline is datasetagnostic; on ModelNet40 we pair federated teachers spanning the full quality range with the distillation objectives. The same two stages apply unchanged to the clinical dataset, where we run the complete multi-seed cross-product of all thirteen FL teachers with all ten KD objectives (130 teacher–objective pairs) and reproduce the same recovery illusion.
4
Experiments
4.1
Centralized Baseline
We train the teacher on the full training split of each dataset over three seeds, using the common 200-epoch training protocol. The teacher reaches 92.26 ± 0.04% instance accuracy on ModelNet40 (89.58 ± 0.10% mean class accuracy) and 100.00 ± 0.00% on the four-class clinical data, using the best checkpoint per run averaged over seeds. The clinical task is essentially solved when trained centrally, which is useful for our purposes: it fixes an unambiguous reference for teacher quality against which the FL degradation and the combined-pipeline behavior can be read. We compare all FL and KD results against these centralized references.
8
Aizierjiang Aiersilan Stage 2: Knowledge Distillation centralized training; 200 epochs, Adam lr=0.001
Stage 1: Federated Learning privacy-protected; 5 clients × 20 rounds × 5 local epochs t
loca Client 1 l up datebroadcast disjoint label shard
.. .
local update Client 3 disjoint label shard .. .
Server Aggregation
s be
t
L F
ec ch
n oi kp
Thirteen FL Algorithms
te Client 5 p da disjoint label shard local u
Teacher (frozen) PointNet++ SSG 5.65 MB | FL checkpoint
soft logits / features
logits
input
Centralized proxy labeled training set
KD Objective Ten KD Objectives
T =2.0, α=0.5, β=0.5
input
train
Student (trained) SmallPointNet2 1.44 MB (−74.51%)
Fig. 1: Overview of the two-stage FL-KD benchmark pipeline. Stage 1 (blue, privacyprotected): Five non-IID clients train on label-skewed slices of the dataset; the server applies one of the thirteen benchmarked FL algorithms to produce a shared PointNet++ SSG teacher. Stage 2 (red): The best FL teacher checkpoint supervises a compact student on the centralized training set with a KD objective, yielding a smaller and faster student model.
4.2
Federated Learning Results
Table 1 reports the best instance accuracy of each FL strategy over twenty communication rounds under the non-IID partition, for both datasets. Per-round convergence trajectories appear in the supplementary material (Appendix C, Figure A7). Federated training degrades severely under the label-skew partition on both datasets (Table 1). On ModelNet40 the strongest method, FedNova, reaches only 76.32%, 15.94 points below centralized; the drift-correction methods (FedProx 71.04%, FedDyn 66.00%, SCAFFOLD 64.26%) improve on plain FedAvg (58.51%), whereas MOON (45.84%) falls below it and Ditto (61.36%) exceeds it only modestly. FedMedian reaches 39.13%, and the four server-side optimizers fall to near-trivial accuracy. The ranking is dataset-dependent, itself a finding: no single algorithm is robust. FedNova, the ModelNet40 leader, drops to mid-pack on the clinical data (50.83%), where FedProx leads at 75.83% and is the most consistent across both; SCAFFOLD falls from strong to 36.67% on the four-class problem, while Ditto and MOON rise into the upper group. The key result is constant: the best method trails centralized by 15.94 points on ModelNet40 and 24.17 on the clinical data, and the four server-side optimizers collapse to near-chance accuracy on both (4.05–6.28% near 2.5% ModelNet40 floor, 25.00–26.25% near the four-class floor). These results point to a vulnerability of this hierarchical 3D extractor to label skew. When each client sees only a few categories, the early set-abstraction layers likely calibrate their geometric priors to locally dominant shapes and transfer poorly after aggregation. The server-side optimizers are the most fragile: for the three adaptive variants, even at a tuned server rate of 0.05 the averaged client updates form an unreliable pseudo-gradient whose second-moment estimates become ill-conditioned, likely amplifying rather than correcting divergence [38]. Adaptive server aggregation is thus a practical risk in label-skewed edge settings,
FL and KD for Point Cloud Classification
9
Table 1: Standalone FL instance accuracy (%, best per run, mean±std over three seeds) on both datasets, with each method’s distinguishing mechanism (5 clients, non-IID label-skew, 20 rounds, 5 local epochs per round). Methods are ordered by ModelNet40 accuracy. The ranking is dataset-dependent: FedNova leads on ModelNet40 but drops to mid-pack on the clinical data, where FedProx leads, while the four server-side optimizers fail on both. On the balanced clinical test set the mean-class accuracy equals the instance accuracy. Method
Distinguishing Mechanism
ModelNet40 ↑ Craniosynostosis ↑
Centralized
Non-federated reference
92.26 ±0.04
100.00 ±0.00
FedNova [48] FedProx [22] FedDyn [1] SCAFFOLD [16] Ditto [20] FedAvg [30] FedBN [23] MOON [19] FedMedian [54] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]
Normalized local averaging Proximal regularization Dynamic regularization Control-variate correction Personalized plus global model Weighted parameter averaging Client-local batch-norm Model-contrastive term Coordinate-wise median Server-side Adagrad Server-side Adam Server-side momentum Server-side Yogi
76.32 ±1.55 71.04 ±0.87 66.00 ±1.28 64.26 ±3.75 61.36 ±4.11 58.51 ±0.25 55.28 ±2.37 45.84 ±7.51 39.13 ±5.79 6.28 ±1.09 4.71 ±0.98 4.05 ±0.00 4.05 ±0.00
50.83 ±8.04 75.83 ±10.63 58.33 ±8.13 36.67 ±1.44 71.25 ±9.92 69.58 ±2.60 65.42 ±4.73 69.17 ±8.78 64.58 ±7.53 26.25 ±2.17 25.00 ±0.00 25.00 ±0.00 25.00 ±0.00
where validation data for tuning is rarely available. This is a worst-case regime; softer partitions would narrow the gap (Section 5). Communication is identical across strategies: the full model occupies 5.65 MB per transmission, and each communication round sends and receives across five clients, totaling 56.5 MB per round, or approximately 1.13 GB over twenty rounds. Wall-clock cost varies with the local objective: MOON and Ditto roughly double FedAvg’s per-round time by training an auxiliary model on every client. Full payloads and times are reported in the supplementary material (Appendix C, Table A6). 4.3
Knowledge Distillation Results
We report the ModelNet40 distillation and combined stages on our primary testbed. The combined stage pairs federated teachers spanning the full quality range, from strong aggregators down to collapsed teachers, with seven distillation objectives (including a Basic KD variant that keeps the cross-entropy term at full weight as a probe); this spread is what makes the recovery illusion measurable. These teachers are individual checkpoints, so their standalone accuracies differ from the multi-seed means in Table 1: SCAFFOLD and FedDyn sit at 8.50% and 16.67% here against multi-seed means of 64.26% and 66.00%, and the centralized reference reaches 92.44%, close to the 92.26% multi-seed mean. The benchmark
10
Aizierjiang Aiersilan
Table 2: KD results on ModelNet40. Teacher: PointNet++ SSG (5.65 MB, 92.44%). Student: SmallPointNet2 (1.44 MB, 74.51% size reduction, approximately 50% lower inference latency). Instance accuracy is the peak over all training epochs. All strategies are run from the same pre-trained teacher checkpoint. The seven shown are six of the ten benchmark objectives plus a full-weight Basic KD probe; all ten are evaluated on the clinical task (Section 4.5). Strategy
Inst. Acc. (%) ↑ Class Acc. (%) ↑ vs. Teacher (%) ↑
Teacher (PointNet++ SSG) [35]
92.44
90.24
0.00
Attention Transfer [57] Basic KD [7] Vanilla KD [11] Self-Distillation [5] Feature KD [40] Cosine Similarity [29] Logit-MSE [2]
92.74 92.69 92.61 92.61 92.57 92.35 87.07
90.05 89.30 89.43 89.66 89.52 90.12 75.08
+0.30 +0.25 +0.17 +0.17 +0.13 −0.09 −5.37
supports the full 13×10 cross-product, which Section 4.5 reports at full multi-seed scale on the clinical task. Table 2 reports instance accuracy for each distillation objective distilled from the centralized ModelNet40 teacher (92.44%), and Figure 2 relates accuracy to model size together with the student-versus-teacher training dynamics. Distillation is an effective compressor for this architecture (Figure 2). Five of the seven objectives meet or surpass the 92.44% teacher on the compact student: Attention Transfer (92.74%), Basic KD (92.69%), Vanilla KD (92.61%), Self-Distillation (92.61%), and Feature KD (92.57%); Cosine Similarity (92.35%) trails by 0.09 points. The lone outlier is Logit-MSE (87.07%): a mean-squared error on the output vectors is scale-sensitive relative to the cross-entropy term and unbalances the loss, slowing convergence (Figure 2b). Compression is uniform across objectives (74.51% smaller, from 5.65 to 1.44 MB, and about 50% lower inference latency). This near-uniform success under a strong teacher is exactly what makes standalone KD a poor probe of the federated teacher: when the labels alone reach the ceiling, the objective is barely tested. The combined experiment exposes that. 4.4
Combined Federated Learning and Knowledge Distillation Results
Table 3 shows instance accuracy for the two-stage combined pipeline across the five FL teachers and the seven KD objectives, and Figure 3 plots each objective’s sensitivity to teacher quality. These five teachers span the full standalone-accuracy range, from FedProx at 67.61% down to SCAFFOLD at 8.50% (FedAvg reaches 60.83% with size-weighted and 56.38% with uniform averaging, and FedDyn 16.67%); we also evaluated FedMedian at 31.16% and the server-side FedAvgM and FedAdam at 4.05%, excluding them as teachers because their near-chance