Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

Benchmarking Federated Learning & Knowledge Distillation for Point Cloud Classification Aizierjiang Aiersilan

arXiv:2607.01272v1 [cs.GR] 30 Jun 2026

University of Macau ezharjan@outlook.com

Abstract. Deploying 3D point cloud analysis in privacy-sensitive and resource-constrained settings faces two coupled barriers: data cannot be centralized for training, and the trained model must run on limited edge hardware. We present a multi-seed benchmark that jointly evaluates federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification. It spans thirteen FL algorithms and ten KD objectives, supporting their full 130-pair teacher–objective cross-product per dataset; every standardized configuration is repeated over three random seeds for 504 training runs in total, with the complete combined grid evaluated at multi-seed scale on the clinical dataset. We characterize federated degradation and the combined-pipeline pitfall on ModelNet40, then validate them on a real-world clinical craniosynostosis dataset of patient head shapes, where the privacy and edge-deployment stakes are concrete. We report three findings. First, under extreme non-independent and identically distributed (non-IID) label skew, standalone FL degrades sharply: on ModelNet40 the strongest method reaches only 76.32% against a 92.26% centralized reference, on the clinical data the best reaches 75.83% against 100%, and the four server-side optimizers collapse to near the chance level; the best algorithm differs by dataset, so none is universally robust. Second, distillation compresses the teacher into a student 74.51% smaller and roughly twice as fast at inference, with five of the seven objectives evaluated on ModelNet40 matching or surpassing the 92.44% teacher. Third, the combined pipeline exposes an evaluation pitfall: when distillation keeps a hard-label cross-entropy term on a labeled proxy split, a collapsed federated teacher at 8.50% paired with Logit-MSE still yields a 92.94% student. This 84.4-point gap reflects the proxy labels rather than the federated model, and the hard-label term reuses the very labels whose privacy motivated federation. Objectives without a hard-label term instead track teacher quality (r ≈ 0.99 on the clinical grid) and collapse when the teacher does. We therefore recommend evaluating FL-KD pipelines with label-free distillation, so that the reported accuracy reflects the federated teacher rather than the proxy. Keywords: Federated learning · Knowledge distillation · 3D point cloud · Model compression · Non-IID · Privacy-preserving learning · Medical imaging · Benchmark

Benchmark & Code: https://ezharjan.github.io/FLKD3DBenchmark

2

1

Aizierjiang Aiersilan

Introduction

3D point cloud analysis underpins applications including autonomous driving, robotics, augmented reality, and medical imaging, where sensor data are naturally represented as irregular point sets in Euclidean space. Architectures such as PointNet [34] and PointNet++ [35] achieve strong classification performance on standard benchmarks, yet their deployment in privacy-sensitive, resourceconstrained settings faces two challenges that have not been studied together in one benchmark. The first challenge is data privacy. In medical [25, 39, 44], industrial [9, 27, 32], and multi-institutional settings [21, 26, 53], raw sensor data cannot be centralized because of regulatory or competitive constraints. Federated learning (FL) [30] addresses this by coordinating model training across clients without exchanging raw data, using iterative local updates and global aggregation. Even so, gradient inversion attacks [6, 61] show that transmitted gradients can still leak private data, and the broader open problems in FL are surveyed in [15]. The statistical heterogeneity of data across clients, the non-IID problem [12, 14, 28], degrades convergence and final accuracy even for well-established aggregation algorithms. These effects are well documented for image classification, but their impact on hierarchical 3D feature extraction remains largely uncharacterized. The second challenge is computational efficiency at deployment. A full PointNet++ model is often unsuitable for resource-constrained edge platforms such as embedded robotic controllers or portable clinical scanners. Knowledge distillation (KD) [11] transfers the capacity of a large teacher into a compact student, producing a smaller and faster network that retains much of the teacher’s accuracy. Several distillation objectives exist for image recognition, including feature hints [40], attention transfer [57], and self-distillation [5], but their effectiveness for 3D point cloud models has not been systematically benchmarked. Combining FL and KD is appealing: FL trains a shared model without centralizing data, and KD then compresses that model for edge deployment. A recent survey [37] identifies the absence of domain-specific benchmarks as a critical gap. In practice, the two-stage pipeline is usually evaluated end to end, by reporting the compressed student’s accuracy, rather than by asking whether that accuracy actually reflects the federated teacher. To the best of our knowledge, no prior work provides a systematic, reproducible evaluation of FL and KD strategies for 3D point cloud classification across the full cross-product of federated teachers and distillation objectives, let alone on clinical 3D shape data; 3DFFL [17] addresses federated few-shot learning for point clouds but does not evaluate KD losses or the combined pipeline. We study two complementary datasets. On ModelNet40 [52], a 40-class benchmark, we characterize federated degradation and isolate the combinedpipeline pitfall against a strong centralized reference. We then validate that pitfall on a real-world clinical craniosynostosis dataset [43] of patient head shapes, where privacy mandates federation [39, 44] and edge compression is a deployment necessity; there the full multi-seed cross-product confirms that hard-label masking conceals federated failure on a privacy-critical task. We make four contributions:

FL and KD for Point Cloud Classification

3

– Benchmark design. The benchmark provides thirteen FL algorithms, a centralized baseline, and ten KD objectives, and supports the complete two-stage cross-product of all 130 teacher–objective pairs per dataset. Our standardized multi-seed evaluation comprises 504 training runs over three seeds: on ModelNet40 the centralized baseline and the thirteen FL teachers, and on the clinical dataset the centralized baseline, the thirteen FL teachers, the ten KD objectives, and the full 130-pair combined grid; the ModelNet40 distillation and combined results come from a separate focused round (Section 4.3) selected for its wide spread of teacher quality. All results are reported as mean and standard deviation. The framework is open-source and is built to be extended with further FL algorithms, KD objectives, and datasets through one shared interface. – Quantified degradation under label skew. On ModelNet40 the strongest FL method trails the centralized reference by about 16 points (FedNova at 76.32% against 92.26%), on the clinical data by about 24 points (FedProx at 75.83% against 100%), and the four server-side optimizers (FedAvgM, FedAdam, FedYogi, FedAdagrad) collapse to near the chance level on both. The method ranking is dataset-dependent, so no single algorithm is robust to label skew. – Effective compression through distillation. On ModelNet40 distillation transfers the teacher’s accuracy into a student that is 74.51% smaller and roughly twice as fast at inference (about 50% lower latency), with five of seven objectives matching or surpassing the 92.44% teacher; the clinical task exhibits the same near-ceiling compression. – A controlled diagnosis of the combined pipeline. The teacher-byobjective grid separates the distillation objectives by their reliance on hard labels. Objectives that keep a hard-label cross-entropy term recover the student to near-centralized accuracy regardless of teacher quality, so that on ModelNet40 a collapsed federated teacher at 8.50% paired with Logit-MSE still yields a 92.94% student, an 84.4-point gap driven by the proxy labels, not by transfer. The feature- and attention-based objectives whose cross-entropy weight is exactly zero instead track the teacher and collapse when it collapses; the full multi-seed cross-product on the clinical data confirms the same split across all thirteen teachers, with the four pure-transfer objectives at r ≈ 0.99. Hard-label distillation thus masks federated failure while reusing exactly the labels that federation was meant to keep private, and we accordingly recommend two controls (distilling with the cross-entropy term removed, or on an unlabeled proxy split) that distinguish genuine knowledge transfer from this artifact.

2

Related Work

2.1

3D Point Cloud Learning

Early volumetric methods discretized shapes onto occupancy grids and applied 3D convolutions [52], but their memory cost grows cubically with resolution.

4

Aizierjiang Aiersilan

PointNet [34] introduced a permutation-invariant architecture that processes raw point sets via shared multi-layer perceptrons and global max pooling. PointNet++ [35] extended this with hierarchical set abstraction layers that capture local geometry at progressively coarser scales through farthest point sampling and ball-query grouping. Later architectures incorporate graph-based aggregation as in DGCNN [49] and self-attention as in Point Transformer [60]. Further advances include kernel-point convolutions (KPConv) [45], random-sampling segmentation with RandLA-Net [13], the Point Cloud Transformer (PCT) [8], PointNeXt [36], and Point Transformer V2 [51]. Sarker et al. [42] survey deep learning for point cloud understanding. We use PointNet++ single-scale grouping (SSG) as the teacher backbone because its hierarchical, locality-sensitive feature extraction provides a demanding setting for federated aggregation under non-IID label skew. 2.2

Federated Learning

We benchmark thirteen FL algorithms spanning the major families of aggregation strategy. FedAvg [30] averages client models weighted by dataset size after local gradient descent. To counter client drift under heterogeneity, FedProx [22] adds a proximal term, SCAFFOLD [16] introduces client-level control variates that reduce gradient variance, FedNova [48] normalizes the accumulated local updates to remove the objective inconsistency caused by differing local-step counts, and FedDyn [1] adds a dynamic per-client regularizer that aligns local and global stationary points. A second family modifies the server-side update: FedAvgM applies server momentum [12], while FedAdam, FedYogi, and FedAdagrad apply adaptive server optimizers [38]. FedMedian [54] replaces averaging with coordinatewise median aggregation for robustness, and FedBN [23] keeps batch-normalization statistics client-local to mitigate feature shift. Two methods tailored to non-IID data are MOON [19], which adds a model-contrastive term aligning client and global representations, and Ditto [20], which learns a personalized model per client alongside the global one. The effects of non-IID partitions on several of these strategies are documented for image classification [12,14,28]; we characterize them for hierarchical 3D point cloud architectures, including the methods most often recommended for label skew. 2.3

Knowledge Distillation

We benchmark ten KD objectives spanning the major families of distillation loss. Hinton et al. [11] showed that a student can match the soft class-probability outputs of a larger teacher via temperature-scaled Kullback–Leibler divergence. FitNets [40] aligned intermediate layer representations through hint-based training. Zagoruyko and Komodakis [57] transferred attention maps derived from intermediate feature tensors. Born-Again Networks [5] showed that self-distillation can yield a student that surpasses its teacher. Our benchmark also evaluates four representative relational and decoupled objectives: contrastive representation distillation (CRD) [46], relational knowledge distillation (RKD) [33], similaritypreserving distillation (SP) [47], and decoupled knowledge distillation (DKD) [59],

FL and KD for Point Cloud Classification

5

alongside simpler logit-matching variants [2, 29]. Other objectives such as the comprehensive feature overhaul [10] and the label-smoothing reinterpretation of KD [56] are not evaluated here. A general survey is given by Gou et al. [7]. For 3D data, structured distillation for detection [58] and adversarial distillation [41] have been explored, but neither benchmarks multiple loss objectives nor connects KD with federated pre-training. Qin et al. [37] survey the combination of FL and KD and identify the absence of domain-specific evaluations as a key open problem. KD has been integrated into FL to mitigate non-IID degradation and catastrophic forgetting in clinical settings [18, 55], yet systematic benchmarking for 3D hierarchical models remains unexplored. A complementary line distills on unlabeled or server-side proxy data [24], the approach our diagnosis ultimately motivates. 2.4

Clinical Shape Analysis

Craniosynostosis is the premature fusion of cranial sutures, which alters head shape. Schaufelberger et al. [43] released a statistical shape model of patients, with instances for the principal suture-fusion types and a control group, giving a privacy-conscious source of patient-like head shapes. Because such scans are distributed across clinics and cannot be pooled freely [25, 39, 44], the setting motivates federated training and compact models for point-of-care devices.

3

Benchmark Design

3.1

Datasets and Non-IID Partitioning

We evaluate on two complementary datasets. ModelNet40 [52] provides point clouds across 40 shape categories; the standard pre-processed normal-resampled split provides 9,843 training and 2,468 test shapes. It serves as a widely used benchmark on which to measure federated degradation at scale. The craniosynostosis dataset [43] provides 3D head-surface instances sampled from a statistical shape model, with four classes (a healthy control group and the coronal, metopic, and sagittal suture-fusion pathologies) and 100 instances per class; we use a deterministic stratified 80/20 split, giving 320 training and 80 test shapes. Each shape is represented by 1,024 points with three spatial coordinates and no surface normals, an input tensor of shape 3 × 1,024. The two datasets are deliberately contrasting: a large-vocabulary collection where the centralized task is moderately hard, and a small four-class clinical collection where the centralized task is essentially solved. We report the primary empirical results on ModelNet40, the more demanding benchmark, and use the clinical collection as a corroborating study; its near-perfect centralized ceiling lets us read off teacher quality unambiguously. Beyond these two, the data loader is dataset-agnostic and currently supports ModelNet10 [52], OmniObject3D [50], YCB [3], and GazeboSim [4] under the same interface, so the protocol can be transferred to other point-cloud sources. To model federated deployment, the training set of each dataset is partitioned among five clients with a label-skew non-IID strategy: all training samples are

6

Aizierjiang Aiersilan

sorted by class label and the ordered sequence is divided into five consecutive equal slices. Each client therefore concentrates on a disjoint segment of the label space, reflecting the class imbalance that arises when different facilities collect data for different purposes. This is a deliberately severe, worst-case form of heterogeneity; we adopt it because it most directly isolates the failure modes the benchmark is designed to surface, and we return to its implications in Section 5. Multi-seed protocol. To make the comparison statistically interpretable rather than dependent on a single run, every configuration (baseline, FL, KD, and combined) is trained independently with three random seeds, {7, 42, 123}. These seeds control the model initialization and the stochastic elements of training, namely mini-batch ordering and point-cloud augmentation; the label-skew partition is deterministic and held fixed across seeds, so the reported variance isolates training-time randomness rather than partition luck. Unless stated otherwise, all reported numbers are the mean over the three seeds, and tables give the corresponding standard deviation. 3.2

Model Architectures

Teacher: PointNet++ SSG. The teacher follows the PointNet++ single-scale grouping architecture [35] on raw coordinates without surface normals, with three hierarchical set-abstraction levels followed by a two-layer classification head. It occupies 5.65 MB on ModelNet40 and 5.62 MB on the four-class clinical data, measured from its parameter and buffer tensors. Student: a compact PointNet++ SSG. The student, SmallPointNet2, keeps the teacher’s hierarchical structure with narrower set-abstraction and classification layers. It occupies 1.44 MB on ModelNet40 (a 74.51% reduction from the teacher) and 1.42 MB on the clinical data, and exposes a penultimate feature so that feature-based distillation has a signal. 3.3

Federated Learning Strategies

We evaluate thirteen FL aggregation algorithms on the teacher architecture, chosen to span the major mechanism families: drift correction, server-side optimization, robust aggregation, and personalization. All configurations share the same federated setting of five clients, five local epochs per round, and twenty communication rounds, with local training using the Adam optimizer at learning rate 0.001 and batch size 24. Each strategy is listed with its distinguishing mechanism alongside its accuracy in Table 1. 3.4

Knowledge Distillation Objectives

We evaluate ten distillation objectives, all transferring knowledge from the PointNet++ SSG teacher to the compact student, spanning the major families of KD loss. Logit matching is represented by Vanilla KD [11], Logit-MSE [2], Cosine [29], and DKD [59]; intermediate-feature and attention alignment by

FL and KD for Point Cloud Classification

7

Feature KD [40], Attention Transfer [57], and SP [47]; relational and contrastive transfer by RKD [33] and CRD [46]; and self-distillation by the Born-Again protocol [5]. All but DKD share a soft-target weight α=0.5, and the feature, attention, SP, and RKD objectives add an intermediate-representation weight β=0.5; the soft-target objectives use temperature T =2.0. For Self-Distillation, an initial compact student trained from the teacher then supervises the final reported student. Central to our combined-pipeline diagnosis is the weight on the hard-label cross-entropy term, LCE . Vanilla KD, Logit-MSE, Cosine, CRD, and SelfDistillation weight it by (1 − α)=0.5, and DKD adds it at full weight, so all six retain a direct supervised signal from the ground-truth labels. The feature, attention, SP, and RKD objectives weight it by (1 − α − β), which is exactly 0 at α=β=0.5, so these four are pure teacher-transfer objectives with no hard-label term.

3.5

Combined Pipeline

The two-stage combined pipeline first trains a federated teacher (PointNet++ SSG) for twenty rounds under the non-IID partition; the best checkpoint then supervises the compact student in a KD run on the centralized training set. Figure 1 illustrates the workflow. This setup mirrors deployments where a periodically aggregated federated model is compressed for edge hardware using a server-side proxy dataset. We run Stage 2 on the fully labeled centralized data precisely to test the pitfall it can create, so that the spread of teacher quality and the role of the hard-label term can both be measured directly. The pipeline is datasetagnostic; on ModelNet40 we pair federated teachers spanning the full quality range with the distillation objectives. The same two stages apply unchanged to the clinical dataset, where we run the complete multi-seed cross-product of all thirteen FL teachers with all ten KD objectives (130 teacher–objective pairs) and reproduce the same recovery illusion.

4

Experiments

4.1

Centralized Baseline

We train the teacher on the full training split of each dataset over three seeds, using the common 200-epoch training protocol. The teacher reaches 92.26 ± 0.04% instance accuracy on ModelNet40 (89.58 ± 0.10% mean class accuracy) and 100.00 ± 0.00% on the four-class clinical data, using the best checkpoint per run averaged over seeds. The clinical task is essentially solved when trained centrally, which is useful for our purposes: it fixes an unambiguous reference for teacher quality against which the FL degradation and the combined-pipeline behavior can be read. We compare all FL and KD results against these centralized references.

8

Aizierjiang Aiersilan Stage 2: Knowledge Distillation centralized training; 200 epochs, Adam lr=0.001

Stage 1: Federated Learning privacy-protected; 5 clients × 20 rounds × 5 local epochs t

loca Client 1 l up datebroadcast disjoint label shard

.. .

local update Client 3 disjoint label shard .. .

Server Aggregation

s be

t

L F

ec ch

n oi kp

Thirteen FL Algorithms

te Client 5 p da disjoint label shard local u

Teacher (frozen) PointNet++ SSG 5.65 MB | FL checkpoint

soft logits / features

logits

input

Centralized proxy labeled training set

KD Objective Ten KD Objectives

T =2.0, α=0.5, β=0.5

input

train

Student (trained) SmallPointNet2 1.44 MB (−74.51%)

Fig. 1: Overview of the two-stage FL-KD benchmark pipeline. Stage 1 (blue, privacyprotected): Five non-IID clients train on label-skewed slices of the dataset; the server applies one of the thirteen benchmarked FL algorithms to produce a shared PointNet++ SSG teacher. Stage 2 (red): The best FL teacher checkpoint supervises a compact student on the centralized training set with a KD objective, yielding a smaller and faster student model.

4.2

Federated Learning Results

Table 1 reports the best instance accuracy of each FL strategy over twenty communication rounds under the non-IID partition, for both datasets. Per-round convergence trajectories appear in the supplementary material (Appendix C, Figure A7). Federated training degrades severely under the label-skew partition on both datasets (Table 1). On ModelNet40 the strongest method, FedNova, reaches only 76.32%, 15.94 points below centralized; the drift-correction methods (FedProx 71.04%, FedDyn 66.00%, SCAFFOLD 64.26%) improve on plain FedAvg (58.51%), whereas MOON (45.84%) falls below it and Ditto (61.36%) exceeds it only modestly. FedMedian reaches 39.13%, and the four server-side optimizers fall to near-trivial accuracy. The ranking is dataset-dependent, itself a finding: no single algorithm is robust. FedNova, the ModelNet40 leader, drops to mid-pack on the clinical data (50.83%), where FedProx leads at 75.83% and is the most consistent across both; SCAFFOLD falls from strong to 36.67% on the four-class problem, while Ditto and MOON rise into the upper group. The key result is constant: the best method trails centralized by 15.94 points on ModelNet40 and 24.17 on the clinical data, and the four server-side optimizers collapse to near-chance accuracy on both (4.05–6.28% near 2.5% ModelNet40 floor, 25.00–26.25% near the four-class floor). These results point to a vulnerability of this hierarchical 3D extractor to label skew. When each client sees only a few categories, the early set-abstraction layers likely calibrate their geometric priors to locally dominant shapes and transfer poorly after aggregation. The server-side optimizers are the most fragile: for the three adaptive variants, even at a tuned server rate of 0.05 the averaged client updates form an unreliable pseudo-gradient whose second-moment estimates become ill-conditioned, likely amplifying rather than correcting divergence [38]. Adaptive server aggregation is thus a practical risk in label-skewed edge settings,

FL and KD for Point Cloud Classification

9

Table 1: Standalone FL instance accuracy (%, best per run, mean±std over three seeds) on both datasets, with each method’s distinguishing mechanism (5 clients, non-IID label-skew, 20 rounds, 5 local epochs per round). Methods are ordered by ModelNet40 accuracy. The ranking is dataset-dependent: FedNova leads on ModelNet40 but drops to mid-pack on the clinical data, where FedProx leads, while the four server-side optimizers fail on both. On the balanced clinical test set the mean-class accuracy equals the instance accuracy. Method

Distinguishing Mechanism

ModelNet40 ↑ Craniosynostosis ↑

Centralized

Non-federated reference

92.26 ±0.04

100.00 ±0.00

FedNova [48] FedProx [22] FedDyn [1] SCAFFOLD [16] Ditto [20] FedAvg [30] FedBN [23] MOON [19] FedMedian [54] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]

Normalized local averaging Proximal regularization Dynamic regularization Control-variate correction Personalized plus global model Weighted parameter averaging Client-local batch-norm Model-contrastive term Coordinate-wise median Server-side Adagrad Server-side Adam Server-side momentum Server-side Yogi

76.32 ±1.55 71.04 ±0.87 66.00 ±1.28 64.26 ±3.75 61.36 ±4.11 58.51 ±0.25 55.28 ±2.37 45.84 ±7.51 39.13 ±5.79 6.28 ±1.09 4.71 ±0.98 4.05 ±0.00 4.05 ±0.00

50.83 ±8.04 75.83 ±10.63 58.33 ±8.13 36.67 ±1.44 71.25 ±9.92 69.58 ±2.60 65.42 ±4.73 69.17 ±8.78 64.58 ±7.53 26.25 ±2.17 25.00 ±0.00 25.00 ±0.00 25.00 ±0.00

where validation data for tuning is rarely available. This is a worst-case regime; softer partitions would narrow the gap (Section 5). Communication is identical across strategies: the full model occupies 5.65 MB per transmission, and each communication round sends and receives across five clients, totaling 56.5 MB per round, or approximately 1.13 GB over twenty rounds. Wall-clock cost varies with the local objective: MOON and Ditto roughly double FedAvg’s per-round time by training an auxiliary model on every client. Full payloads and times are reported in the supplementary material (Appendix C, Table A6). 4.3

Knowledge Distillation Results

We report the ModelNet40 distillation and combined stages on our primary testbed. The combined stage pairs federated teachers spanning the full quality range, from strong aggregators down to collapsed teachers, with seven distillation objectives (including a Basic KD variant that keeps the cross-entropy term at full weight as a probe); this spread is what makes the recovery illusion measurable. These teachers are individual checkpoints, so their standalone accuracies differ from the multi-seed means in Table 1: SCAFFOLD and FedDyn sit at 8.50% and 16.67% here against multi-seed means of 64.26% and 66.00%, and the centralized reference reaches 92.44%, close to the 92.26% multi-seed mean. The benchmark

10

Aizierjiang Aiersilan

Table 2: KD results on ModelNet40. Teacher: PointNet++ SSG (5.65 MB, 92.44%). Student: SmallPointNet2 (1.44 MB, 74.51% size reduction, approximately 50% lower inference latency). Instance accuracy is the peak over all training epochs. All strategies are run from the same pre-trained teacher checkpoint. The seven shown are six of the ten benchmark objectives plus a full-weight Basic KD probe; all ten are evaluated on the clinical task (Section 4.5). Strategy

Inst. Acc. (%) ↑ Class Acc. (%) ↑ vs. Teacher (%) ↑

Teacher (PointNet++ SSG) [35]

92.44

90.24

0.00

Attention Transfer [57] Basic KD [7] Vanilla KD [11] Self-Distillation [5] Feature KD [40] Cosine Similarity [29] Logit-MSE [2]

92.74 92.69 92.61 92.61 92.57 92.35 87.07

90.05 89.30 89.43 89.66 89.52 90.12 75.08

+0.30 +0.25 +0.17 +0.17 +0.13 −0.09 −5.37

supports the full 13×10 cross-product, which Section 4.5 reports at full multi-seed scale on the clinical task. Table 2 reports instance accuracy for each distillation objective distilled from the centralized ModelNet40 teacher (92.44%), and Figure 2 relates accuracy to model size together with the student-versus-teacher training dynamics. Distillation is an effective compressor for this architecture (Figure 2). Five of the seven objectives meet or surpass the 92.44% teacher on the compact student: Attention Transfer (92.74%), Basic KD (92.69%), Vanilla KD (92.61%), Self-Distillation (92.61%), and Feature KD (92.57%); Cosine Similarity (92.35%) trails by 0.09 points. The lone outlier is Logit-MSE (87.07%): a mean-squared error on the output vectors is scale-sensitive relative to the cross-entropy term and unbalances the loss, slowing convergence (Figure 2b). Compression is uniform across objectives (74.51% smaller, from 5.65 to 1.44 MB, and about 50% lower inference latency). This near-uniform success under a strong teacher is exactly what makes standalone KD a poor probe of the federated teacher: when the labels alone reach the ceiling, the objective is barely tested. The combined experiment exposes that. 4.4

Combined Federated Learning and Knowledge Distillation Results

Table 3 shows instance accuracy for the two-stage combined pipeline across the five FL teachers and the seven KD objectives, and Figure 3 plots each objective’s sensitivity to teacher quality. These five teachers span the full standalone-accuracy range, from FedProx at 67.61% down to SCAFFOLD at 8.50% (FedAvg reaches 60.83% with size-weighted and 56.38% with uniform averaging, and FedDyn 16.67%); we also evaluated FedMedian at 31.16% and the server-side FedAvgM and FedAdam at 4.05%, excluding them as teachers because their near-chance

,QVWDQFH$FFXUDF\

FL and KD for Point Cloud Classification          

$FFXUDF\YV0RGHO6L]H VL]HUHGXFWLRQ

7HDFKHU0% $WWHQWLRQ7UDQVIHU %DVLF.' 9DQLOOD.' 6HOI'LVWLOODWLRQ )HDWXUHEDVHG &RVLQH6LPLODULW\ /RJLW06(







0RGHO6L]H0%



11

6WXGHQWYV7HDFKHU$FFXUDF\ 

/RJLW06( &RVLQH 6LPLODULW\ )HDWXUHEDVHG

 

6HOI'LVWLOODWLRQ



9DQLOOD.'

 

%DVLF.' $WWHQWLRQ 7UDQVIHU



7HDFKHU







,QVWDQFH$FFXUDF\

(a) Accuracy vs. model size





(b) Training-epoch convergence

Fig. 2: Efficiency and distillation performance on ModelNet40. (a) Instance accuracy versus model size (left) and each objective’s student accuracy relative to the 92.44% teacher (right): all SmallPointNet2 students (1.44 MB) are 74.51% smaller than the teacher (5.65 MB), with most objectives within 0.3 points of the teacher. (b) Training-epoch accuracy for six of the seven objectives (Basic KD overlaps Vanilla KD and is omitted for clarity): Logit-MSE converges slower and plateaus below the teacher baseline, whereas the others asymptote at or above 92.35% within 100 epochs.

logits carry no usable distillation gradient. It shows a consistent dichotomy that turns on a single design choice, the weight of the hard-label cross-entropy term. Objectives that keep a hard-label term mask the teacher. The five objectives with a positive cross-entropy coefficient (Vanilla KD, Basic KD, SelfDistillation, Logit-MSE, and Cosine Similarity) recover near-centralized accuracy from every teacher: setting aside the runs that stopped before epoch 200 (marked †), every completed cell for these five lies between 92.13% and 92.94%. The peak cell, SCAFFOLD + Logit-MSE at 92.94%, exceeds the 92.44% centralized reference by 0.50 points despite an FL teacher that converges to only 8.50% on its own, a gap of 84.4 points above the teacher it distilled from, which cannot reflect genuine teacher-to-student transfer. Seven of the ten highest-accuracy configurations pair a hard-label objective with a collapsed teacher (three from SCAFFOLD at 8.50%, four from FedDyn at 16.67%). Objectives with no hard-label term reveal the teacher. Feature- and attention-based objectives behave oppositely. For collapsed teachers (SCAFFOLD at 8.50%, FedDyn at 16.67%) they reach only 15.82% to 18.20%, comparable to the teacher itself, and even with the FedProx teacher (67.61%) they stay in the 53% to 55% range. At α=β=0.5 their cross-entropy coefficient (1−α−β) is exactly 0, removing the labeled-data fallback: both remaining terms depend entirely on teacher quality, so a collapsed teacher drives the student toward meaningless representations. These objectives are thus a label-free probe of teacher quality, showing genuine transfer unpropped by proxy labels. Interpretation. Both behaviors follow from the loss L = wCE LCE (ŷ, y) + Ltransfer on the labeled centralized set. When wCE >0 and the teacher has collapsed, its soft targets give little useful gradient and the well-conditioned crossentropy term dominates, so the student learns the proxy labels directly; when wCE =0, the teacher is the only signal, so a failed teacher yields a failed student. An FL-KD evaluation that uses hard labels therefore cannot measure the federated teacher: it reduces the pipeline to centralized training on the proxy labels, reusing

12

Aizierjiang Aiersilan

Table 3: Two-stage combined pipeline: instance accuracy (%) on ModelNet40 (focused round, individual checkpoints). Each cell is the best instance accuracy of a SmallPointNet2 student trained with the given KD objective after federated pre-training of the teacher with the given FL strategy; “Std.” is the teacher’s own standalone instance accuracy. Column headers: Vanilla KD (VKD) [11], Basic KD (BKD) [7], Self-Distillation (SD) [5], Attention Transfer (AT) [57], Feature KD (FKD) [40], Logit-MSE (MSE) [2], Cosine Similarity (Cos) [29]. FedAvg (van.) uses uniform (unweighted) client averaging; the other FedAvg uses size-weighted averaging. Bold marks the best value in each KD-objective column. † Denotes runs stopped before 200 epochs; values are the best accuracy recorded to that point. These five teachers were chosen to span the full standalone-accuracy range (“Std.”); the complete 13 × 10 multi-seed cross-products appear in the supplementary material (ModelNet40) and in Section 4.5 (clinical task). FL Teacher

Std. VKD ↑ BKD ↑

FedAvg [30] FedAvg (van.) [30] FedProx [22] FedDyn [1] SCAFFOLD [16]

60.83 56.38 67.61 16.67 8.50

92.61 92.57 92.41 92.65 92.65

SD ↑

AT ↑ FKD ↑ MSE ↑ Cos ↑

92.57 92.45 50.74 92.53 92.65 47.31 92.13 73.85† 53.84 92.77 92.78 15.82† 92.61 92.45 16.99†

50.16 51.46 54.16 18.20 16.99

92.29† 92.82 66.97† 92.73 92.94

91.67† 87.84† 92.49 88.10† 92.33

the very labels whose privacy motivated federation. A sound evaluation distills without hard labels, by aggregating client soft predictions on unlabeled data [24] or omitting the cross-entropy term; the feature-based curves of Figure 3 are exactly that measurement. Two controls expose the illusion: a no-cross-entropy control (α=1, LCE removed) and an unlabeled-proxy control. Both leave the teacher as the sole signal, so a sound student falls back toward its teacher’s standalone accuracy (a SCAFFOLD-conditioned student toward 8.50%, not 92.94%); residual recovery signals cross-entropy-driven masking. 4.5

Real-World Validation: Clinical Craniosynostosis

The ModelNet40 grid isolates the recovery illusion under controlled conditions; the clinical craniosynostosis dataset shows that it survives on real patient data, where the stakes are concrete. These head shapes sit in separate clinics, cannot be pooled freely [39, 44], and must run on portable point-of-care scanners, so federation and compression are operational requirements rather than conveniences. We therefore run the complete cross-product of all thirteen FL teachers and ten KD objectives (130 teacher–objective pairs) at full multi-seed scale (three seeds). Standalone distillation compresses the 100% centralized teacher into a 1.42 MB student (74.7% smaller, roughly 2× faster) at near-ceiling accuracy across all ten objectives (supplementary material, Appendix B, Table A4); the unambiguous ceiling lets us read teacher quality directly. The combined grid reproduces the ModelNet40 dichotomy, and the hard-label cross-entropy term again decides the outcome (Table 4; see also the supplementary material, Appendix B, Figure A4). The six objectives that keep a cross-entropy

FL and KD for Point Cloud Classification

Distilled from collapsed SCAFFOLD (8.50%)

Centralized 92.44%

92.94

Vanilla KD Basic KD

60

Self-Distillation

40

Cosine Similarity

Vanilla KD Basic KD Self-Distillation Logit-MSE

20 10

Cosine Similarity Attention Transfer Feature KD

20 30 40 50 60 FL teacher standalone accuracy (%)

70

(a) Student vs. teacher accuracy

Feature KD

16.99

Attention Transfer

16.99

0

hard-label term pure transfer

centralized 92.44%

Logit-MSE

80

teacher 8.50%

Student instance accuracy (%)

13

92.65 92.61 92.45 92.33

20 40 60 80 100 Student instance accuracy (%)

(b) Recovery from a collapsed teacher

Fig. 3: The recovery illusion on ModelNet40 (focused round). (a) Student instance accuracy versus FL-teacher standalone accuracy for the seven distillation objectives of Table 3: the five objectives with a positive cross-entropy coefficient (Vanilla KD, Basic KD, Self-Distillation, Logit-MSE, Cosine Similarity) stay near the 92% centralized line for every teacher, even one collapsed to 8.50%, because the labeled proxy split supervises the student directly, whereas Attention Transfer and Feature KD, whose cross-entropy coefficient is exactly 0, fall in step with the teacher. (b) Student accuracy distilled from the collapsed SCAFFOLD teacher (8.50% standalone) under each objective, ordered by accuracy: the five hard-label objectives recover to the 92.44% centralized ceiling, a gap of up to 84.4 points above the teacher, while Attention Transfer and Feature KD stay near the teacher’s own level. The full 13 × 10 multi-seed heatmaps appear in the supplementary material.

term recover the student from any teacher: their accuracy varies by at most 7.5 points across the thirteen teachers and does not correlate with teacher quality (r from −0.53 to +0.37). A teacher frozen at the 25% chance level (FedAvgM) still yields a 97.5% student under Vanilla KD and 100% under CRD. The four pure-transfer objectives, whose cross-entropy coefficient is 0 at α=β=0.5, instead track the teacher almost perfectly (r=0.986 to 0.990): they span 57.5 to 58.3 points across teachers, fall to chance when the teacher does (FedAvgM with Attention Transfer or SP yields 25.0%), and recover only behind a strong teacher (FedProx with Feature KD reaches 84.2%, tracking its 75.83% teacher). The clinical data thus turns the diagnosis into a deployment warning. A hardlabel-distilled student here logs near-100% accuracy from a federated teacher no better than chance, while the cross-entropy term silently reuses the patient labels that federation was meant to protect. Only label-free distillation reports the federated teacher’s true quality. Per-cell means, standard deviations, and per-seed values appear in the supplementary material (Appendix C, Tables A8 and A9; Appendix D, Table A13).

5

Discussion and Limitations

Why does FL degrade severely for 3D architectures? Extreme label skew forces each client’s layers to overfit local categories, producing gradient

14

Aizierjiang Aiersilan

Table 4: Two-stage combined pipeline on the clinical task, summarized by KD objective over all thirteen FL teachers (mean instance accuracy, %, three seeds). “CE wt.” is the weight on the hard-label cross-entropy term. “Chance teacher” uses FedAvgM [12], a teacher at the 25% chance level; “best teacher” uses FedProx [22] at 75.83%. “Range” is the spread of student accuracy across the thirteen teachers, and r is the Pearson correlation between student accuracy and the teacher’s standalone accuracy. The six objectives that keep a cross-entropy term recover the student regardless of teacher quality (r ≈ 0); the four with zero cross-entropy weight track the teacher (r ≈ 0.99). Per-cell values appear in the supplementary material (Appendix C, Tables A8 and A9). KD Objective

CE wt.

Chance teacher

Best teacher

Range

r

97.5 99.2 99.6 100.0 97.5 93.8

99.6 99.6 100.0 99.6 99.2 96.7

2.5 1.2 0.8 0.8 3.8 7.5

+0.37 +0.19 −0.05 −0.53 +0.10 −0.10

Pure teacher-transfer objectives (1 − α − β = 0) Feature KD [40] 0 27.1 Attention Transfer [57] 0 25.0 Relational KD [33] 0 27.5 Similarity-Preserving [47] 0 25.0

84.2 83.3 81.7 82.9

57.5 58.3 57.5 58.3

+0.99 +0.99 +0.99 +0.99

Objectives retaining a hard-label term Vanilla KD [11] 0.5 Self-Distillation [5] 0.5 Cosine Similarity [29] 0.5 Contrastive (CRD) [46] 0.5 Logit-MSE [2] 0.5 Decoupled KD [59] 1.0

heterogeneity that standard aggregation cannot handle. Adaptive server optimizers suffer most, likely from ill-conditioned second-moment estimates under heterogeneous updates [38]. That this holds on both a 40-class benchmark and a four-class clinical dataset shows it is not an artifact of label-space size. No universally robust algorithm. The dataset-dependent ranking (Table 1) cautions against choosing an FL method from a single dataset: the leader on one dataset slips on the other, and even FedProx, the most consistent of the thirteen, never closes the gap to centralized training on both. That the contrastive, personalized, and adaptive server-side methods all leave this gap indicates these objectives do not by themselves overcome the extreme label skew studied here. The hard-label term governs the combined pipeline. An FL-KD pipeline’s reported accuracy is governed by whether the distillation loss keeps a hard-label term, not by the federated teacher: with such a term the student reaches the ceiling from any teacher. A faithful assessment must therefore distill on unlabeled data or drop the cross-entropy term, and report teacher-tracking accuracy. Hierarchical versus transformer backbones. The degradation may also depend on the backbone. The fixed-radius neighborhood aggregation that may underlie PointNet++’s sensitivity to label skew is specific to hierarchical models. Transformer backbones such as PCT [8] and Point Transformer V2 [51], and kernel-point methods such as KPConv [45], aggregate features through different mechanisms whose behavior under parameter averaging is not obvious in advance.

FL and KD for Point Cloud Classification

15

Whether such architectures degrade less than hierarchical models under non-IID federation is an open question that our single-backbone study cannot settle. Limitations. Several boundaries remain. First, we evaluate two datasets, ModelNet40 and a statistical-shape-model craniosynostosis set; larger real-world scanned collections remain to be tested. Second, the ModelNet40 combined diagnosis uses teachers spanning the full quality range down to collapsed checkpoints, and the clinical dataset confirms it at full multi-seed scale over the complete 13 × 10 grid, whose 100% ceiling saturates the standalone KD numbers and leaves the combined grid as the diagnostic. Third, we study one non-IID regime, the extreme disjoint label-skew partition; softer partitions would narrow the gap, and a heterogeneity sweep is left to future work. Fourth, the seeds vary initialization and optimization but not the deterministic partition, so the reported deviations capture training-time variance. Fifth, the architecture is a fixed PointNet++ SSG teacher and compact student; whether the effects persist for transformer or kernel-point backbones is open. Finally, all runs fix five clients, twenty rounds, and homogeneous client architectures; heterogeneous client models would require aggregation across dissimilar parameter spaces.

6

Conclusion

We presented a multi-seed benchmark for FL and KD in 3D point cloud classification, spanning thirteen FL algorithms, ten KD objectives, and their 130-pair cross-product. Standalone federated training degrades sharply under extreme label skew on both datasets, the method ranking is dataset-dependent, and the four server-side optimizers collapse to near the chance level, so no single algorithm is robust. Distillation compresses the strong centralized teacher into a student 74.51% smaller and roughly twice as fast at near-ceiling accuracy. The combined teacher-by-objective grid then exposes the central caution: any objective that keeps a hard-label cross-entropy term recovers the student to the centralized ceiling whatever the federated teacher, so its reported accuracy measures the proxy labels, not the federated model; only label-free objectives track teacher quality. The clinical craniosynostosis data shows the effect is not specific to ModelNet40: on a privacy-critical task, hard-label distillation reports a near-100% student from a chance-level teacher, while reusing the very labels federation was meant to protect. We therefore recommend evaluating FL-KD pipelines with label-free distillation, so that the reported accuracy measures the federated teacher. For reproducibility, the source code and the trained model checkpoints for every reported run are publicly available at https://ezharjan.github.io/ FLKD3DBenchmark, and new backbones, tasks, datasets, algorithms, and student architectures integrate through one shared interface. We envision this benchmark and its label-free evaluation protocol as a robust foundation for advancing privacypreserving, edge-deployable 3D point cloud systems, and we welcome community contributions to expand its scope.

16

Aizierjiang Aiersilan

References 1. Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., Saligrama, V.: Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 (2021), https://arxiv.org/abs/2111.04263, accessed: 2026-0629 4, 9, 12, 28, 41, 43, 45, 47, 51, 53, 56 2. Ba, L.J., Caruana, R.: Do deep nets really need to be deep? Advances in neural information processing systems 27 (2014), https://proceedings.neurips.cc/paper/ 2014/hash/b0c355a9dedccb50e5537e8f2e3f0810-Abstract.html, accessed: 202606-29 5, 6, 10, 12, 14, 29, 32, 44, 45, 47, 52, 56 3. Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., Dollar, A.M.: The ycb object and model set: Towards common benchmarks for manipulation research. In: 2015 international conference on advanced robotics (ICAR). pp. 510–517. IEEE (2015), https://doi.org/10.1109/ICAR.2015.7251504, accessed: 2026-06-29 5, 31 4. Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T.B., Vanhoucke, V.: Google scanned objects: A high-quality dataset of 3d scanned household items. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 2553–2560. IEEE (2022), https://doi.org/10.1109/ ICRA46639.2022.9811809, accessed: 2026-06-29 5, 31 5. Furlanello, T., Lipton, Z., Tschannen, M., Itti, L., Anandkumar, A.: Born again neural networks. In: International conference on machine learning. pp. 1607–1616. PMLR (2018), https://proceedings.mlr.press/v80/furlanello18a.html, accessed: 2026-06-29 2, 4, 7, 10, 12, 14, 29, 32, 44, 45, 47, 52, 56 6. Geiping, J., Bauermeister, H., Dröge, H., Moeller, M.: Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems 33, 16937–16947 (2020), https://proceedings.neurips.cc/paper/2020/ hash/c4ede56bbd98819ae6112b20ac6bf145-Abstract.html, accessed: 2026-06-29 2 7. Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. International journal of computer vision 129(6), 1789–1819 (2021), https://doi.org/10. 1007/s11263-021-01453-z, accessed: 2026-06-29 5, 10, 12 8. Guo, M.H., Cai, J.X., Liu, Z.N., Mu, T.J., Martin, R.R., Hu, S.M.: Pct: Point cloud transformer. Computational visual media 7(2), 187–199 (2021), https://doi.org/ 10.1007/s41095-021-0229-5, accessed: 2026-06-29 4, 14 9. Hao, M., Li, H., Luo, X., Xu, G., Yang, H., Liu, S.: Efficient and privacy-enhanced federated learning for industrial artificial intelligence. IEEE Transactions on Industrial Informatics 16(10), 6532–6542 (2019), https://doi.org/10.1109/TII.2019. 2945367, accessed: 2026-06-29 2 10. Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., Choi, J.Y.: A comprehensive overhaul of feature distillation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1921–1930 (2019), https://openaccess.thecvf. com/content_ICCV_2019/html/Heo_A_Comprehensive_Overhaul_of_Feature_ Distillation_ICCV_2019_paper.html, accessed: 2026-06-29 5 11. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015), https://arxiv.org/abs/1503.02531, accessed: 2026-06-29 2, 4, 6, 10, 12, 14, 29, 32, 44, 45, 47, 52, 56 12. Hsu, T.M.H., Qi, H., Brown, M.: Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019), https://arxiv.org/abs/1909.06335, accessed: 2026-06-29 2, 4, 9, 14, 28, 41, 43, 45, 47, 51, 54, 58

FL and KD for Point Cloud Classification

17

13. Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., Markham, A.: Randla-net: Efficient semantic segmentation of large-scale point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11108–11117 (2020), https://openaccess.thecvf.com/content_ CVPR_2020/html/Hu_RandLA-Net_Efficient_Semantic_Segmentation_of_LargeScale_Point_Clouds_CVPR_2020_paper.html, accessed: 2026-06-29 4 14. Jimenez G., D.M., Solans, D., Heikkila, M., Vitaletti, A., Kourtellis, N., Anagnostopoulos, A., Chatzigiannakis, I.: Non-iid data in federated learning: A survey with taxonomy, metrics, methods, frameworks and future directions. arXiv preprint arXiv:2411.12377 (2024), https://arxiv.org/abs/2411.12377, accessed: 2026-0629 2, 4 15. Kairouz, P., McMahan, H.B., et al.: Advances and open problems in federated learning. Foundations and trends in machine learning 14(1-2), 1–210 (2021), https: //doi.org/10.1561/2200000083, accessed: 2026-06-29 2 16. Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: Stochastic controlled averaging for federated learning. In: International conference on machine learning. pp. 5132–5143. PMLR (2020), https://proceedings.mlr. press/v119/karimireddy20a.html, accessed: 2026-06-29 4, 9, 12, 28, 41, 43, 45, 47, 51, 54, 57 17. Khan, A.A., Ahmad, K.M., Shafiq, S., Amin, W., Kumar, R.: 3dffl: privacypreserving federated few-shot learning for 3d point clouds in autonomous vehicles. Scientific Reports 14(1), 19589 (2024), https://doi.org/10.1038/s41598-02470326-5, accessed: 2026-06-29 2 18. Kim, S., Park, H., Kang, M., Jin, K.H., Adeli, E., Pohl, K.M., Park, S.H.: Federated learning with knowledge distillation for multi-organ segmentation with partially labeled datasets. Medical image analysis 95, 103156 (2024), https://doi.org/10. 1016/j.media.2024.103156, accessed: 2026-06-29 5 19. Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10713–10722 (2021), https://openaccess.thecvf.com/content/CVPR2021/html/ Li_Model-Contrastive_Federated_Learning_CVPR_2021_paper.html, accessed: 2026-06-29 4, 9, 28, 41, 43, 45, 47, 51, 52, 57 20. Li, T., Hu, S., Beirami, A., Smith, V.: Ditto: Fair and robust federated learning through personalization. In: International conference on machine learning. pp. 6357–6368. PMLR (2021), https://proceedings.mlr.press/v139/li21h.html, accessed: 2026-06-29 4, 9, 28, 41, 43, 45, 47, 51, 52, 57 21. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37(3), 50–60 (2020), https://doi.org/10.1109/MSP.2020.2975749, accessed: 2026-06-29 2 22. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2, 429–450 (2020), https://proceedings.mlsys.org/paper_files/paper/2020/ hash/1f5fe83998a09396ebe6477d9475ba0c-Abstract.html, accessed: 2026-06-29 4, 9, 12, 14, 28, 41, 43, 45, 47, 51, 52, 56 23. Li, X., Jiang, M., Zhang, X., Kamp, M., Dou, Q.: Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 (2021), https://arxiv.org/abs/2102.07623, accessed: 2026-06-29 4, 9, 28, 41, 43, 45, 47, 51, 53, 57

18

Aizierjiang Aiersilan

24. Lin, T., Kong, L., Stich, S.U., Jaggi, M.: Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems 33, 2351–2363 (2020), https://proceedings.neurips.cc/paper/2020/hash/ 18df51b97ccd68128e994804f3eccc87-Abstract.html, accessed: 2026-06-29 5, 12 25. Liu, S., Yang, H.H., Tao, Y., Feng, Y., Hao, J., Liu, Z.: Privacy-preserved federated learning for 3d tooth segmentation in intra-oral mesh scans. Frontiers in Communications and Networks 3, 907388 (2022), https://doi.org/10.3389/frcmn.2022. 907388, accessed: 2026-06-29 2, 5 26. Long, G., Tan, Y., Jiang, J., Zhang, C.: Federated learning for open banking. In: Federated learning: privacy and incentive, pp. 240–254. Springer (2020), https: //doi.org/10.1007/978-3-030-63076-8_17, accessed: 2026-06-29 2 27. Lu, Y., Huang, X., Dai, Y., Maharjan, S., Zhang, Y.: Blockchain and federated learning for privacy-preserved data sharing in industrial iot. IEEE Transactions on Industrial Informatics 16(6), 4177–4186 (2019), https://doi.org/10.1109/TII. 2019.2942190, accessed: 2026-06-29 2 28. Lu, Z., Pan, H., Dai, Y., Si, X., Zhang, Y.: Federated learning with non-iid data: A survey. IEEE Internet of Things Journal 11(11), 19188–19209 (2024), https: //doi.org/10.1109/JIOT.2024.3376548, accessed: 2026-06-29 2, 4 29. Luo, C., Zhan, J., Xue, X., Wang, L., Ren, R., Yang, Q.: Cosine normalization: Using cosine similarity instead of dot product in neural networks. In: International conference on artificial neural networks. pp. 382–391. Springer (2018), https: //doi.org/10.1007/978-3-030-01418-6_38, accessed: 2026-06-29 5, 6, 10, 12, 14, 29, 32, 44, 45, 47, 52, 56 30. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communicationefficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. Pmlr (2017), https://proceedings.mlr.press/v54/ mcmahan17a.html, accessed: 2026-06-29 2, 4, 9, 12, 28, 41, 43, 45, 47, 51, 52, 57 31. Mendoza, C.S., Safdar, N., Okada, K., Myers, E., Rogers, G.F., Linguraru, M.G.: Personalized assessment of craniosynostosis via statistical shape modeling. Medical Image Analysis 18(4), 635–646 (2014), https://doi.org/10.1016/j.media.2014. 02.008, accessed: 2026-06-29 31 32. Nguyen, D.C., Ding, M., Pathirana, P.N., Seneviratne, A., Li, J., Poor, H.V.: Federated learning for internet of things: A comprehensive survey. IEEE communications surveys & tutorials 23(3), 1622–1658 (2021), https://doi.org/10.1109/COMST. 2021.3075439, accessed: 2026-06-29 2 33. Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3967–3976 (2019), https://openaccess.thecvf.com/content_CVPR_ 2019/html/Park_Relational_Knowledge_Distillation_CVPR_2019_paper.html, accessed: 2026-06-29 4, 7, 14, 29, 32, 44, 45, 47, 52, 56 34. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652–660 (2017), https://openaccess. thecvf.com/content_cvpr_2017/html/Qi_PointNet_Deep_Learning_CVPR_2017_ paper.html, accessed: 2026-06-29 2, 4 35. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017), https://proceedings.neurips.cc/paper/2017/hash/ d8bf84be3800d12f74d8b05e9b89836f-Abstract.html, accessed: 2026-06-29 2, 4, 6, 10, 32, 44

FL and KD for Point Cloud Classification

19

36. Qian, G., Li, Y., Peng, H., Mai, J., Hammoud, H., Elhoseiny, M., Ghanem, B.: Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in neural information processing systems 35, 23192– 23204 (2022), https://proceedings.neurips.cc/paper_files/paper/2022/ hash / 9318763d049edf9a1f2779b2a59911d3 - Abstract - Conference . html, accessed: 2026-06-29 4 37. Qin, L., Zhu, T., Zhou, W., Yu, P.S.: Knowledge distillation in federated learning: A survey on long lasting challenges and new solutions. International Journal of Intelligent Systems 2025(1), 7406934 (2025), https://doi.org/10.1155/int/ 7406934, accessed: 2026-06-29 2, 5 38. Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečnỳ, J., Kumar, S., McMahan, H.B.: Adaptive federated optimization. arXiv preprint arXiv:2003.00295 (2020), https://arxiv.org/abs/2003.00295, accessed: 2026-06-29 4, 8, 9, 14, 28, 41, 43, 45, 47, 51, 54, 55, 58, 59 39. Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H.R., Albarqouni, S., Bakas, S., Galtier, M.N., Landman, B.A., Maier-Hein, K., et al.: The future of digital health with federated learning. NPJ digital medicine 3(1), 119 (2020), https: //doi.org/10.1038/s41746-020-00323-1, accessed: 2026-06-29 2, 5, 12 40. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014), https://arxiv. org/abs/1412.6550, accessed: 2026-06-29 2, 4, 7, 10, 12, 14, 29, 32, 44, 45, 47, 52, 56 41. Sanjay, S., Akash, J., Dimple, A.S., et al.: Adversarial learning based knowledge distillation on 3d point clouds. In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 2932–2941. IEEE (2025), https: //openaccess.thecvf.com/content/WACV2025/html/J_Adversarial_Learning_ Based_Knowledge_Distillation_on_3D_Point_Clouds_WACV_2025_paper.html, accessed: 2026-06-29 5 42. Sarker, S., Sarker, P., Stone, G., Gorman, R., Tavakkoli, A., Bebis, G., Sattarvand, J.: A comprehensive overview of deep learning techniques for 3d point cloud classification and semantic segmentation. Machine Vision and Applications 35(4) (2024), https://doi.org/10.1007/s00138-024-01543-1, accessed: 2026-06-29 4 43. Schaufelberger, M., Kühle, R., Wachter, A., Weichel, F., Hagen, N., Ringwald, F., Eisenmann, U., Hoffmann, J., Engel, M., Freudlsperger, C., et al.: A statistical shape model of craniosynostosis patients and 100 model instances of each pathology. Zenodo: Geneve, Switzerland (2021), https://doi.org/10.5281/zenodo.10167123, accessed: 2026-06-29 2, 5, 31 44. Sheller, M.J., Edwards, B., Reina, G.A., et al.: Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific Reports 10, 12598 (Jul 2020), https://doi.org/10.1038/s41598-020-69250-1, accessed: 2026-06-29 2, 5, 12 45. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6411–6420 (2019), https://openaccess.thecvf.com/content_ICCV_2019/html/Thomas_KPConv_ Flexible_and_Deformable_Convolution_for_Point_Clouds_ICCV_2019_paper. html, accessed: 2026-06-29 4, 14 46. Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019), https://arxiv.org/abs/1910.10699, accessed: 2026-06-29 4, 7, 14, 29, 32, 44, 45, 47, 52, 56

20

Aizierjiang Aiersilan

47. Tung, F., Mori, G.: Similarity-preserving knowledge distillation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1365–1374 (2019), https://openaccess.thecvf.com/content_ICCV_2019/html/Tung_SimilarityPreserving_Knowledge_Distillation_ICCV_2019_paper.html, accessed: 2026-0629 4, 7, 14, 29, 32, 44, 45, 47, 52, 56 48. Wang, J., Liu, Q., Liang, H., Joshi, G., Poor, H.V.: Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems 33, 7611–7623 (2020), https://proceedings.neurips. cc/paper/2020/hash/564127c03caab942e503ee6f810f54fd- Abstract.html, accessed: 2026-06-29 4, 9, 28, 41, 43, 45, 47, 51, 53, 56 49. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog) 38(5), 1–12 (2019), https://doi.org/10.1145/3326362, accessed: 2026-06-29 4 50. Wu, T., Zhang, J., Fu, X., Wang, Y., Ren, J., Pan, L., Wu, W., Yang, L., Wang, J., Qian, C., et al.: Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 803–814 (2023), https: //doi.org/10.1109/CVPR52729.2023.00084, accessed: 2026-06-29 5, 31, 60 51. Wu, X., Lao, Y., Jiang, L., Liu, X., Zhao, H.: Point transformer v2: Grouped vector attention and partition-based pooling. Advances in Neural Information Processing Systems 35, 33330–33342 (2022), https://proceedings.neurips.cc/ paper_files/paper/2022/hash/d78ece6613953f46501b958b7bb4582f-AbstractConference.html, accessed: 2026-06-29 4, 14 52. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1912–1920 (2015), https:// openaccess.thecvf.com/content_cvpr_2015/html/Wu_3D_ShapeNets_A_2015_ CVPR_paper.html, accessed: 2026-06-29 2, 3, 5, 31, 60 53. Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10(2), 1–19 (2019), https://doi.org/10.1145/3298981, accessed: 2026-06-29 2 54. Yin, D., Chen, Y., Kannan, R., Bartlett, P.: Byzantine-robust distributed learning: Towards optimal statistical rates. In: International conference on machine learning. pp. 5650–5659. Pmlr (2018), https://proceedings.mlr.press/v80/yin18a.html, accessed: 2026-06-29 4, 9, 28, 41, 43, 45, 47, 51, 53, 58 55. Yu, L., Huang, J.: Cyclic federated learning method based on distribution information sharing and knowledge distillation for medical data. Electronics 11(23), 4039 (2022), https://doi.org/10.3390/electronics11234039, accessed: 2026-06-29 5 56. Yuan, L., Tay, F.E., Li, G., Wang, T., Feng, J.: Revisiting knowledge distillation via label smoothing regularization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3903–3911 (2020), https://openaccess.thecvf.com/content_CVPR_2020/html/Yuan_Revisiting_ Knowledge_Distillation_via_Label_Smoothing_Regularization_CVPR_2020_ paper.html, accessed: 2026-06-29 5 57. Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016), https://arxiv.org/abs/1612.03928, accessed: 2026-06-29 2, 4, 7, 10, 12, 14, 29, 32, 44, 45, 47, 52, 56

FL and KD for Point Cloud Classification

21

58. Zhang, L., Dong, R., Tai, H.S., Ma, K.: Pointdistiller: Structured knowledge distillation towards efficient and compact 3d detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 21791–21801 (2023), https://openaccess.thecvf.com/content/CVPR2023/ html/Zhang_PointDistiller_Structured_Knowledge_Distillation_Towards_ Efficient_and_Compact_3D_Detection_CVPR_2023_paper.html, accessed: 202606-29 5 59. Zhao, B., Cui, Q., Song, R., Qiu, Y., Liang, J.: Decoupled knowledge distillation. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 11953–11962 (2022), https://openaccess.thecvf.com/ content/CVPR2022/html/Zhao_Decoupled_Knowledge_Distillation_CVPR_2022_ paper.html, accessed: 2026-06-29 4, 6, 14, 29, 32, 44, 45, 47, 52, 56 60. Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 16259–16268 (2021), https://openaccess.thecvf.com/content/ICCV2021/html/Zhao_Point_ Transformer_ICCV_2021_paper.html, accessed: 2026-06-29 4 61. Zhu, L., Liu, Z., Han, S.: Deep leakage from gradients. Advances in neural information processing systems 32 (2019), https://proceedings.neurips.cc/paper/ 2019/hash/60a6c4002cc7b29142def8871531281a-Abstract.html, accessed: 202606-29 2

Benchmarking Federated Learning & Knowledge Distillation for Point Cloud Classification Supplementary Material Aizierjiang Aiersilan University of Macau ezharjan@outlook.com

This document is the supplementary material for the paper “Benchmarking Federated Learning & Knowledge Distillation for Point Cloud Classification.” Appendix A documents the implementation in enough detail to reproduce all 504 runs: the shared optimization settings, the teacher and student architectures, the data augmentation, the deterministic non-independent and identically distributed (non-IID) label-skew partition, the per-method federated learning (FL) update rules, the ten knowledge distillation (KD) loss formulations, and the two diagnostic controls, each accompanied by supporting pseudocode or a parameter table. Appendix B presents the complete craniosynostosis study, covering both the standalone distillation results and the full 13 × 10 combinedpipeline grid. Appendix C gives the complete per-configuration numbers behind the summary tables and, in its final subsection, the full multi-seed ModelNet40 combined-pipeline grid; Appendix D lists every individual per-seed accuracy, for both the standalone runs and the clinical and ModelNet40 combined grids, before averaging; and Appendix E reports a generalization study on two further point-cloud datasets (ModelNet10 and OmniObject3D). References to the main paper are given by explicit numbers (for example, “Section 4.5 of the main paper”), whereas references prefixed with the letter A (for example, Table A8) point to tables, figures, and algorithms within this supplement. Throughout, accuracies are test instance accuracy in percent, reported as the mean over the three seeds {7, 42, 123} with standard deviation where shown.

Notation and Abbreviations The abbreviations and symbols used in the main paper and in this supplement are collected here so that they can be looked up at a glance. Each abbreviation is also defined at its first use in the running text. Abbreviations. Abbreviation

Meaning

FL KD

Federated learning Knowledge distillation

FL and KD for Point Cloud Classification: Supplementary Material

23

Abbreviation

Meaning

non-IID

Non-independent and identically distributed (the data-partition regime) Single-scale grouping (the PointNet++ teacher and student backbone) Multi-layer perceptron Cross-entropy (hard-label loss) Kullback–Leibler (divergence) Mean squared error Attention Transfer (KD objective) Similarity-Preserving distillation (KD objective) Relational Knowledge Distillation (KD objective) Contrastive Representation Distillation (KD objective) Decoupled Knowledge Distillation (KD objective) Target-class / non-target-class KD (the two terms of DKD)

SSG MLP CE KL MSE AT SP RKD CRD DKD TCKD NCKD DGCNN PCT KPConv GPU MB

/

Dynamic Graph Convolutional Neural Network Point Cloud Transformer Kernel Point Convolution Graphics processing unit Megabyte (model-size unit)

Symbols. Symbol

Meaning

K

Number of federated clients (here K=5)

Dk

Private data shard held by client k

θ, θg

Local and global (broadcast) model parameters

α

Soft-target (KD) weight (α=0.5)

β

Intermediate-representation weight (β=0.5)

T

Distillation temperature (T =2.0)

LCE

Hard-label cross-entropy loss on the true labels

Ltransfer

Generic teacher-transfer loss term

wCE

Weight applied to the hard-label cross-entropy term

DKL

Kullback–Leibler divergence

pTs ,

Temperature-softened student / teacher class distributions

pTt

zs , zt

Student / teacher output (log-probability) vectors

fs , ft

Student / teacher penultimate feature descriptors

ϕ

Learnable linear projection mapping the student feature to the teacher width

24

Aizierjiang Aiersilan

Symbol

Meaning

fˆ

ℓ2 -normalized feature, fˆ=f /∥f ∥2

As , At

Attention descriptors used by Attention Transfer

Gs , Gt

Row-normalized Gram (sample-similarity) matrices used by SP

ψD , ψA

Pairwise-distance and triplet-angle potentials used by RKD

LCRD

In-batch contrastive (InfoNCE-style) CRD loss

ηs

Server-side learning rate (server-optimizer methods)

µ, λ, τ

Method-specific regularization / contrastive-temperature hyperparameters

αD , βD

DKD weights on the TCKD and NCKD terms (1 and 8)

r

Pearson correlation between student and standalone teacher accuracy

A

Implementation Details

We document the full experimental protocol so that every number reported in the paper can be reproduced. The description is stated at the level of the learning procedure rather than any particular software interface, and the three benchmark stages (federated training, distillation, and their two-stage combination) are summarized as Algorithms A1, A2, and A3. Shared optimization protocol. Every model receives an input tensor of 1,024 points with three spatial coordinates and no surface normals, that is, a tensor of shape 3 × 1,024. All training, whether centralized, federated, or distillation, uses the Adam optimizer (β1 =0.9, β2 =0.999, ϵ=10−8 , weight decay 10−4 ) with learning rate 0.001 and batch size 24. The centralized teacher and every distilled student are trained for 200 epochs under a step learning-rate schedule that multiplies the rate by 0.7 every 20 epochs. Federated training runs for 20 communication rounds of 5 local epochs each across five clients; the per-method federated hyperparameters are listed in Table A2. For every run we retain the checkpoint with the highest test instance accuracy, and all reported accuracies use that best checkpoint. Output convention. Both backbones emit log-probability vectors, that is, a log-softmax over the class logits, rather than raw logits, and all distillation objectives are defined consistently on these outputs. The soft-target Kullback– Leibler term and the hard-label cross-entropy term therefore follow their standard formulation, while the two logit-space objectives (Logit-MSE and Cosine) act on the log-probability output vectors. This single convention keeps the objectives directly comparable, because every loss receives inputs of the same form. Network architectures. The teacher is a PointNet++ single-scale-grouping classifier with three hierarchical set-abstraction levels. The first two levels sample 512 and 128 centroids by ball-query grouping at radii 0.2 and 0.4 with 32 and 64 neighbors respectively, and the third level pools all remaining points into a single

FL and KD for Point Cloud Classification: Supplementary Material

25

global feature. The per-level shared multi-layer-perceptron widths are [64, 64, 128], [128, 128, 256], and [256, 512, 1024], so the teacher produces a 1,024-dimensional global descriptor, which is passed through two fully connected layers of widths 512 and 256 (each with batch normalization and dropout 0.4) before the linear classification head. The compact student preserves this three-level hierarchy but narrows every stage: it samples 256 and 64 centroids at the same radii 0.2 and 0.4 with 16 and 32 neighbors, then pools globally, with multi-layer-perceptron widths [32, 32, 64], [64, 64, 128], and [128, 256, 512] and a classification head of widths 256 and 128. The student therefore yields a 512-dimensional penultimate descriptor, which it exposes so that the feature-based distillation objectives have an intermediate signal to match. Data augmentation. Each training batch is augmented on the fly by three operations applied in sequence. First, random per-cloud point dropout removes a fraction of points drawn uniformly in [0, 0.875], resetting each dropped point to the cloud’s first point so that the point count is preserved. Second, random isotropic scaling multiplies the coordinates by a factor drawn uniformly in [0.8, 1.25]. Third, random translation shifts the cloud by an offset drawn uniformly in [−0.1, 0.1] independently along each axis. Only the spatial coordinates are transformed, and test clouds are evaluated without any augmentation. Non-IID label-skew partition. Stage 1 uses the deterministic label-skew partition of Algorithm A1: the training set is sorted by class label and cut into five contiguous slices of equal size, one per client, so that clients hold disjoint segments of the label space. This is a deliberately severe, worst-case form of heterogeneity, and it is held fixed across seeds so that the reported variance reflects training-time randomness rather than partition luck. On the balanced four-class clinical data (80 training shapes per class) the rule yields the composition in Table A1: two clients see a single class and the other three see two adjacent classes, so no client observes more than half of the label space and the central classes are split across clients. On ModelNet40 the same rule gives each client roughly 1,969 shapes spanning about eight contiguous categories; because the ModelNet40 class sizes are not uniform, the category boundaries do not fall exactly on slice edges, yet each client still observes a disjoint block of the 40 categories. Softer schemes such as a Dirichlet partition would reduce the heterogeneity. FL update rules. Every federated round follows the generic template of Algorithm A2: the server broadcasts the current global model, each of the five clients optimizes it locally for five epochs on its disjoint label-skewed shard, and the server aggregates the returned models. The thirteen algorithms differ only in the local objective, the server aggregation rule, or both, and we group them by the mechanism they add. Client optimization uses the shared Adam configuration for all methods except SCAFFOLD, whose control-variate correction is applied through explicit gradient-descent steps at the same learning rate. The drift-correction family keeps the server average but constrains the local update. FedProx adds a proximal penalty µ2 ∥θ − θg ∥2 that pulls each client toward the broadcast global model θg , with µ=0.01. FedDyn adds a dynamic regularizer that combines a linear term in an accumulated per-client gradient

26

Aizierjiang Aiersilan

1: Input: training set with class labels y1 , . . . , yN ; number of clients K 2: sort the sample indices by their class label into a single ordered sequence 3: cut the sorted sequence into K contiguous slices of near-equal size 4: assign slice k to client k as its private shard Dk ▷ disjoint label blocks 5: return the client shards D1 , . . . , DK Algorithm A1: Deterministic label-skew partition used in Stage 1. Sorting by label and cutting into equal contiguous slices gives each client a disjoint block of the label space, the worst-case non-IID regime studied here. The procedure depends only on the dataset, so it is identical across seeds. Table A1: Client composition of the label-skew partition on the four-class clinical training set (320 shapes, 80 per class, classes ordered as the control group and the coronal, metopic, and sagittal fusions). Each client holds 64 shapes drawn from at most two adjacent classes. Client 1

Client 2

Client 3

Client 4

Client 5

Control Coronal Metopic Sagittal

64 – – –

16 48 – –

– 32 32 –

– – 48 16

– – – 64

Total

64

64

64

64

64

with a quadratic proximal term of strength α=0.01, so that the local stationary points align with the global one over rounds. SCAFFOLD maintains server and client control variates c and ck and replaces the local gradient g by the corrected direction g + c − ck , refreshing ck from the option-II rule after each client pass. FedNova rescales each client’s accumulated update by its number of local steps τk before averaging and then reapplies the effective step count, which removes the objective inconsistency that arises when clients take different numbers of steps because their shards differ in size. The server-optimization family keeps a plain local objective and instead treats the average client displacement ∆ = θg − θ̄ as a pseudo-gradient at the server. FedAvgM applies momentum, v ← βv + ∆ then θ ← θ − ηs v, with ηs =1.0 and β=0.9. FedAdam, FedYogi, and FedAdagrad maintain first- and second-moment √ estimates m and v of the pseudo-gradient and update θ ← θ + ηs m/( v + ϵ); the three share ηs =0.05, β1 =0.9, ϵ=10−3 , and differ only in the second-moment recursion (an exponential moving average for Adam, a sign-stabilized variant for Yogi, and a cumulative sum for Adagrad). The moderate ηs =0.05 already reflects a tuned, non-trivial step size for the three adaptive optimizers, so their collapse under label skew is not a mere step-size artifact. The robust-aggregation method, FedMedian, replaces the coordinate-wise mean with the coordinate-wise median across client models. The remaining methods address feature shift and personalization. FedBN averages all parameters except

FL and KD for Point Cloud Classification: Supplementary Material

27

1: Input: global model θ; clients 1, . . . , K with disjoint label-skewed shards D1 , . . . , DK ; local epochs E 2: for each client k = 1, . . . , K in parallel do 3: broadcast the global model: θk ← θ 4: for E local epochs over Dk do 5: θk ← local optimization step under the method’s objective ▷ plain, proximal, dynamic, control-variate, or contrastive 6: end for 7: end for 8: θ′ ← Aggregate(θ1 , . . . , θK ) under the method’s server rule ▷ weighted mean, median, normalized average, or server optimizer 9: evaluate θ′ on the held-out test set and update the best checkpoint 10: return θ′ as the new global model Algorithm A2: One communication round of Stage 1 federated training. The template is shared by all thirteen algorithms; the local objective in line 5 and the server rule in line 8 are the method-specific components described above and parameterized in Table A2.

the batch-normalization statistics, which each client keeps local. MOON adds a model-contrastive term that pulls a client’s representation toward that of the global model and away from that of its previous local model, with weight µ=1.0 and temperature τ =0.5. Ditto trains the shared model with a plain local objective and, in addition, a per-client personalized model regularized toward the shared model with strength λ=0.1; the shared model is what the server aggregates. Distillation loss formulations. Table A3 gives the loss formulation of each of the ten distillation objectives. All objectives are defined on the log-probability outputs described above and share the intermediate-representation weight β=0.5 where one applies and the temperature T =2.0 in every soft-target term; all but DKD also share the soft-target weight α=0.5. Each Kullback–Leibler term is scaled by T 2 , the standard correction that keeps the magnitude of the soft-target gradient comparable to that of the hard-label gradient. In the notation of the table, pTs and pTt are the temperature-softened student and teacher distributions and zs , zt are the output vectors. The intermediate features are the penultimate global descriptors; because the student descriptor is 512-dimensional and the teacher descriptor is 1,024-dimensional, a learnable linear projection ϕ maps the student feature to the teacher width, after which both are ℓ2 -normalized [ ˆ (written fˆ=f /∥f ∥2 ), so Feature KD matches ϕ(f s ) to ft under a mean-squared error. The attention descriptors As and At are the channel-wise ℓ2 -normalized squared activations of these descriptors, that is, the attention-transfer construction adapted to a globally pooled feature rather than a spatial map. Gs and Gt are the row-normalized sample-similarity (Gram) matrices computed over the batch and used by SP, and ψD , ψA are the pairwise-distance and triplet-angle relational potentials used by RKD. The contrastive term LCRD is an in-batch

28

Aizierjiang Aiersilan

Table A2: Per-method hyperparameters of the thirteen FL algorithms, grouped by mechanism family. The notation x(v) means hyperparameter x is set to value v, and ηs is the server-side learning rate. A dash marks methods that add no hyperparameter beyond the shared federated setting (five clients, five local epochs, twenty rounds, and the shared Adam configuration). Client optimization is the shared Adam configuration for every method except SCAFFOLD, whose corrected update uses gradient descent at the same learning rate. Configuration

Key Parameters

FedAvg [30] – FedProx [22] µ(0.01) SCAFFOLD [16] – FedNova [48] – FedDyn [1] α(0.01) MOON [19] µ(1.0), τ (0.5) Ditto [20] λ(0.1) FedBN [23] – FedMedian [54] – FedAvgM [12] ηs (1.0), β(0.9) FedAdam [38] ηs (0.05), β1 (0.9), β2 (0.999), ϵ(10−3 ) FedYogi [38] ηs (0.05), β1 (0.9), β2 (0.999), ϵ(10−3 ) FedAdagrad [38] ηs (0.05), β1 (0.9), ϵ(10−3 )

normalized-temperature cross-entropy between student and teacher embeddings: each is mapped by a learnable head to a 128-dimensional space and compared at contrastive temperature 0.07, taking the teacher embedding of the same sample as the positive and the other in-batch teacher embeddings as negatives. Decoupled KD (DKD) splits the soft target into a target-class term (TCKD) and a nontarget-class term (NCKD), weighted by αD =1 and βD =8 and combined with a full-weight cross-entropy term, all at the shared temperature T =2.0. Two-stage pipeline and diagnostic controls. The combined pipeline runs the two stages in sequence (Algorithm A3). Stage 1 trains a federated teacher under the label-skew partition and keeps its best checkpoint. Stage 2 freezes that teacher and distills the compact student on the centralized training split, which acts as a labeled server-side proxy, using one of the distillation objectives. Because the proxy split is labeled, the hard-label cross-entropy term, where it is present, can drive the student directly from the labels rather than from the teacher, which is the mechanism behind the recovery illusion analyzed in Section 4.4 of the main paper. To separate genuine teacher-to-student transfer from this label-driven recovery, the benchmark implements two controls that leave the teacher as the only learning signal. The no-cross-entropy control removes the hard-label term (equivalently, it sets the soft-target weight to α=1 for the logit objectives), distilling on the same proxy inputs without using their labels.

FL and KD for Point Cloud Classification: Supplementary Material

29

Table A3: The ten KD loss objectives, using α=0.5, β=0.5, and T =2.0 except where noted (DKD). LCE denotes cross-entropy on true labels; fˆ=f /∥f ∥2 denotes an ℓ2 normalized feature vector and ϕ a learnable linear projection. The feature, attention, SP, and RKD objectives weight LCE by (1 − α − β)=0 at α=β=0.5, leaving no hardlabel term, and every soft-target Kullback–Leibler term carries the T 2 scaling. For Self-Distillation, pTsprev is the previous student generation. For DKD, TCKD and NCKD are the target-class and non-target-class soft terms weighted by αD =1 and βD =8 and combined with a full-weight LCE , all at the shared T =2.0. Strategy

Loss Formulation

Vanilla KD [11]

(1 − α)LCE + αT 2 DKL (pTt ∥ pTs )

Self-Distill. [5]

(1 − α)LCE + αT 2 DKL (pTsprev ∥ pTs )

Logit-MSE [2]

(1 − α)LCE + αLMSE (zs , zt )

Cosine [29]

(1 − α)LCE + α 1 − cos(zs , zt )

Feature KD [40]

\ ˆ (1 − α − β)LCE + αT 2 DKL (pTt ∥ pTs ) + βLMSE ϕ(f s ), ft

Attention [57]

(1 − α − β)LCE + αT 2 DKL (pTt ∥ pTs ) + βLMSE (As , At )

SP [47]

(1 − α − β)LCE + αT 2 DKL (pTt ∥ pTs ) + β∥Gs − Gt ∥2F

RKD [33]

(1 − α − β)LCE + αT 2 DKL (pTt ∥ pTs ) + β(ψD + ψA )

CRD [46]

(1 − α)LCE + αLCRD (fs , ft )

DKD [59]

LCE + αD TCKD + βD NCKD

The unlabeled-proxy control withholds the proxy labels entirely, so the student is supervised only by the teacher’s outputs on the proxy inputs. Under either control a faithful pipeline should fall back toward the teacher’s own standalone accuracy, and any accuracy well above it signals label-driven masking rather than transfer. Evaluation protocol. For every run we record the best test instance and mean-class accuracy over training, together with the model size (parameters plus buffers), the inference latency of a held-out evaluation batch on a single GPU, and, for federated runs, the per-round communication payload and wall-clock time. Accuracy is computed in fp32 from a full confusion matrix, from which we read both the overall instance accuracy and the macro per-class accuracy (the mean of the per-class recalls). On the balanced clinical test set (20 shapes per class) the macro class accuracy coincides with the instance accuracy, which is why the two columns are equal there. The label-skew partition is deterministic given the dataset, so the only source of run-to-run variation is the random seed, which controls the model initialization, the mini-batch ordering, and the augmentation sampling. Training environment and footprint. All runs use a single shared implementation and are trained on a single GPU per run in fp32. Reported model footprints are the combined size of the parameter and buffer tensors at fp32, which

30

Aizierjiang Aiersilan

1: Input: client shards D1 , . . . , DK ; proxy split P; distillation objective L; mode ∈ {labeled, no-CE, unlabeled} 2: Stage 1. run Algorithm A2 for R rounds and keep the best teacher checkpoint θT ; then freeze θT 3: Stage 2. initialize the compact student θS 4: set the hard-label weight wCE to the objective’s default if mode is labeled, else 0 ▷ both controls drop LCE 5: for 200 epochs over the proxy split P do 6: for each batch x ⊂ P do 7: y ← proxy labels of x if mode is not unlabeled, else none 8: read the frozen teacher’s outputs θT (x) 9: update θS by minimizing L on θT (x) and, weighted by wCE , on y 10: end for 11: end for 12: evaluate θS on the test set and keep the best checkpoint 13: return the distilled student θS Algorithm A3: Two-stage federated-then-distillation pipeline with the two diagnostic controls. The default mode distills on the labeled proxy split; the two controls remove the label signal so that the teacher is the only supervision and the measured accuracy reflects genuine transfer.

is why they grow slightly with the width of the output layer: the teacher occupies 5.65 MB on the 40-class ModelNet40 and 5.62 MB on the four-class clinical data, and the compact student occupies 1.44 MB and 1.42 MB respectively, a reduction of 74.51% and 74.7%. Inference latency is reported as the mean over 100 timed forward passes on a single held-out evaluation batch, taken after 10 warm-up passes with on-device timing, and is therefore subject to hardware-load variation. Communication cost is identical across strategies because each transmits the full model, and the per-round payloads and wall-clock times are listed in Table A6. Benchmark scope. The benchmark comprises 504 training runs. On ModelNet40 we run a centralized baseline and the thirteen FL algorithms (14 configurations). On the clinical dataset we run the centralized baseline, the thirteen FL algorithms, the ten KD objectives, and the full combined grid of thirteen FL teachers crossed with ten KD objectives (1 + 13 + 10 + 130 = 154 configurations). Each of the 168 configurations is repeated over the three seeds {7, 42, 123}, giving 42 runs on ModelNet40 and 462 on the clinical dataset. The standalone distillation and combined-pipeline study on ModelNet40 reported in the main paper comes from a separate focused round that predates this standardized multi-seed protocol and is not counted among the 504 runs; it is retained because it spans federated teachers across the full quality range, down to collapsed checkpoints, which is what makes the recovery illusion measurable. For completeness we also ran the full multi-seed ModelNet40 combined grid (Section C.1); like the additional-dataset runs of Appendix E, it likewise lies outside the 504-run count.

FL and KD for Point Cloud Classification: Supplementary Material

31

Reproducibility and extensibility. The study is built to be reproduced and extended. Data access goes through a single dataset-agnostic interface that already covers six point-cloud sources (ModelNet40 [52], ModelNet10 [52], OmniObject3D [50], YCB [3], GazeboSim [4], and the clinical craniosynostosis dataset [43]) with stratified per-class splitting and automatic handling of under-populated classes, so a new dataset can be added without changing the training protocol. Beyond the reported configuration, the framework provides a heterogeneous compact DGCNN student (about 1.5 MB) as an alternative to the same-family PointNet++ student, a multi-teacher distillation objective in addition to the ten benchmarked here, a sweep over the distillation temperature T ∈ {1, 2, 4, 8}, and Dirichlet label partitions that interpolate between the worst-case split studied here and milder heterogeneity. The two diagnostic controls of Algorithm A3 are first-class options, so the recommended label-free evaluation can be reproduced directly. Standalone part- and semantic-segmentation baselines are also included for completeness, though they fall outside the federated-distillation scope of this study. Finally, every training job checkpoints and resumes idempotently and the full grid is enumerated automatically, which is how the multi-seed benchmark of 504 runs was executed at scale; this design lets the community add new datasets, federated algorithms, distillation objectives, and student architectures through the same shared interface. Project page, model checkpoints, and community. The full benchmark, comprising the source code, configuration files, and step-by-step instructions for reproducing every one of the 504 runs, is hosted at the project page, https: //ezharjan.github.io/FLKD3DBenchmark. The complete set of trained model checkpoints, namely the saved network weights produced by every centralized, federated, distilled, and combined configuration in this study, is openly archived in the accompanying model repository at https://huggingface.co/ezharjan/ FLKD3DBenchmark, so that any reported result can be reloaded and verified directly without retraining. Our broader aim is to offer the community a shared and openly extensible foundation for privacy-preserving, edge-deployable 3D point cloud learning, one in which new federated algorithms, distillation objectives, datasets, and student architectures can be contributed through a single common interface and compared under the same label-free evaluation protocol we advocate here. We regard this not as a finished artifact but as an ongoing endeavour: together we can build benchmarks that make 3D vision trustworthy to deploy wherever data cannot be centralized. Contributions, extensions, and critiques are warmly welcomed, and issues, pull requests, and new benchmark submissions are all encouraged.

B

Clinical Craniosynostosis Study

Craniosynostosis is the premature fusion of cranial sutures, which alters head shape in characteristic ways; statistical shape modeling has long been used to assess it from 3D surfaces [31]. Schaufelberger et al. [43] released a statistical shape model of patients together with model-generated instances for the principal

32

Aizierjiang Aiersilan

Table A4: KD instance accuracy (%, best per run, mean±std over three seeds) on the clinical task, for the compact student distilled from the PointNet++ SSG teacher. Methods are ordered by accuracy. All ten objectives transfer the teacher’s accuracy into the compact student with at most a 1.25-point loss. On the balanced test set the mean-class accuracy equals the instance accuracy. Strategy

Craniosynostosis ↑

Teacher (PointNet++ SSG) [35]

100.00 ±0.00

Attention Transfer [57] Relational KD [33] Similarity-Preserving [47] Vanilla KD [11] Cosine Similarity [29] Contrastive (CRD) [46] Feature KD [40] Logit-MSE [2] Self-Distillation [5] Decoupled KD [59]

100.00 ±0.00 100.00 ±0.00 100.00 ±0.00 100.00 ±0.00 99.58 ±0.72 99.58 ±0.72 99.58 ±0.72 99.58 ±0.72 99.58 ±0.72 98.75 ±0.00

suture-fusion types and a control group, a privacy-conscious source of patient-like head shapes that is naturally distributed across clinics and so motivates both federated training and compact models for point-of-care devices. This appendix details the standalone clinical distillation behind the real-world validation of Section 4.5 of the main paper, together with the complete numerical tables. All clinical results use the standardized three-seed protocol over the full cross-product of thirteen FL teachers and ten KD objectives, and the 100% centralized ceiling makes teacher quality unambiguous. The clinical federated convergence is plotted alongside ModelNet40 in Figure A7, and the standalone FL accuracies are listed in Table 1 of the main paper and Table A5. B.1

Standalone Knowledge Distillation

Table A4 reports accuracy for each of the ten distillation objectives on the clinical task; Figure A2 shows the same ranking as a bar chart, Figure A1 relates accuracy to model size and inference latency, and Figure A3 isolates the compression dividend. Per-seed values are in Table A7. When the teacher is the strong centralized model, distillation is a dependable compressor. All ten objectives land within 1.25 points of the 100% teacher on the compact student: Attention Transfer, Relational KD, Similarity-Preserving distillation, and Vanilla KD reach 100% exactly, five further objectives reach 99.58%, and DKD trails marginally at 98.75%. The compression is uniform across objectives because they share the same teacher and student: the student is 74.7% smaller (from 5.62 to 1.42 MB) and roughly twice as fast at inference, requiring about 64 ms per held-out evaluation batch on average against the teacher’s 126 ms, close to a halving of latency. These timings are wall-clock averages on a single

FL and KD for Point Cloud Classification: Supplementary Material

33

Efficiency frontier — craniosynostosis Accuracy vs latency

100

100

90

90

80

80

70

60

50

FL + KD Federated Distilled Centralised Pareto frontier

40

30

2

3

4

Model size (MB)

5

Top-1 accuracy (%)

Top-1 accuracy (%)

Accuracy vs model size

70

60

50

FL + KD Federated Distilled Centralised Pareto frontier

40

30

60

80

100

120

140

Inference time (ms / batch)

Fig. A1: Efficiency of distillation on the clinical task. Test accuracy versus model size (left) and versus inference latency (right), colored by family. The compact KD students cluster at the top left, near the teacher’s accuracy at 1.42 MB and about 60 to 69 ms per evaluation batch, against the 5.62 MB, ≈126 ms teacher. The standalone FL models keep the teacher’s size and latency yet span the full accuracy range, and the combined students inherit the compact size while their accuracy spreads according to objective (Section 4.4 of the main paper).

GPU and are subject to hardware-load variation. The near-uniform success here is exactly what makes the standalone KD result a poor probe of the federated teacher in the combined pipeline: when the supervisory labels alone suffice to reach the ceiling, the distillation objective is barely tested. B.2

Combined Federated Learning and Knowledge Distillation Pipeline

This subsection reports the full evidence for the two-stage pipeline on the clinical task, in which each of the thirteen federated teachers is frozen after Stage 1 and then distilled into the compact student under each of the ten objectives, forming the complete 13 × 10 grid of 130 teacher–objective pairs evaluated over three seeds. Figure A4 renders the resulting student accuracy as a teacher-by-objective heatmap, and Tables A8 and A9 give the per-cell means and standard deviations behind it, from which the per-objective summary of Table 4 of the main paper is computed; Table A13 lists the individual per-seed values. As the analysis in Section 4.5 of the main paper establishes, the outcome is governed by a single design choice, the weight the objective places on the hard-label cross-entropy term, and the numbers tabulated here make that dependence explicit. The heatmap separates the ten objectives into two visually distinct groups. The six objectives that keep a cross-entropy term (Vanilla KD, Self-Distillation, Cosine, CRD, DKD, and Logit-MSE) form a uniformly bright band: the student

34

Aizierjiang Aiersilan

Knowledge-distillation accuracy — craniosynostosis

102

100.0

100.0

100.0

Teacher (centralised) (100.0%)

100.0

99.6

Top-1 accuracy (%)

100

99.6

99.6

99.6

99.6 98.8

98

96

94

92

90

88

n io nt

e tt

a

sp

rk

d

lla

ni

va

ne

si

co

d cr

e ur

at fe

se

m

t_ gi lo

lf se

d

dk

Fig. A2: Standalone distillation accuracy on the clinical task. Test instance accuracy of the ten KD objectives distilling the centralized (100%) teacher into the compact student (mean±std over three seeds); the dashed line marks the teacher. Four objectives reach 100%, five reach 99.58%, and Decoupled KD trails at 98.75%, so every objective lands within 1.25 points of the teacher. This is the bar-chart view of Table A4.

recovers to between 91.7% and 100% for every teacher, including the server-side optimizers frozen at the 25% four-class chance level. A teacher that learned nothing thus appears to yield a near-perfect student, because the labeled proxy split supplies a supervised signal that bypasses the teacher entirely. The four pure-transfer objectives (Feature KD, Attention Transfer, RKD, and SP), whose cross-entropy coefficient (1 − α − β) is exactly 0 at α=β=0.5, instead darken in step with the teacher: they reach the low-to-mid 80% range behind the strongest teacher (FedProx, 75.83% standalone) and fall to near the 25% chance level behind the collapsed server-side optimizers. Because the heatmap lists both axes in the benchmark’s native order rather than grouping them, the two kinds of column are interleaved rather than contiguous; Tables A8 and A9 instead group the six hard-label objectives apart from the four pure-transfer ones and order the teachers by descending standalone accuracy, so that the trend along each column can be read directly. The per-cell numbers make the dichotomy quantitative. Reading down the six hard-label columns of Table A8, student accuracy stays near the ceiling as the teacher degrades from FedProx (75.83% standalone) to the chance-level optimizers: across the thirteen teachers it varies by at most 7.5 points (decoupled

FL and KD for Point Cloud Classification: Supplementary Material

35

Distillation: accuracy kept, cost shed — craniosynostosis Accuracy retained per KD method

Compression dividend (shared student)

vanilla

100.0

sp

100.0

rkd

100.0

attention

100.0

logit_mse

99.6

self

99.6

feature

99.6

crd

99.6

64 ms

5.62 MB Model size

cosine dkd

126 ms −49% (2.0× faster)

Inference time

−75% (3.9× smaller)

99.6

Teacher (100.0%) 94

96

1.42 MB

Teacher Student

98.8

98

Student top-1 accuracy (%)

100

102

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Relative to teacher (×)

Fig. A3: Distillation keeps accuracy while shedding cost (clinical task). (left) Accuracy retained by each objective, all within 1.25 points of the 100% teacher. (right) The compression dividend of the shared compact student: model size falls from 5.62 to 1.42 MB (a 74.7% reduction, 3.9× smaller) and mean inference latency from about 126 to 64 ms (−49%, 2.0× faster). Latencies are wall-clock averages on a single GPU and are subject to hardware-load variation.

KD) and by under 4 points for the other five, and its correlation with teacher quality is negligible (Pearson r between −0.53 and +0.37 in Table 4 of the main paper). The four pure-transfer columns behave oppositely, spanning 57.5 to 58.3 points from the best to the worst teacher and tracking standalone teacher accuracy almost perfectly (r from 0.986 to 0.990). Two cells make the contrast concrete: behind FedAvgM, a teacher stuck at the 25% chance level, Vanilla KD still records a 97.5% student and CRD a 100% student, whereas Attention Transfer and SP report 25.0%, exactly the teacher’s own level. The standard deviations in Table A9 reinforce this reading: the hard-label cells are tight across seeds (typically under 1.5 points), whereas the pure-transfer cells reach standard deviations above 13 points, because each seed retrains the federated teacher and the pure-transfer student inherits that variation instead of being anchored by the fixed proxy labels. Table A13 exposes the same pattern run by run, confirming that the hard-label recovery holds on every individual seed while the pure-transfer accuracy rises and falls with the teacher checkpoint. The clinical grid therefore reproduces, on real patient-derived shapes and at full multi-seed scale, the recovery illusion isolated under controlled conditions: hard-label distillation reports a near-ceiling student from a federated teacher that never learned the task, while silently reusing the patient labels that federation is meant to protect, so that only the label-free objectives report the teacher’s true quality. No positive synergy, and a leaderboard that rewards collapse. Two further views sharpen the same conclusion. Figure A5 reports, for every teacher– objective cell, the combined student’s accuracy minus the better of its two components in isolation (the federated teacher on its own and the student distilled

36

Aizierjiang Aiersilan

directly from the centralized teacher). No configuration improves on its best single component by more than about half a point. The six hard-label objectives sit essentially at zero: the two-stage pipeline merely reproduces what centralized distillation already delivers through the proxy labels, adding no transfer of its own. The four pure-transfer objectives are sharply negative, falling as much as 75 points below their best component for the collapsed server-side optimizers, because once the hard-label term is removed the failed teacher is the only remaining signal. The combined pipeline therefore never synthesizes a student better than its parts, and the apparent gains under hard-label distillation are an artifact of the labeled proxy rather than genuine synergy. Figure A6 makes the same point from the top of the ranking: of the twelve highest-scoring teacher–objective combinations, four pair a hard-label objective with a server-side optimizer frozen near the 25% four-class chance level (FedAdagrad+Cosine, FedAvgM+CRD, FedAdam+CRD, and FedYogi+Cosine), each reaching 100%. A leaderboard built on hard-label distillation would therefore crown federated teachers that never learned the task.

FL and KD for Point Cloud Classification: Supplementary Material

37

Combined FL × KD accuracy — craniosynostosis 100 mean (%)

99

99

75

56

56

98

100

100

97

57

56

99

58

60

99

98

99

100

98

57

57

ditto

99

82

83

100

99

100

99

96

85

83

moon

100

75

74

100

99

100

100

97

78

75

fedbn 100

76

76

100

99

100

100

98

77

75

fedmedian 100

69

70

100

100

100

99

98

70

68

fedadagrad

99

32

31

99

99

100

100

99

32

32

fedyogi

99

27

28

100

99

100

100

98

31

29

fedadam

99

27

30

100

99

100

100

96

31

28

fedavgm

98

27

25

99

98

100

100

94

27

25

feddyn 100

64

63

99

96

100

100

92

63

64

scaffold

100

30

30

99

99

100

99

98

32

33

fedprox

100

84

83

100

99

100

100

97

82

83

fedavg

98

77

76

99

98

100

99

97

77

78

90

80

70

60

la

nil

va

re ion nt te at

tu

fea

e e lf se sin ms co it_ g o l

KD strategy

crd

d

dk

rkd

sp

Top-1 accuracy (%)

FL teacher

100

fednova

50

40

30 80

100

mean (%)

best: fedavg+cosine = 100.0%

Fig. A4: The recovery illusion and its control on the clinical task. Student accuracy for each FL teacher (rows) and KD objective (columns); the bars above each column give its mean over the thirteen teachers. The six columns that keep a hard-label cross-entropy term (vanilla, self, logit-MSE, cosine, CRD, DKD) are uniformly bright: the student recovers to roughly 92% to 100% even when the teacher is at the 25% chance level. The four feature-based columns (feature, attention, RKD, SP) instead darken in step with the teacher, collapsing to near-chance accuracy for the failed server-side optimizers and recovering only for the better teachers. Both axes follow the benchmark’s native order rather than being grouped, so the hard-label and feature-based columns are interleaved; Table A8 regroups them.

38

Aizierjiang Aiersilan

Where does FL+KD beat its parts? — craniosynostosis fedavg

-2

-22

-24

-1

-2

+0

-0

-2

-23

-22

fedprox

-0

-15

-17

+0

-0

+0

+0

-2

-18

-17

scaffold

-0

-69

-70

-0

-1

+0

-0

-1

-68

-67

feddyn

-0

-36

-37

-0

-4

+0

+0

-7

-37

-36

fedavgm

-2

-72

-75

-0

-2

+0

+0

-5

-72

-75

60

FL teacher

20 fedadam

-1

-73

-70

+0

-1

+0

+0

-3

-69

-72

fedyogi

-1

-72

-72

+0

-1

+0

+0

-0

-69

-71

fedadagrad

-1

-67

-69

-0

-1

+0

+0

+0

-68

-68

fedmedian

-0

-30

-30

+0

+0

+0

-0

-1

-30

-32

fedbn

-0

-23

-24

+0

-0

+0

+0

-1

-23

-25

moon

+0

-24

-26

+0

-1

+0

+0

-2

-22

-25

ditto

-1

-17

-17

+0

-1

+0

-0

-3

-15

-17

fednova

-1

-42

-40

-1

-1

-0

+0

-1

-43

-43

0

−20

Synergy: combined − best single (pp)

40

sp

d rk

d dk

cr d

si ne

−60

co

lf

m se lo gi t_

se

io n nt te at

ur e at fe

va

ni

lla

−40

KD strategy

Fig. A5: The combined pipeline shows no positive synergy (clinical task). For every FL teacher (rows) and KD objective (columns), the cell gives the combined student’s instance accuracy minus the better of its two stand-alone components (the federated teacher alone and the student distilled from the centralized teacher), in percentage points (mean over three seeds). Values at or below zero mean the two-stage pipeline does not improve on simply distilling from the centralized teacher. The six hard-label objectives (vanilla, self, logit-MSE, cosine, CRD, DKD) stay close to zero, reproducing centralized distillation through the proxy labels, whereas the four puretransfer objectives (feature, attention, RKD, SP) fall up to roughly 75 points below their best component for the collapsed server-side optimizers.

FL and KD for Point Cloud Classification: Supplementary Material

39

Top 12 FL+KD combinations (craniosynostosis) fedadagrad+cosine

100.0±0.0

ditto+self

100.0±0.0

fedavgm+crd

100.0±0.0

moon+vanilla

100.0±0.0

fedavg+cosine

100.0±0.0

fedadam+crd

100.0±0.0

fedyogi+cosine

100.0±0.0

fedprox+cosine

100.0±0.0

scaffold+vanilla

99.6±0.7

ditto+cosine

99.6±0.7

fedbn+cosine

99.6±0.7

fedbn+crd

99.6±0.7

98.50

98.75

99.00

99.25

99.50

99.75

100.00

100.25

100.50

Top-1 accuracy (%)

Fig. A6: The leaderboard rewards collapsed teachers (clinical task). The twelve highest-accuracy FL+KD combinations on the clinical task, each as mean±std over three seeds. All exceed 99.5%, yet several pair a hard-label objective (cosine, CRD, vanilla, self) with a federated teacher frozen near the 25% four-class chance level (FedAdagrad, FedAvgM, FedAdam, FedYogi), so the ranking reflects the proxy labels rather than the quality of the federated teacher.

40

C

Aizierjiang Aiersilan

Complete Numerical Results

This appendix collects the complete per-configuration numbers behind every summary in the main paper, so that each reported value can be traced to its source. It is organised in three movements: the standalone federated results, then the clinical combined pipeline, and finally the ModelNet40 counterpart. The standalone federated results open the appendix. Table A5 extends the comparison of Table 1 of the main paper with mean-class accuracy on both datasets and confirms that the instance-accuracy ranking survives under the class-balanced metric. Two figures then make those same endpoints legible at a glance. Figure A7 traces the per-round accuracy of all thirteen algorithms, so that the drift-correction methods are seen climbing steadily across the twenty rounds while the four server-side optimizers never leave the chance level, the clinical curves looking noisier only because that test set holds eighty shapes. Figure A8 then redraws the final accuracies as ranked bar charts, whose reordering between the two datasets is precisely the dataset-dependence discussed in Section 4.2 of the main paper. The cost behind these accuracies is reported beside them: Table A6 gives the per-round communication payload, identical across strategies because each transmits the full model, together with the wall-clock time that sets the lightweight aggregators apart from the contrastive and personalized methods (MOON and Ditto) that train an auxiliary model on every client, and Figure A9 turns that observation into a picture in which every method stacks on a single vertical line, since higher accuracy is never bought with more communication. The clinical combined pipeline follows. Table A7 unpacks each standalone-KD mean into its three per-seed values and shows that the sub-point spread in Table A4 is seed noise, after which Tables A8 and A9 lay out the full 13 × 10 teacher-by-objective matrix, mean and per-cell standard deviation, that underlies the summary of Table 4 of the main paper. Read from top to bottom, the six hard-label columns hold near the ceiling as the federated teacher degrades while the four pure-transfer columns fall in step with it, which is the recovery illusion of Figure 3 of the main paper made numerical. The ModelNet40 material then arrives as Section C.1, the full multi-seed combined grid, whose per-cell means and standard deviations appear in Tables A10 and A11, reproducing the same split on the larger label space. Unless noted otherwise, every value is a mean over the three seeds {7, 42, 123}, with the standard deviation given wherever it is shown. C.1

ModelNet40 Multi-Seed Combined Grid

The controlled combined-pipeline diagnosis of Section 4.4 of the main paper uses a focused round of hand-picked ModelNet40 teachers spanning the full quality range down to collapsed checkpoints. For completeness we also ran the full multi-seed combined grid on ModelNet40, crossing all thirteen standardized FL teachers with all ten KD objectives over the three seeds {7, 42, 123} (390 runs). These reuse the standardized protocol but lie outside the 504-run count of the main benchmark, exactly as the additional-dataset runs of Appendix E do;

FL and KD for Point Cloud Classification: Supplementary Material

41

Table A5: Complete standalone FL results on both datasets (5 clients, non-IID labelskew, 20 rounds, 5 local epochs per round), as mean±std over three seeds. Methods are ordered by ModelNet40 instance accuracy; Inst. and Class denote instance and mean-class accuracy (%). On the balanced clinical test set the two coincide. ModelNet40 Method

Craniosynostosis

Inst. ↑

Class ↑

Inst. ↑

Class ↑

Centralized

92.26 ±0.04

89.58 ±0.10

100.00 ±0.00

100.00 ±0.00

FedNova [48] FedProx [22] FedDyn [1] SCAFFOLD [16] Ditto [20] FedAvg [30] FedBN [23] MOON [19] FedMedian [54] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]

76.32 ±1.55 71.04 ±0.87 66.00 ±1.28 64.26 ±3.75 61.36 ±4.11 58.51 ±0.25 55.28 ±2.37 45.84 ±7.51 39.13 ±5.79 6.28 ±1.09 4.71 ±0.98 4.05 ±0.00 4.05 ±0.00

71.28 ±0.96 66.83 ±0.82 60.14 ±2.24 54.48 ±3.62 55.00 ±5.42 52.85 ±0.91 48.08 ±2.53 38.49 ±10.06 37.69 ±5.40 4.19 ±1.13 3.27 ±0.70 2.50 ±0.00 2.50 ±0.00

50.83 ±8.04 75.83 ±10.63 58.33 ±8.13 36.67 ±1.44 71.25 ±9.92 69.58 ±2.60 65.42 ±4.73 69.17 ±8.78 64.58 ±7.53 26.25 ±2.17 25.00 ±0.00 25.00 ±0.00 25.00 ±0.00

50.83 ±8.04 75.83 ±10.63 58.33 ±8.13 36.67 ±1.44 71.25 ±9.92 69.58 ±2.60 65.42 ±4.73 69.17 ±8.78 64.58 ±7.53 26.25 ±2.17 25.00 ±0.00 25.00 ±0.00 25.00 ±0.00

we report them because they reproduce, at multi-seed scale, the same dichotomy that the clinical grid shows. Figure A10 renders the grid as a teacher-by-objective heatmap, and Tables A10 and A11 give the per-cell means and standard deviations behind it (teachers ordered by descending standalone accuracy and the objectives grouped into the six that keep a hard-label term and the four pure-transfer ones, as in the clinical Table A8). The six hard-label columns stay high across almost every teacher (column means about 84 to 92%): the two plain Kullback–Leibler objectives (Vanilla KD and Self-Distillation) recover the student to about 92% from essentially every teacher, including the server-side optimizers frozen near the 4% chance level, with the only material dip (FedAdam, to 86.6% and 87.5%) traceable to a single divergent seed (Table A11); Decoupled KD likewise recovers from every teacher, though at a lower and more variable level (column mean about 88%); and the three objectives that instead match the teacher directly in logit or feature space (Logit-MSE, Cosine, CRD) recover from almost all teachers but can collapse on the single most degenerate teacher (FedAvgM), reflecting the greater sensitivity of direct logit- and feature-matching to a fully degenerate teacher on the 40-class task. The four pure-transfer columns instead track teacher quality (column means ≈41%), falling to the chance level when the teacher collapses. The strongest single cell pairs a FedYogi teacher frozen at the 4.05% chance level with Logit-MSE at 92.8%, an 88.8-point gap that again reflects the proxy labels rather than the teacher. Figures A11 and A12 confirm the reading from two further angles, exactly as on the clinical task: the combined

42

Aizierjiang Aiersilan Federated convergence — modelnet40

60

Federated convergence — craniosynostosis

moon fedmedian fedadagrad fedadam fedavgm fedyogi Centralised (92.3%)

100

Test instance accuracy (%)

Test instance accuracy (%)

80

fednova fedprox scaffold feddyn fedavg ditto fedbn

40

20

80

60

fedprox fedavg fedmedian moon ditto fedbn fednova

feddyn scaffold fedadagrad fedadam fedavgm fedyogi Centralised (100.0%)

40

20

0 2.5

5.0

7.5

10.0

12.5

Communication round

(a) ModelNet40

15.0

17.5

20.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Communication round

(b) Craniosynostosis

Fig. A7: FL convergence under the non-IID label-skew partition: per-round test instance accuracy (mean over three seeds) for all thirteen algorithms; the dashed line marks the centralized reference. (a) On ModelNet40, FedNova and FedProx lead, a broad middle band of FedDyn, SCAFFOLD, Ditto, FedAvg, FedBN, and MOON follows, FedMedian trails near 39%, and the four server-side optimizers stay near the chance level. (b) On the clinical task, FedProx leads, a broad middle group follows, SCAFFOLD stalls near 37%, and the four server-side optimizers stay at the 25% four-class chance level; the trajectories are noisier because the test set has only 80 shapes. Each legend is ordered by final accuracy.

pipeline shows no positive synergy, and the top of the leaderboard is populated by collapsed teachers paired with hard-label objectives. Figure A13 shows the corresponding accuracy–size–latency trade-off.

FL and KD for Point Cloud Classification: Supplementary Material

Federated strategy accuracy — modelnet40

Federated strategy accuracy — craniosynostosis 100

Centralised (92.3%)

Centralised (100.0%)

90

80

76.3 71.0

80 66.0

64.3

61.4

60

58.5

Top-1 accuracy (%)

Top-1 accuracy (%)

43

55.3 45.8 39.1

40

20

75.8 71.2

70

69.6

69.2 65.4

64.6 58.3

60

50.8

50 40

36.7

30 6.3

0

x ro

a

ov

dn

fe

dp

fe

dd

fe

yn

ld

fo

af sc

o tt

di

vg

da

fe

db

fe

n

n oo

an

m

dm

fe

i ed

d

4.7

ra ag

d da

fe

4.1

gi

gm

m

da

da fe

v da

fe

26.2

25.0

25.0

25.0

20

4.1

x ro

o dy

dp

fe

fe

tt

di

(a) ModelNet40

o

vg da

fe

n oo

m

n

ed

dm

fe

a

yn

n

ia

db

fe

dd

fe

ov

dn

fe

ld

fo

af

sc

ad

da

fe

i

og

m

m

da

gr

da

da

fe

vg

da

fe

dy

fe

(b) Craniosynostosis

Fig. A8: Federated strategy accuracy ranked (best per run, mean±std over three seeds). Bar-chart view of Table A5; the dashed line marks the centralized reference. (a) On ModelNet40 the order is FedNova, FedProx, FedDyn, SCAFFOLD, Ditto, FedAvg, FedBN, MOON, FedMedian, then the four server-side optimizers near the chance level. (b) On the clinical task the order changes to FedProx, Ditto, FedAvg, MOON, FedBN, FedMedian, FedDyn, FedNova, SCAFFOLD, then the four server-side optimizers at the 25% chance level. The reordering between panels is the datasetdependent ranking discussed in Section 4.2 of the main paper.

Table A6: Per-round communication and wall-clock cost of federated training. Communication is the full model payload aggregated over the five clients (uplink plus downlink) and is therefore identical across strategies; wall-clock time is the mean seconds per communication round on a single GPU. The clinical dataset is far smaller, so its rounds are seconds rather than minutes; strategies that train an auxiliary model on every client (MOON, Ditto) are markedly slower per round on ModelNet40. ModelNet40 Method FedNova [48] FedProx [22] FedDyn [1] SCAFFOLD [16] Ditto [20] FedAvg [30] FedBN [23] MOON [19] FedMedian [54] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]

Craniosynostosis

Comm. (MB) ↓

Time (s) ↓

Comm. (MB) ↓

Time (s) ↓

56.5 56.5 56.5 56.5 56.5 56.5 56.5 56.5 56.5 56.5 56.5 56.5 56.5

558 502 506 500 1081 497 545 940 566 565 495 504 563

56.2 56.2 56.2 56.2 56.2 56.2 56.2 56.2 56.2 56.2 56.2 56.2 56.2

13.0 13.6 14.3 13.2 25.7 12.9 12.6 32.1 12.5 12.6 12.6 12.7 12.6

44

Aizierjiang Aiersilan

Accuracy vs communication cost — modelnet40

Accuracy vs communication cost — craniosynostosis 100 100

80

fednova

60

feddyn scaffold ditto fedavg fedbn

Top-1 accuracy (%)

Top-1 accuracy (%)

fedprox

80

fedprox ditto fedavg moon fedbn fedmedian feddyn

60

fednova

scaffold

40

moon fedmedian

40

20 fedadagrad fedadam fedavgm fedyogi

fedadagrad fedadam fedavgm fedyogi

0

20 53

54

55

56

57

58

59

60

53

54

55

56

57

58

59

60

Communication / round (MB)

Communication / round (MB)

(a) Craniosynostosis

(b) ModelNet40

Fig. A9: Accuracy versus per-round communication cost. Best-checkpoint test accuracy against the per-round communication payload (MB, summed over the five clients’ uplink and downlink). Because every strategy transmits the full model, the per-round payload is identical across methods (about 56.2 MB on the clinical task and 56.5 MB on ModelNet40; Table A6), so all strategies share a single vertical line and accuracy varies only along it: higher accuracy is not bought with more communication. The dashed line marks the centralized reference. (a) On the clinical task the strategies spread from FedProx at the top down to the four server-side optimizers at the 25% chance level. (b) On ModelNet40 they spread from FedNova down to the four server-side optimizers near the floor.

Table A7: Per-seed standalone KD test instance accuracy (%) on the clinical task, for the compact student distilled from the centralized teacher. Methods are ordered by mean accuracy. Column headers are the random seeds. Strategy

7↑

42 ↑

123 ↑

Teacher [35]

100.00

100.00

100.00

Attention Transfer [57] Relational KD [33] Similarity-Preserving [47] Vanilla KD [11] Cosine Similarity [29] Contrastive (CRD) [46] Feature KD [40] Logit-MSE [2] Self-Distillation [5] Decoupled KD [59]

100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.75 100.00 98.75

100.00 100.00 100.00 100.00 98.75 98.75 98.75 100.00 98.75 98.75

100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.75

FL and KD for Point Cloud Classification: Supplementary Material

45

Table A8: Complete combined-pipeline results on the clinical task: mean student instance accuracy (%) over three seeds for every FL teacher (rows, with standalone accuracy in parentheses, ordered by it) and KD objective (columns). Columns: Van. (Vanilla KD [11]), Self (Self-Distillation [5]), Cos. (Cosine [29]), CRD (Contrastive [46]), DKD (Decoupled KD [59]), LMSE (Logit-MSE [2]), Feat. (Feature KD [40]), Att. (Attention [57]), RKD (Relational KD [33]), SP (Similarity-Preserving [47]). The first six columns keep a hard-label term; the last four do not. Standard deviations are in Table A9. Hard-label objectives Teacher (std.%)

Pure transfer objectives

Van. ↑ Self ↑ Cos. ↑ CRD ↑ DKD ↑ LMSE ↑ Feat. ↑ Att. ↑ RKD ↑ SP ↑

FedProx [22] (75.8) 99.6 99.6 100.0 99.6 Ditto [20] (71.2) 99.2 100.0 99.6 99.2 FedAvg [30] (69.6) 97.9 98.8 100.0 99.2 MOON [19] (69.2) 100.0 99.6 99.6 99.6 FedBN [23] (65.4) 99.6 99.6 99.6 99.6 FedMedian [54] (64.6) 99.6 99.6 99.6 99.2 FedDyn [1] (58.3) 99.6 99.2 99.6 99.6 FedNova [48] (50.8) 99.2 98.8 99.2 99.6 SCAFFOLD [16] (36.7) 99.6 99.2 99.6 99.2 FedAdagrad [38] (26.2) 98.8 99.2 100.0 99.6 FedAdam [38] (25.0) 99.2 99.6 99.6 100.0 FedAvgM [12] (25.0) 97.5 99.2 99.6 100.0 FedYogi [38] (25.0) 99.2 99.6 100.0 99.6

96.7 96.2 96.7 97.1 97.5 97.5 91.7 97.9 97.9 99.2 96.2 93.8 98.3

99.2 98.8 97.9 98.8 99.2 99.6 95.8 98.3 98.8 98.8 98.8 97.5 98.8

84.2 82.5 77.1 75.4 76.2 69.2 63.8 57.9 30.4 32.5 26.7 27.1 27.5

83.3 83.3 76.2 74.2 76.2 69.6 63.3 59.6 29.6 30.8 29.6 25.0 27.9

81.7 85.0 77.1 77.5 76.7 70.0 63.3 56.7 32.5 31.7 31.2 27.5 30.8

82.9 83.3 77.5 75.4 75.4 67.9 64.2 57.5 32.9 32.5 28.3 25.0 28.8

Table A9: Standard deviation (%, three seeds) of the combined-pipeline student instance accuracy reported in Table A8. Column abbreviations as in Table A8. The pure-transfer objectives show the larger spread, reflecting their dependence on the variable federated teacher.

Hard-label objectives

Pure transfer objectives

Teacher

Van. Self Cos. CRD DKD LMSE Feat.

Att. RKD

SP

FedProx [22] Ditto [20] FedAvg [30] MOON [19] FedBN [23] FedMedian [54] FedDyn [1] FedNova [48] SCAFFOLD [16] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]

0.72 0.72 0.72 0.00 0.72 0.72 0.72 1.44 0.72 1.25 0.72 1.25 0.72

6.41 0.72 4.51 13.25 5.73 5.20 13.13 10.10 3.15 1.91 1.91 0.00 1.44

9.04 1.91 6.61 8.32 5.64 6.88 13.25 7.81 1.91 1.25 3.82 0.00 2.17

0.72 0.00 0.00 0.72 0.72 0.72 0.72 0.00 0.72 1.44 0.72 1.44 0.72

0.00 0.72 0.00 0.72 0.72 0.72 0.72 0.72 0.72 0.00 0.72 0.72 0.00

0.72 0.72 0.72 0.72 0.72 0.72 0.72 0.72 0.72 0.72 0.00 0.00 0.72

0.72 1.25 1.44 1.44 1.25 1.25 2.60 1.91 0.72 1.44 1.25 2.50 1.44

0.72 1.25 1.44 0.00 1.44 0.72 1.91 0.72 1.25 0.00 1.25 0.00 1.25

5.64 3.31 6.88 9.21 7.50 4.39 13.17 8.78 3.15 0.00 1.91 2.60 2.17

7.53 2.50 5.91 8.20 6.41 3.31 12.58 8.32 0.00 1.91 2.17 2.17 4.02

46

Aizierjiang Aiersilan

Combined FL × KD accuracy — modelnet40 92

92

50

41

41

84

85

88

85

41

41

fednova

92

78

78

92

91

92

92

87

78

78

ditto

92

53

53

93

93

92

92

91

53

53

moon

92

43

43

92

93

92

92

91

44

43

fedbn

93

49

49

92

93

92

92

91

49

49

fedmedian

93

34

34

93

93

92

92

92

35

34

fedadagrad

92

6

6

92

92

92

92

88

6

6

fedyogi

93

4

4

93

93

92

92

92

4

4

fedadam

87

6

5

88

71

92

92

73

5

6

fedavgm

92

4

4

92

4

4

4

92

4

4

feddyn

92

67

67

92

90

92

92

82

67

67

scaffold

92

63

63

92

92

92

92

84

64

63

fedprox

92

70

70

92

92

92

92

86

70

70

fedavg

93

52

52

93

93

92

92

91

52

52

90

80

la

nil

va

re ion nt te at

tu

fea

e e lf se sin ms co it_ log

KD strategy

crd

d

dk

rkd

sp

70

60

50

40

Top-1 accuracy (%)

FL teacher

mean (%)

100

30

20

10 50

100

mean (%)

best: fedyogi+logit_mse = 92.8%

Fig. A10: Multi-seed combined-pipeline grid on ModelNet40. Student instance accuracy for each FL teacher (rows) and KD objective (columns), mean over three seeds, with per-column means in the bars above. The hard-label columns (vanilla, self, logit-MSE, cosine, CRD, DKD) stay high while the pure-transfer columns (feature, attention, RKD, SP) track the teacher; the best cell pairs the chance-level FedYogi teacher with Logit-MSE at 92.8%. Both axes follow the benchmark’s native order.

FL and KD for Point Cloud Classification: Supplementary Material

47

Table A10: Complete multi-seed combined-pipeline results on ModelNet40: mean student instance accuracy (%) over three seeds for every FL teacher (rows, with standalone accuracy in parentheses, ordered by it) and KD objective (columns). Columns: Van. (Vanilla KD [11]), Self (Self-Distillation [5]), Cos. (Cosine [29]), CRD (Contrastive [46]), DKD (Decoupled KD [59]), LMSE (Logit-MSE [2]), Feat. (Feature KD [40]), Att. (Attention [57]), RKD (Relational KD [33]), SP (Similarity-Preserving [47]). The first six columns keep a hard-label term; the last four do not. Bold marks the best value in each column. This is the multi-seed counterpart of the focused-round grid in Table 3 of the main paper; standard deviations are in Table A11. Hard-label objectives

Pure transfer objectives

Teacher (std.%)

Van. ↑ Self ↑ Cos. ↑ CRD ↑ DKD ↑ LMSE ↑ Feat. ↑ Att. ↑ RKD ↑ SP ↑

FedNova [48] (76.3) FedProx [22] (71.0) FedDyn [1] (66.0) SCAFFOLD [16] (64.3) Ditto [20] (61.4) FedAvg [30] (58.5) FedBN [23] (55.3) MOON [19] (45.8) FedMedian [54] (39.1) FedAdagrad [38] (6.3) FedAdam [38] (4.7) FedAvgM [12] (4.1) FedYogi [38] (4.1)

92.4 92.0 91.8 92.5 92.4 92.5 92.6 92.5 92.5 92.3 86.6 92.2 92.6

92.4 92.2 91.8 92.5 92.6 92.7 92.5 92.4 92.6 92.4 87.5 92.3 92.6

92.3 92.3 92.2 92.1 92.2 92.4 92.1 92.0 92.3 92.2 92.2 4.1 92.2

92.2 92.2 92.4 92.0 92.2 92.3 92.2 92.1 92.1 92.0 92.0 4.1 91.9

87.2 86.1 82.0 84.3 91.1 91.3 91.3 91.0 92.2 88.4 73.2 91.9 91.8

91.3 91.7 89.7 91.6 92.6 92.6 92.7 92.6 92.7 92.4 70.9 4.1 92.8

77.8 70.1 67.2 63.3 53.2 52.1 49.1 43.1 34.1 5.5 6.0 4.1 4.1

77.8 69.7 67.1 63.3 53.3 52.1 48.7 43.0 33.8 5.8 5.5 4.1 4.1

77.6 69.8 66.9 63.5 53.2 51.9 48.6 43.7 34.8 6.3 5.4 4.1 4.1

77.9 69.9 67.5 63.4 53.1 51.7 48.8 42.9 33.8 6.2 6.0 4.1 4.1

Table A11: Standard deviation (%, three seeds) of the ModelNet40 combined-pipeline student instance accuracy reported in Table A10. Column abbreviations as in Table A10. The pure-transfer objectives show the larger spread, reflecting their dependence on the variable federated teacher; the adaptive server-side teachers (notably FedAdam) additionally inflate a few hard-label cells, where a single seed failed to converge.

Hard-label objectives

Pure transfer objectives

Teacher

Van. Self Cos. CRD DKD LMSE Feat. Att. RKD

SP

FedNova [48] FedProx [22] FedDyn [1] SCAFFOLD [16] Ditto [20] FedAvg [30] FedBN [23] MOON [19] FedMedian [54] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]

0.08 0.27 0.37 0.11 0.15 0.21 0.16 0.06 0.19 0.47 9.88 0.02 0.09

0.61 0.35 2.50 4.40 3.32 1.62 1.15 7.09 5.70 1.32 2.96 0.00 0.00

0.14 0.15 0.26 0.04 0.12 0.17 0.10 0.06 0.19 0.42 8.86 0.17 0.10

0.17 0.14 0.25 0.14 0.31 0.05 0.06 0.22 0.02 0.15 0.05 0.00 0.29

0.06 0.55 0.13 0.13 0.55 0.08 0.17 2.10 1.15 0.13 2.09 0.63 0.10 0.47 0.15 0.05 0.32 0.10 0.16 0.66 0.10 0.14 0.33 0.20 0.16 0.10 0.04 0.19 6.00 0.41 0.08 29.01 37.48 0.00 0.23 0.00 0.21 0.23 0.08

0.79 0.70 2.12 4.55 3.44 1.72 1.20 7.21 5.32 1.67 3.05 0.00 0.02

0.62 0.79 2.36 4.60 3.58 1.18 1.10 7.51 6.00 1.31 2.06 0.00 0.05

0.44 0.58 3.10 4.29 3.64 1.36 1.20 6.77 5.31 1.39 1.37 0.00 0.05

48

Aizierjiang Aiersilan

Where does FL+KD beat its parts? — modelnet40 fedavg

+34

-6

-6

+34

+34

+34

+34

+33

-7

-7

fedprox

+21

-1

-1

+21

+21

+21

+21

+15

-1

-1

scaffold

+28

-1

-1

+28

+27

+28

+28

+20

-1

-1

feddyn

+26

+1

+1

+26

+24

+26

+26

+16

+1

+1

fedavgm

+88

+0

+0

+88

+0

+0

+0

+88

+0

+0

fedadam

+82

+1

+1

+83

+66

+87

+87

+68

+1

+1

fedyogi

+89

+0

+0

+89

+89

+88

+88

+88

+0

+0

fedadagrad

+86

-1

-1

+86

+86

+86

+86

+82

+0

-0

fedmedian

+53

-5

-5

+53

+54

+53

+53

+53

-4

-5

fedbn

+37

-6

-7

+37

+37

+37

+37

+36

-7

-6

moon

+47

-3

-3

+47

+47

+46

+46

+45

-2

-3

ditto

+31

-8

-8

+31

+31

+31

+31

+30

-8

-8

fednova

+16

+2

+1

+16

+15

+16

+16

+11

+1

+2

80

40

FL teacher

20

0

−20

−40

Synergy: combined − best single (pp)

60

sp

d rk

d dk

cr d

si ne

−80

co

lf

m se lo gi t_

se

io n nt te at

ur e at fe

va

ni

lla

−60

KD strategy

Fig. A11: No positive synergy on ModelNet40. For every FL teacher (rows) and KD objective (columns), the combined student’s instance accuracy minus the better of its two stand-alone components (the federated teacher alone and the student distilled from the centralized teacher), in percentage points (mean over three seeds). The six hard-label objectives stay near zero, while the four pure-transfer objectives fall well below their best component for the collapsed teachers, mirroring the clinical result of Figure A5.

FL and KD for Point Cloud Classification: Supplementary Material

49

Top 12 FL+KD combinations (modelnet40) 92.8±0.1

fedyogi+logit_mse

92.7±0.0

fedmedian+logit_mse

92.7±0.1

fedbn+logit_mse

fedavg+self

92.7±0.2

moon+logit_mse

92.6±0.2

92.6±0.1

fedyogi+self

fedbn+vanilla

92.6±0.2

fedmedian+self

92.6±0.2

92.6±0.1

fedavg+logit_mse

ditto+self

92.6±0.1

ditto+logit_mse

92.6±0.1

92.6±0.1

fedyogi+vanilla

92.0

92.2

92.4

92.6

92.8

93.0

93.2

Top-1 accuracy (%)

Fig. A12: The ModelNet40 leaderboard also rewards collapsed teachers. The twelve highest-accuracy FL+KD combinations on ModelNet40, each as mean±std over three seeds. Several pair a hard-label objective with a federated teacher frozen near the 4% chance level (for example FedYogi+Logit-MSE at 92.8%), so the ranking reflects the proxy labels rather than the quality of the federated teacher.

50

Aizierjiang Aiersilan

Efficiency frontier — modelnet40 Accuracy vs model size

Accuracy vs latency

80

Top-1 accuracy (%)

Top-1 accuracy (%)

80

60

40

FL + KD Federated Centralised Pareto frontier

20

0

2

3

4

Model size (MB)

5

60

40

FL + KD Federated Centralised Pareto frontier

20

0

115

120

125

130

135

Inference time (ms / batch)

Fig. A13: Accuracy versus model size and latency on ModelNet40. Test accuracy against model size (left) and inference latency (right), colored by family. The compact students cluster at the upper left near the teacher’s accuracy at 1.44 MB and at lower latency, the standalone federated models keep the teacher’s 5.65 MB footprint while spanning the full accuracy range, and the combined students inherit the compact size. As on the clinical task, these wall-clock latencies are averages on a single GPU and are subject to hardware-load variation, so the latency margin in this multi-seed run is narrower than the roughly twofold speed-up measured for the same compact student under the lighter-load focused round (Section 4.3 of the main paper). This is the ModelNet40 counterpart of Figure A1.

FL and KD for Point Cloud Classification: Supplementary Material

51

Table A12: Per-seed standalone FL test instance accuracy (%) on both datasets. Methods are ordered by mean ModelNet40 accuracy. Column headers are the random seeds. ModelNet40

D

Craniosynostosis

Method

7↑

42 ↑

123 ↑

7↑

42 ↑

123 ↑

Centralized

92.26

92.30

92.22

100.00

100.00

100.00

FedNova [48] FedProx [22] FedDyn [1] SCAFFOLD [16] Ditto [20] FedAvg [30] FedBN [23] MOON [19] FedMedian [54] FedAdagrad [38] FedAdam [38] FedAvgM [12] FedYogi [38]

75.16 71.68 66.94 60.41 59.64 58.79 57.90 42.91 44.04 7.54 4.05 4.05 4.05

75.73 71.39 64.55 67.91 66.05 58.31 53.28 40.24 40.60 5.75 4.25 4.05 4.05

78.08 70.06 66.53 64.47 58.39 58.43 54.66 54.38 32.74 5.55 5.83 4.05 4.05

45.00 80.00 66.25 37.50 60.00 67.50 68.75 60.00 63.75 25.00 25.00 25.00 25.00

47.50 83.75 58.75 35.00 75.00 72.50 67.50 77.50 72.50 25.00 25.00 25.00 25.00

60.00 63.75 50.00 37.50 78.75 68.75 60.00 70.00 57.50 28.75 25.00 25.00 25.00

Per-Seed Results

For full transparency we report the individual test instance accuracy of every run, before averaging. Table A12 gives the per-seed standalone FL results on both datasets and Table A7 the per-seed standalone KD results, while Tables A13 and A14 list the per-seed accuracy of all 130 combined configurations on the clinical task and on ModelNet40 respectively, complementing the mean and standard-deviation matrices in Tables A8 and A9 (clinical) and Tables A10 and A11 (ModelNet40). The per-seed spread is small for the converged methods; the larger spread of the feature-based combined runs reflects their dependence on the variable federated teacher rather than seed noise, as is also presented in the analysis in Section 4.4 of the main paper.

52

Aizierjiang Aiersilan

Table A13: Per-seed combined-pipeline test instance accuracy (%) on the clinical task, for every FL teacher (with its standalone accuracy) crossed with every KD objective. Column headers are the random seeds. The six hard-label objectives recover the student regardless of teacher quality, whereas the four pure-transfer objectives (Feature KD, Attention Transfer, Relational KD, Similarity-Preserving) track the teacher and fall toward the 25% chance level when it collapses. For reference, the ten KD objectives are Vanilla KD [11], Self-Distillation [5], Cosine Similarity [29], Contrastive Representation Distillation (CRD) [46], Decoupled KD [59], Logit-MSE [2], Feature KD [40], Attention Transfer [57], Relational KD [33], and Similarity-Preserving distillation [47]. FL Teacher (standalone)

KD Objective

7↑

42 ↑

123 ↑

FedProx [22] (75.8%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 100.00 100.00 100.00 97.50 98.75 83.75 85.00 82.50 87.50

98.75 98.75 100.00 98.75 96.25 98.75 90.00 88.75 88.75 88.75

100.00 100.00 100.00 100.00 96.25 100.00 78.75 76.25 73.75 72.50

Ditto [20] (71.2%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 100.00 100.00 100.00 95.00 100.00 80.00 82.50 82.50 81.25

98.75 100.00 98.75 98.75 97.50 97.50 81.25 83.75 85.00 83.75

98.75 100.00 100.00 98.75 96.25 98.75 86.25 83.75 87.50 85.00

FedAvg [30] (69.6%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

98.75 98.75 100.00 100.00 97.50 98.75 73.75 75.00 75.00 75.00

97.50 98.75 100.00 98.75 97.50 96.25 85.00 81.25 83.75 85.00

97.50 98.75 100.00 98.75 95.00 98.75 72.50 72.50 72.50 72.50

MOON [19] (69.2%)

Vanilla KD Self-Distillation Cosine Similarity

100.00 98.75 100.00

100.00 100.00 98.75

100.00 100.00 100.00

continued on next page

FL and KD for Point Cloud Classification: Supplementary Material

53

Table A13, continued FL Teacher (standalone)

KD Objective

7↑

42 ↑

123 ↑

Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 98.75 98.75 65.00 60.00 68.75 66.25

98.75 96.25 98.75 78.75 76.25 78.75 77.50

100.00 96.25 98.75 82.50 86.25 85.00 82.50

FedBN [23] (65.4%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 100.00 100.00 100.00 98.75 100.00 76.25 75.00 75.00 75.00

98.75 98.75 98.75 98.75 96.25 97.50 83.75 82.50 83.75 81.25

100.00 100.00 100.00 100.00 97.50 100.00 68.75 71.25 71.25 70.00

FedMedian [54] (64.6%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 98.75 100.00 100.00 98.75 100.00 73.75 73.75 73.75 75.00

100.00 100.00 98.75 98.75 97.50 100.00 68.75 71.25 68.75 67.50

98.75 100.00 100.00 98.75 96.25 98.75 65.00 63.75 67.50 61.25

FedDyn [1] (58.3%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 100.00 100.00 100.00 93.75 96.25 76.25 76.25 75.00 76.25

98.75 98.75 98.75 98.75 88.75 93.75 65.00 63.75 65.00 66.25

100.00 98.75 100.00 100.00 92.50 97.50 50.00 50.00 50.00 50.00

FedNova [48] (50.8%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD

100.00 98.75 98.75 100.00 100.00

97.50 98.75 98.75 98.75 97.50

100.00 98.75 100.00 100.00 96.25

continued on next page

54

Aizierjiang Aiersilan

Table A13, continued FL Teacher (standalone)

KD Objective

7↑

42 ↑

123 ↑

Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

98.75 48.75 48.75 47.50 48.75

97.50 58.75 61.25 58.75 60.00

98.75 66.25 68.75 63.75 63.75

SCAFFOLD [16] (36.7%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 98.75 100.00 98.75 97.50 98.75 27.50 26.25 32.50 32.50

98.75 98.75 98.75 98.75 97.50 97.50 33.75 32.50 32.50 35.00

100.00 100.00 100.00 100.00 98.75 100.00 30.00 30.00 32.50 31.25

FedAdagrad [38] (26.2%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

98.75 100.00 100.00 100.00 100.00 98.75 32.50 32.50 33.75 32.50

97.50 97.50 100.00 98.75 100.00 98.75 32.50 31.25 30.00 31.25

100.00 100.00 100.00 100.00 97.50 98.75 32.50 28.75 31.25 33.75

FedAdam [38] (25.0%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

100.00 100.00 100.00 100.00 95.00 100.00 25.00 27.50 32.50 27.50

98.75 98.75 98.75 100.00 97.50 97.50 26.25 30.00 32.50 25.00

98.75 100.00 100.00 100.00 96.25 98.75 28.75 31.25 28.75 32.50

FedAvgM [12] (25.0%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD

98.75 100.00 100.00 100.00 96.25 97.50 30.00

96.25 97.50 98.75 100.00 93.75 97.50 25.00

97.50 100.00 100.00 100.00 91.25 97.50 26.25

continued on next page

FL and KD for Point Cloud Classification: Supplementary Material

55

Table A13, continued FL Teacher (standalone)

FedYogi [38] (25.0%)

KD Objective

7↑

42 ↑

123 ↑

Attention Transfer Relational KD Similarity-Preserving

25.00 30.00 25.00

25.00 26.25 25.00

25.00 26.25 25.00

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

98.75 100.00 100.00 100.00 97.50 98.75 26.25 28.75 26.25 31.25

98.75 98.75 100.00 98.75 97.50 97.50 30.00 28.75 33.75 27.50

100.00 100.00 100.00 100.00 100.00 100.00 26.25 26.25 32.50 27.50

56

Aizierjiang Aiersilan

Table A14: Per-seed combined-pipeline test instance accuracy (%) on ModelNet40, for every FL teacher (with its standalone accuracy) crossed with every KD objective. Column headers are the random seeds. As on the clinical task, the six hard-label objectives recover the student regardless of teacher quality, whereas the four pure-transfer objectives (Feature KD, Attention Transfer, Relational KD, Similarity-Preserving) track the teacher and fall toward the chance level (about 4%) when it collapses; the per-seed values also expose the single seeds in which a few hard-label cells under the near-degenerate FedAdam teacher failed to converge. This is the multi-seed counterpart of the focusedround grid in Table 3 of the main paper and of the per-cell summary in Tables A10 and A11. For reference, the ten KD objectives are Vanilla KD [11], Self-Distillation [5], Cosine Similarity [29], Contrastive Representation Distillation (CRD) [46], Decoupled KD [59], Logit-MSE [2], Feature KD [40], Attention Transfer [57], Relational KD [33], and Similarity-Preserving distillation [47]. FL Teacher (standalone)

KD Objective

7↑

42 ↑

123 ↑

FedNova [48] (76.3%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.46 92.30 92.14 92.26 87.12 91.29 77.07 77.07 77.07 77.59

92.38 92.42 92.46 92.14 86.67 91.25 77.80 78.24 77.67 77.51

92.30 92.59 92.38 92.22 87.76 91.49 78.65 78.00 77.92 78.61

FedProx [22] (71.0%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.22 92.30 92.14 92.14 86.75 91.77 70.26 69.37 70.02 69.81

91.73 92.02 92.30 92.38 85.70 91.69 70.75 70.54 70.22 70.30

92.18 92.22 92.42 92.18 85.94 91.61 69.37 69.04 69.12 69.61

FedDyn [1] (66.0%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.22 92.10 92.38 92.54 84.40 91.05 67.46 67.26 67.02 67.34

91.53 91.69 91.90 92.22 80.47 89.26 64.95 64.63 63.70 65.07

91.65 91.61 92.26 92.46 81.16 88.90 69.17 69.33 69.89 70.06

continued on next page

FL and KD for Point Cloud Classification: Supplementary Material Table A14, continued FL Teacher (standalone)

KD Objective

7↑

42 ↑

123 ↑

SCAFFOLD [16] (64.3%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.42 92.42 92.06 91.86 85.33 92.18 59.04 59.12 59.68 59.36

92.38 92.46 92.06 91.90 85.66 91.61 68.11 68.23 68.15 68.07

92.59 92.50 92.30 92.10 81.89 90.92 62.88 62.60 62.76 62.64

Ditto [20] (61.4%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.26 92.71 91.90 92.26 90.68 92.46 50.12 49.92 49.84 50.00

92.54 92.63 92.50 92.14 91.61 92.75 56.93 57.05 57.05 56.60

92.30 92.46 92.10 92.06 91.00 92.54 52.67 52.88 52.59 52.76

FedAvg [30] (58.5%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.79 92.79 92.38 92.30 91.69 92.50 50.69 51.50 50.89 50.24

92.38 92.46 92.38 92.38 91.25 92.59 51.58 51.26 51.34 51.42

92.46 92.71 92.46 92.30 91.09 92.71 54.01 53.40 53.44 53.44

FedBN [23] (55.3%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.63 92.34 92.10 92.06 92.06 92.67 50.41 49.92 49.96 50.08

92.79 92.46 92.18 92.18 90.92 92.75 48.06 47.73 47.65 47.81

92.46 92.54 92.06 92.38 90.92 92.54 48.78 48.58 48.26 48.58

MOON [19] (45.8%)

Vanilla KD

92.50

92.42

92.54

continued on next page

57

58

Aizierjiang Aiersilan Table A14, continued FL Teacher (standalone)

KD Objective

7↑

42 ↑

123 ↑

Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.38 92.30 91.98 90.68 92.87 38.90 38.49 39.10 38.57

92.42 91.94 91.98 91.33 92.50 39.06 38.90 40.48 39.14

92.50 91.90 92.22 90.92 92.54 51.46 51.70 51.46 51.13

FedMedian [54] (39.1%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.42 92.46 92.30 91.94 92.06 92.71 39.71 40.11 40.56 39.79

92.42 92.54 92.26 92.22 92.14 92.75 33.43 32.98 33.55 33.23

92.75 92.83 92.26 92.22 92.26 92.79 29.13 28.20 30.15 28.44

FedAdagrad [38] (6.3%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.42 92.46 92.18 92.18 91.37 92.46 7.33 7.25 7.86 7.66

92.63 92.71 92.38 91.94 92.26 92.75 4.05 4.86 5.75 5.67

91.73 91.90 92.10 91.82 81.44 91.94 5.19 5.15 5.23 5.15

FedAdam [38] (4.7%)

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.26 92.75 92.18 92.06 91.53 92.91 4.05 4.09 4.70 4.09

75.20 77.31 92.10 91.90 39.75 27.67 4.46 4.46 4.50 4.50

92.38 92.54 92.18 92.02 88.29 92.26 9.52 7.82 6.97 9.40

FedAvgM [12] (4.0%)

Vanilla KD Self-Distillation Cosine Similarity

92.22 92.18 4.05

92.26 92.26 4.05

92.26 92.50 4.05

continued on next page

FL and KD for Point Cloud Classification: Supplementary Material Table A14, continued FL Teacher (standalone)

FedYogi [38] (4.0%)

KD Objective

7↑

42 ↑

123 ↑

Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

4.05 92.14 4.05 4.05 4.05 4.05 4.05

4.05 91.69 4.05 4.05 4.05 4.05 4.05

4.05 91.98 4.05 4.05 4.05 4.05 4.05

Vanilla KD Self-Distillation Cosine Similarity Contrastive (CRD) Decoupled KD Logit-MSE Feature KD Attention Transfer Relational KD Similarity-Preserving

92.46 92.54 92.38 91.98 91.61 92.75 4.09 4.13 4.05 4.05

92.63 92.75 92.42 92.10 92.06 92.87 4.05 4.05 4.05 4.05

92.63 92.63 91.90 91.69 91.77 92.91 4.05 4.05 4.13 4.05

59

60

E

Aizierjiang Aiersilan

Generalization to Additional Datasets

The dataset-agnostic loader described in Appendix A lets the same standalone protocols run on point-cloud sources beyond the two datasets of the main multiseed benchmark. To show that the framework and its conclusions are not specific to ModelNet40 and the clinical task, we report standalone federated-learning and knowledge-distillation results on two further datasets: ModelNet10 [52], a tenclass subset of the ModelNet shapes, and OmniObject3D [50], a large-vocabulary collection of real-scanned objects. These auxiliary runs reuse the identical training configuration, the same compact student, and the same deterministic non-IID label-skew partition; they lie outside the 504-run count of the main paper and are provided only to illustrate the framework’s reach. None of these results appears elsewhere in this supplement. Across both datasets the same two patterns recur that the main paper documents on ModelNet40 and the clinical task: most of the server-side federated optimizers collapse toward the chance level while the drift-correction methods learn, and distillation transfers a strong centralized teacher into the compact student at a small accuracy cost. Federated convergence. Figure A14 plots the per-round federated accuracy on both datasets and Figure A15 the corresponding best-accuracy ranking; the pattern matches Figure A7. A leading group of drift-correction and robustaggregation methods climbs steadily over the twenty rounds, whereas the serverside optimizers lag far behind: FedAvgM and FedYogi sit at the chance level on both datasets, FedAdam barely rises above it, and FedAdagrad collapses on OmniObject3D while reaching only about 20% on ModelNet10. On ModelNet10 the strongest teacher (FedNova) reaches about 74% instance accuracy against a 95.4% centralized reference, and on OmniObject3D the strongest (FedProx) reaches about 58% against a 79.5% centralized reference, so the federated accuracy gap widens with task difficulty, exactly as on the two main datasets. Standalone distillation. Figure A16 reports the ten distillation objectives distilling the centralized teacher into the compact student. On ModelNet10 every objective recovers the teacher almost exactly, landing between 95.1% and 95.7% against the 95.4% centralized accuracy. On the much harder OmniObject3D, eight of the ten objectives cluster between 74.4% and 75.2%, within roughly five points of the 79.5% teacher, while Decoupled KD trails at 70.9% and Logit-MSE collapses to 3.9%. The Logit-MSE failure echoes its weak standing on ModelNet40 (Section 4.3 of the main paper) and confirms that the raw logit-matching objective is the least robust of the ten as the label space grows. Compression. Because the same compact student is distilled on every dataset, the compression is dataset-independent: the student keeps its ≈1.4 MB footprint against the ≈5.6 MB teacher and runs at roughly half the teacher’s inference latency, just as on ModelNet40 and the clinical task, and on the easier ModelNet10 it gives up almost no accuracy for that reduction.

FL and KD for Point Cloud Classification: Supplementary Material

FL convergence — modelnet10

50 40 30 20

Test instance accuracy (%)

Test instance accuracy (%)

60

FL convergence — omni_object3d

60

fednova scaffold feddyn fedprox fedmedian fedbn fedavg ditto moon fedadagrad fedadam fedyogi fedavgm

70

61

fedprox moon ditto fedbn fedavg feddyn fednova fedmedian scaffold fedadagrad fedyogi fedadam fedavgm

50

40

30

20

10

10 0 2.5

5.0

7.5

10.0

12.5

15.0

17.5

2.5

20.0

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Communication round

Communication round

(a) ModelNet10

(b) OmniObject3D

Fig. A14: Federated convergence on two additional datasets. Per-round test instance accuracy of all thirteen FL algorithms under the non-IID label-skew partition. (a) On ModelNet10, FedNova leads near 74%, a broad middle band follows, and the four server-side optimizers trail at the bottom, with FedAvgM and FedYogi at the 10% chance level. (b) On the large-vocabulary OmniObject3D, FedProx leads near 58% and all four server-side optimizers collapse to near the chance level. The trends reproduce those of Figure A7 on the two main datasets.

Federated learning accuracy — omni_object3d

Federated learning accuracy — modelnet10

40

37.0

20.3

20

14.6

11.0

11.0

57.0

56.6

55.6

55.4

52.2 46.4

45.4

40 30 20 13.0

10 1.2

gi dy o

da d

fe

1.1

0.8

0.8

fe

n

ld fo

ed ia

af sc

dm

n

va dn o

g

fe

fe

da v

dd y

fe

fe

o

db n fe

n oo

di tt

m

vg da

fe

fe

dy

og

i

m

m

ad

da

gr

da

da

fe

o

n oo m

da fe

vg

tt

da fe

di

n

n ia

db

ed

fe

fe

dm

yn

ro x

dd

dp

fe

fe

a

ld

ov

fo af sc

dn fe

(a) ModelNet10

dp ro x

0

0

gm

45.8

58.1

50

da v

46.6

60

fe

56.2

d

56.6

am

65.8

da d

72.6

fe

73.1

60

70

ra

74.5

Test instance accuracy (%)

80

Centralised (79.5%)

80

fe

Test instance accuracy (%)

Centralised (95.4%)

ag

100

(b) OmniObject3D

Fig. A15: Federated strategy accuracy on two additional datasets (best per run, mean±std over three seeds). Ranked bar-chart view of the convergence endpoints in Figure A14; the dashed line marks the centralized reference. (a) On ModelNet10 the order runs FedNova (74.5%), SCAFFOLD, FedDyn, FedProx, then a middle band, down to the four server-side optimizers (11 to 20%). (b) On OmniObject3D the order runs FedProx (58.1%), MOON, Ditto, then a middle band, down to SCAFFOLD (13%) and the four server-side optimizers near zero.

62

Aizierjiang Aiersilan

Knowledge distillation accuracy — omni_object3d

Knowledge distillation accuracy — modelnet10 95.7

95.6

95.4

95.4

95.4

95.4

95.3

95.3

95.3

95.1

Centralised (95.4%)

80

60

40

20

80

Test instance accuracy (%)

Test instance accuracy (%)

100

Centralised (79.5%) 75.2

75.1

74.9

74.8

74.8

74.7

74.6

74.4 70.9

70 60 50 40 30 20 10

3.9

d

se m t_

gi lo

lf se

dk

sp

n

lla ni va

ne

io

at

te nt

ur e

si

at

co

fe

d rk

a

se m

ill

gi t_ lo

cr d

va n

ur e at fe

n

sp

lf

io

se

nt

at

te

ne co

si

d

d rk

dk

(a) ModelNet10

cr d

0

0

(b) OmniObject3D

Fig. A16: Standalone knowledge distillation on two additional datasets. Test instance accuracy of the ten KD objectives distilling the centralized teacher into the compact student; the dashed line marks the centralized reference. (a) On ModelNet10 all ten objectives reach the 95% ceiling. (b) On OmniObject3D most objectives land near 75%, Decoupled KD trails at 70.9%, and Logit-MSE collapses to 3.9%, the same fragility it shows on ModelNet40.

Related documents

• CACTUS: Mask-Guided Semantic Clean-Label Backdoors in Decentralized Federated Learning#924332
• CACTUS: Mask-Guided Semantic Clean-Label Backdoors in Decentralized Federated Learning#920101
• Discovering Collaboration from Novelty: Random Network Distillation for Clustered Federated Learning#321835
• Beyond Weights and Gradients: A Taxonomy of Federated Learning Messages#280212
• DAGLFNet: Deep Feature Attention Guided Global and Local Feature Fusion for Pseudo-Image Point Cloud Segmentation#972969