JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
1
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
arXiv:2605.05959v1 [cs.AI] 7 May 2026
Xinghao Wu, Jianwei Niu, Fellow, IEEE, Guogang Zhu, Xuefeng Liu, Shaojie Tang and Jiayuan Zhang
Abstract—Heterogeneous federated learning (HtFL) aims to enable collaboration among clients that differ in both data distributions and model architectures. Prototype-based methods, which communicate class-level feature centers (prototypes) instead of full model parameters, have recently shown strong potential for HtFL. Existing prototype-based HtFL methods typically reuse the MSE-based or cosine-based alignment mechanism developed for homogeneous FL when aligning client-specific representations with global prototypes. These approaches are essentially coordinate alignment, where representations of clients are forced to match the global prototypes in the embedding space in an element-wise manner. Such alignment implicitly assumes that all clients should map their representations into the feature subspace1 defined by the global prototypes. This assumption is reasonable in homogeneous FL, where all clients share the same feature extractor. However, it becomes problematic in HtFL, since heterogeneous feature extractors naturally induce client-specific feature subspaces, and forcing all clients to optimize within a single global subspace unnecessarily suppresses their learning capacity. We observe that coordinate alignment implicitly couples two distinct objectives: aligning inter-class semantic structure, which is directly beneficial for classification, and enforcing a shared feature basis, which is unnecessary and even harmful under model heterogeneity. Building on this insight, we design FedSAF, a Federated Structural Alignment Framework for HtFL, which shifts the alignment objective from absolute coordinates to inter-class relational structure. We demonstrate both theoretically and empirically that structural alignment consistently outperforms coordinate alignment in heterogeneous settings. Moreover, our framework is orthogonal to the choice of global prototype construction. Experiments on multiple benchmarks show that, even with vanilla prototype aggregation, our structural alignment already outperforms state-of-the-art prototype-based HtFL methods by up to 3.52%. Index Terms—Heterogeneous Federated Learning, Prototype Alignment, Structural Alignment. Xinghao Wu, Guogang Zhu, and Jiayuan Zhang are with the State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100091, China (e-mail: [email protected], buaa [email protected], [email protected]). Jianwei Niu and Xuefeng Liu are with the State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100091, China, and also with Zhongguancun Laboratory, Beijing 100194, China (e-mail: [email protected], liu [email protected]). Shaojie Tang is with the Center for AI Business Innovation, Department of Management Science and Systems, School of Management, University at Buffalo, NY 14260 USA (e-mail: [email protected]). Corresponding author: Xuefeng Liu. 1 In this paper, we use feature subspace in an informal sense to denote a (typically low-dimensional) representation space induced by a model; it is not required to be a linear subspace in the strict algebraic sense. We use feature space and feature subspace interchangeably unless otherwise specified.
I. I NTRODUCTION Federated Learning (FL) [1], [2] enables clients to collaboratively train models without exposing their raw data. A primary challenge in FL is data heterogeneity, where the data distributions across clients are non-independent and identically distributed (non-IID). Numerous studies have sought to address this issue: some focus on improving the robustness of a single global model [3]–[7], while others design personalized models [8]–[11] tailored to local client distributions. However, these approaches typically assume that all clients adopt the same model architecture, which is often impractical in realworld scenarios. For instance, in cross-silo FL, institutions often require customized architectures and may be unable to disclose model details due to intellectual property (IP) constraints. In cross-device FL, resource heterogeneity leads to diverse feasible model sizes across devices. To overcome this limitation, Heterogeneous Federated Learning (HtFL) [12] has recently emerged as a promising direction, enabling collaboration across clients with both data and model heterogeneity. The key challenge is how to extract global knowledge and enable mutual assistance among clients when direct parameter aggregation is infeasible. Early solutions rely on public datasets and apply knowledge distillation to transfer knowledge [13]–[15]. However, acquiring suitable public datasets is difficult in practice, and domain mismatch often hinders performance. Other approaches introduce auxiliary models or modules as carriers of global knowledge [16], [17], but these incur substantial computation and communication costs and may alter local model structures. A particularly appealing line of work is prototype-based HtFL [18]–[20], where clients exchange class prototypes (feature centers). This strategy substantially reduces communication and computation overhead compared to full-model distillation or auxiliary models, making it highly practical. Prototype-based HtFL faces two fundamental challenges: (1) how to construct global prototypes that effectively capture shared characteristics across clients, and (2) how to align local representations with global prototypes so that global knowledge is transferred to local models. Most existing works primarily focus on the first challenge [19]–[21], while the second has received considerably less attention. Current solutions typically reuse alignment objectives from homogeneous FL, such as L2 distance (e.g., MSE loss) [22], [23] or cosine similarity (e.g., contrastive loss) [24]–[26]. These approaches are essentially coordinate alignment, where client representations
Effective Dimensionality
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
CIFAR-10
60
CIFAR-100
Class 0 Class 1 Class 2
250
50 40
200
30
150
20
100
10
50
0
2
Homo-Shared
Homo-Local
Hetero-Local
0
Client 1 Subspace Client 2 Subspace
Global Subspace Coordinate Alignment
4
Homo-Shared
Homo-Local
Hetero-Local
are forced to match the global prototypes in the embedding space in an element-wise manner. This form of alignment implicitly couples two distinct objectives: (i) aligning the inter-class semantic structure encoded by prototypes, and (ii) mapping all client representations into the feature subspace defined by the global prototypes. In homogeneous FL, where all clients share a single global feature extractor and their representations naturally lie in the same feature space, both objectives are jointly satisfied at negligible cost. In HtFL, however, this assumption no longer holds. As demonstrated in Fig. 1, we compare the feature space discrepancy across clients under three settings: (1) all clients share the same feature extractor; (2) all clients use the same backbone architecture but maintain independent parameters; and (3) clients employ different feature extractors. We stack prototypes from all clients and perform singular value decomposition (SVD) to compute the effective dimensionality. A higher effective dimensionality indicates larger discrepancies among client feature spaces. Compared with homogeneous FL (Homo-Shared and Homo-Local), the model-heterogeneous scenario (Hetero-Local) exhibits a substantially higher effective dimensionality, suggesting that in HtFL, clients naturally tend to optimize within their own feature subspaces. In such a scenario, forcing all clients to optimize within a single global feature subspace can suppress model learning capacity and limit the benefits of collaboration. The above analysis suggests that only the first objective, aligning inter-class semantic structure, is essential for effective knowledge transfer, whereas enforcing a shared feature basis is unnecessary and potentially harmful in HtFL. Building on this insight, we propose FedSAF, a Federated Structural Alignment Framework tailored to HtFL. Unlike coordinate alignment, which couples semantic structure transfer with feature-space enforcement, our framework encourages clients to preserve only the inter-class relational structure conveyed by global prototypes, which is directly tied to classification performance, while allowing each client to optimize within its own feature space. A toy example illustrating the difference between the two paradigms is shown in Fig. 2. Different colors represent prototypes from different clients, and different markers correspond to different classes. In Fig. 2(a), each
Client 1 Subspace Client 2 Subspace Global Subspace
Global Structure Target Structure Structure Alignment
2
2
4
2
2
Z 0
Z 0
2
2
4
Fig. 1. Comparison of effective dimensionality in homogeneous and heterogeneous FL. We evaluate the feature space discrepancy across three settings: (1) Homo-Shared: clients share both identical model architectures and parameters; (2) Homo-Local: clients share the same architecture but maintain independent parameters; (3) Hetero-Local: clients employ diverse model architectures. The significantly higher effective dimensionality in HtFL suggests that heterogeneous models naturally induce distinct feature subspaces, making rigid coordinate-level alignment less appropriate as it may suppress model individuality.
Class 0 Class 1 Class 2
4
6
4
2
Y
0
2
4
6
6
0
2
4
2
4
6
6
4
2
X
Y
(a) Coordinate Alignment
0
4
6
6
4
0
2
4
6
X
(b) Structural Alignment
Fig. 2. Conceptual comparison between Coordinate Alignment and Structural Alignment in HtFL. (a) Coordinate alignment enforces pointwise matching between client representations and global prototypes in a shared feature space, which may suppress model individuality. (b) Our proposed structural alignment encourages clients to share the semantic structure encoded by interclass relations while allowing each client to optimize within its own feature subspace Cifar10, label shift (88.06%) Communication efficiency Cifar100, label shift (1.69MB) (44.26%) 88.1% 1.7MB
44.3% 86.4%
Training efficiency (35.131s)
1.8MB 1.8MB
36.1s
Convergence (38) 38.0
Fairness 6.1%
37.1s
1.9MB 38.1s
70.4
102.8
9.7%
7.9%
135.2 11.4% 38.4%
(6.12%)
Cross-device scenario (43.53%)
83.2%
61% 42.6%
42.1% 43.1%
40.3%
59% 56%
41.7%
13.2% 51.5% 36.7% 81.7% 36.3%41.0%41.1%
39.0%
43.5%
84.8%
1.9MB81.6%40.9% 39.0s 52% 167.6 11.0%
37.6%
40.1% 41.8%
PACS, feature shift (61.56%)
43.4%
35.1s
54% 12.1% 53.0%
13.1%
54.5%
14.1%
56.0%
83.4%
42.3%
DomainNet, feature shift 15.1% (15.14%)
AG57.5% NEWS, label shift (57.49%)
85.1% 86.8%
43.5%
88.6%
44.6% 44.2%
41.6%
MSE Cosine Ours-GCSA Ours-RCSA
Amazon Review, feature shift (88.56%)
45.8% 45.2%
Model hetero, HtM10 Shakespeare, real world (41.61%) (45.75%) Model hetero, HtFE4 (45.19%)
Fig. 3. Comprehensive comparison between coordinate alignment (MSE, Cosine) and structural alignment (Ours-GCSA, Ours-RCSA) across multiple evaluation dimensions, including various datasets, non-IID scenarios, model heterogeneity levels, and efficiency metrics. Our structural alignment consistently outperforms coordinate alignment across nearly all dimensions.
client’s local prototypes are forced to align point-to-point with the global prototypes in a shared space (blue plane). In Fig. 2(b), each client optimizes within its own space (red and green planes), while only the inter-class structure defined by the global prototypes (blue triangle) is preserved. To realize this framework, we instantiate it with two effective structural alignment losses. As partially summarized in Fig. 3, our structural alignment outperforms coordinate alignment across nearly all evaluation dimensions, including diverse datasets, non-IID scenarios (label shift, feature shift, and real-world), multiple modalities (vision and language), varying levels of model heterogeneity, cross-device scalability, and training efficiency. Moreover, our alignment framework is orthogonal to the choice of global prototype construction. Even
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
with vanilla prototype aggregation, our method significantly outperforms state-of-the-art prototype-based HtFL methods, and can be further combined with advanced prototype construction strategies to achieve additional gains. Our contributions can be summarized as follows: • We identify a critical yet overlooked limitation in existing prototype-based HtFL methods: the reliance on coordinate alignment. We demonstrate, both theoretically and empirically, that forcing heterogeneous clients to align with a single global feature space suppresses the individuality of client-specific feature spaces. • We propose a Structural Alignment Framework that shifts the alignment objective from absolute coordinates to relational geometry. We provide two concrete instantiations, GCSA and RCSA, which effectively transfer global semantic knowledge without compromising the unique feature spaces of local models. • We conduct extensive evaluations across diverse non-IID settings, modalities, and model heterogeneity levels. The results verify that our method significantly outperforms existing baselines. Furthermore, our approach serves as a plug-and-play module that is orthogonal to prototype construction methods, offering a flexible solution for enhancing various HtFL systems. II. R ELATED W ORK Heterogeneous Federated Learning. Heterogeneous federated learning (HtFL) enables client collaboration under both data and model heterogeneity, where a key challenge is extracting and transferring global knowledge without direct model aggregation [12], [27]. Unlike homogeneous FL that assumes identical model structures, HtFL allows clients to maintain personalized architectures suited to their computational resources and task requirements. Existing HtFL methods mainly fall into five categories. (1) Sub-model extraction methods [28]–[30] allow clients to train smaller sub-models extracted from a larger global model, but assume clients train portions of a shared architecture rather than fully independent models. (2) Knowledge distillation methods [13], [17], [31]–[33] transfer knowledge through soft labels or intermediate representations on a shared dataset, often incurring substantial computational overhead or facing challenges when suitable proxy data is unavailable. (3) Mutual learning methods [34]–[36] introduce auxiliary models for bidirectional knowledge transfer, but bring additional computation and communication costs due to auxiliary model training. (4) Partial parameter sharing methods [37]–[39] assume part of client models remain homogeneous for aggregation, which limits architectural flexibility and may expose partial model structures. In addition to the above four categories, prototype-based methods [18]–[21], [40], [41] have emerged as an appealing alternative due to their lower computational and communication costs, as clients only need to share class-wise feature centers without requiring public data or auxiliary models. Existing prototype-based HtFL methods primarily differ in how they construct global prototypes. FedProto [18] directly
3
aggregates local prototypes uploaded by clients via weighted averaging. AlignFed [21] pre-defines a set of global prototypes as uniformly distributed points on a hypersphere, providing a fixed geometric anchor for alignment. FedTGP [19] and FedSA [41] randomly initialize a set of learnable global prototypes and iteratively update them using local prototypes from clients. FedKTL [40] and FedTSP [20] take a different approach by extracting global prototypes from pre-trained image generation models or pre-trained language models, leveraging external semantic priors to enrich prototype quality. Despite these advances in prototype construction, the critical question of how to effectively align local representations with global prototypes under heterogeneous feature spaces remains largely underexplored. Most existing methods simply reuse coordinate-level alignment losses (e.g., MSE or cosine similarity) inherited from homogeneous FL, without accounting for the fundamental mismatch between client-specific feature spaces. Our work is orthogonal to these prototype construction strategies and focuses specifically on this overlooked alignment problem. III. M ETHODOLOGY A. Preliminary Heterogeneous Federated Learning. We consider an HtFL system with N clients coordinated by a central server. Each client aims to train heterogeneous personalized models {wi }N i=1 to perform well on its own data distribution Di , where Di ̸= Dj for any i ̸= j. Following FedProto, each wi is decomposed into a feature extractor fi : X → Rd , parameterized by θi , and a classifier hi : Rd → RC , parameterized by ϕi . Here, X denotes the input space (e.g., raw images). d is the feature dimension, and C is the number of classes. The local learning objective on client i is to minimize its empirical loss LD over the local distribution Di . The global HtFL objective can thus be written as: min
{θi ,ϕi }N i=1
N X 1 i=1
N
LD (θi , ϕi ; Di ).
(1)
Prototype-Based HtFL. Prototype-based methods maintain a server-side set of class prototypes Pg = {Pgc ∈ Rd }C c=1 , where Pgc denotes the global semantic representation of class c. During local training, client i uses Pg as class-level guidance by aligning its representations z = fi (x; θi ) with the corresponding prototype Pgy . A typical local objective is Li = LD (θi , ϕi ; x, y)+λR(fi (x; θi ), Pgy ), where x, y ∼ Ditrain , (2) where R denotes an alignment loss, Ditrain denotes the local training set, and λ balances supervised learning and prototype guidance. While existing work largely focuses on improving how Pg is constructed, our work addresses a complementary and equally important question: how to design the alignment function R so that it effectively transfers prototype knowledge to clients with heterogeneous feature spaces. Global Prototype Construction. Existing prototype-based HtFL methods mainly differ in how they obtain Pg . A canonical baseline is aggregation (e.g., FedProto): each client forms
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
4
local class prototypes Pic for class c, and the server aggregates them as PN c c i=1 ωi Pi Pg = P , (3) N i=1 ωi where ωi is a client-specific weight (e.g., proportional to its local sample size). Other approaches [21], [42] construct fixed prototypes from external priors, Pgc = ac ,
(4)
or treat Pgc as trainable parameters optimized on the server [19], [41], for example, min c
{Pg }
N X X
Lproto Pic , Pgc ,
(5)
i=1 c∈Ci
where Lproto encourages consistency between learnable global prototypes and client local prototypes. In this work, we remain agnostic to the specific mechanism used to construct Pg . We only require that the server can broadcast Pg each round. In our experiments, we adopt the vanilla aggregation-based prototype construction method (i.e., FedProto) by default. Coordinate Alignment. Most existing prototype-based HtFL methods reuse coordinate-level losses such as MSE or cosine similarity, which enforce element-wise matching between z and Pgy : Lcoord (z, Pgy ) = Dcoord (z, Pgy ),
(6)
where Dcoord is instantiated as an ℓ2 loss ∥z−Pgy ∥22 or a cosinebased loss enforcing directional colinearity. This design is natural when all representations reside in a shared feature subspace. However, under model heterogeneity, different encoders can realize the same semantics in different (feature) coordinate bases. The next subsection makes this hidden coupling explicit and motivates a shift from coordinate matching to geometry matching. B. Motivation: Coordinate Alignment Couples Geometry Matching and Basis Matching Coordinate alignment operates in the raw feature coordinates, and thus implicitly assumes that client representations and global prototypes are expressed in a shared feature subspace (i.e., a shared basis). This assumption is often acceptable in homogeneous FL, but becomes fragile in HtFL where heterogeneous feature extractors naturally induce client-specific feature spaces related by unknown transformations. To expose what coordinate alignment actually enforces, we analyze its batch form. Given a mini-batch of n samples, let Z ∈ Rn×d stack client representations (row-wise) and let P ∈ Rn×d stack their corresponding global prototypes. Let Ẑ and P̂ be the row-wise ℓ2 -normalized versions of Z and P .2 Proposition 1 (Implicit Decomposition of Coordinate Alignment). Let Lcoord (Z, P ) := ∥Ẑ − P̂ ∥2F . (7) 2 With row-wise normalization, MSE minimization on Ẑ, P̂ is equivalent to cosine-similarity maximization up to constants.
Let R⋆ be the (orthogonal) Procrustes solution R⋆ ∈ arg min ∥Ẑ−P̂ R∥2F ⇐⇒ R⋆ ∈ arg max ⟨Ẑ, P̂ R⟩, R∈O(d)
R∈O(d)
(8) where ⟨A, B⟩ := tr(A⊤ B) is the Frobenius inner product. Then Lcoord admits the exact decomposition Lcoord (Z, P ) = min ∥Ẑ − P̂ R∥2F + 2 ⟨Ẑ, P̂ R⋆ ⟩ − ⟨Ẑ, P̂ ⟩ . R∈O(d) {z } {z } | | Lshape
Lrigid
(9) Moreover, Lrigid ≥ 0, and Lrigid = 0 iff the identity rotation is optimal for the Procrustes problem, i.e., ⟨Ẑ, P̂ ⟩ = maxR∈O(d) ⟨Ẑ, P̂ R⟩. Eq. (9) reveals that minimizing coordinate alignment implicitly minimizes two coupled objectives: (1) Geometry/structure matching (Lshape ). This term measures the discrepancy between Ẑ and P̂ up to an optimal rotation. In particular, Lshape (Z, P ) = 0 ⇐⇒ ∃R ∈ O(d) : Ẑ = P̂ R ⇐⇒ Ẑ Ẑ ⊤ = P̂ P̂ ⊤ .
(10)
Hence Lshape captures a rotation-invariant notion of relational geometry (e.g., Gram-matrix relations), which is the desirable component for transferring class relations across clients. (2) Rigid basis-matching penalty (Lrigid ). This term is the extra cost incurred by enforcing alignment in a fixed global basis (effectively R = I), instead of allowing the optimal rotation R⋆ . In homogeneous FL, feature spaces are naturally aligned and typically R⋆ ≈ I, making Lrigid negligible, as evidenced by the low effective dimensionality in the HomoShared and Homo-Local settings in Fig. 1. In HtFL, however, R⋆ can deviate significantly from I, so driving Lcoord toward zero implicitly pressures the client to “bend” its representation toward the global basis, potentially suppressing model individuality. This analysis points to a more suitable objective for HtFL: transferring knowledge by matching relations/geometry rather than enforcing pointwise coordinate equality. In what follows, we formalize a structure-alignment framework that isolates the geometry-matching component while avoiding rigid basis matching. C. Structure-Alignment Framework Motivated by the above decomposition, we generalize prototype alignment from pointwise coordinate matching to structure-level matching. Let P ∈ Rn×d and Q ∈ Rn×d denote two sets of vectors arranged as matrices, where each row is a prototype or an instance feature. We define a structure operator S : Rn×d → Z,
(11)
which maps a set of representations to a structural descriptor S(P ) that captures relations among rows (e.g., pairwise distances/angles, Gram matrices, or covariance statistics). Structural alignment is then defined as Lstruct (P, Q) = D S(P ), S(Q) , (12)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Server
Client 1
Global prototypes 𝑃𝑔 = 𝑝𝑔𝑐
𝐶
Broadcast
𝑐=1
𝑃𝑔
Prototype construction
5
Mini-batch 𝑥, 𝑦 5𝑗=1
Local training with structural alignment
Global prototypes 𝑃𝑔
Feature extractor 𝑓𝑖 Representations 𝑧 = 𝑓𝑖 (𝒙) 𝑦1 = 1 𝑦2 = 2 𝑦3 = 1
Client 2 Local training with structural alignment
…
𝑦4 = 3 𝑦5 = 2
𝑃𝑖
Client 𝑖 Local training with structural alignment
：
Class 2
：
Class 3
：
Structural Alignment Insight Aligning structure from global prototypes
Client 𝑖 feature space
Global feature space
S(𝑃𝑔 )
𝑃𝑔
Structure operator 𝑆(⋅)
𝑃𝑖
S(𝑃𝑖 )
Prototype construction
𝑧ǁ
Structure operator 𝑆(⋅)
𝑧
Classifier ℎ𝑖
S(𝑧) Supervised loss 𝐿sup
(1) Federated Pipeline
𝐿proto 𝐷(𝑆(𝑃𝑖 ), 𝑆(𝑃𝑔 ))
S(𝑧)ǁ
Zoom in
Upload local prototypes
Class 1
𝐿inst 𝐷(𝑆(𝑧), 𝑆(𝑧)) ǁ
𝐿local = 𝐿sup + 𝜆𝐿proto + 𝛾𝐿inst
(2) Local Training of Client 𝒊
Fig. 4. Overview of FedSAF. (1) Federated Pipeline: The server constructs global class prototypes Pg by aggregating uploaded local prototypes and subsequently broadcasts Pg to heterogeneous clients. (2) Local Training on Client i: For a given mini-batch, the client extracts representations z = fi (x) and forms batch-wise local prototypes Pi . A structure operator S(·) is applied to compute the semantic structures of both local and global prototypes, utilizing a prototype-level loss Lproto to transfer inter-class relations. The client further assembles z̃ by indexing into Pg according to the label sequence of z, and applies an instance-level loss Linst to steer individual sample features toward a geometry compatible with the global prototype structure.
where D is a discrepancy measure defined in the structural space Z (e.g., Frobenius MSE between matrices, or 1 minus cosine similarity between vectorized structure descriptors). This formulation decouples alignment from absolute coordinates: Lstruct can remain small even when P and Q differ by client-specific transformations, as long as their internal semantic geometry is consistent. Instantiations of Structural Alignment. Different choices of the structure operator S and the discrepancy measure D yield different losses. In this work, we consider the following two concrete instantiations. (1) Gram-Cosine structural alignment (GCSA). Let p̄ := 1 ⊤ 1 P ∈ R1×d be the mean row vector, Pc := P − 1p̄ be the n centered matrix, and define the centered Gram matrix KP := Pc Pc⊤ . We measure similarity via cosine similarity between Gram matrices: ⟨KP , KQ ⟩ LGCSA (P, Q) = 1 − . ∥KP ∥F ∥KQ ∥F
(13)
GCSA is invariant to translation (via centering), rotation, and positive scaling, as formalized in Proposition 2. Proposition 2 (Structure invariance of GCSA). Let P, Q ∈ Rn×d . Suppose Q is obtained from P via an affine featurespace transformation Q = αP R + 1b⊤ , where α > 0, R⊤ R = I, and b ∈ Rd . Then the Gram Cosine structural alignment loss satisfies LGCSA (P, Q) = 0. (2) RDM-Cosine structural alignment (RCSA). To directly match pairwise relational geometry, we use a representational dissimilarity matrix (RDM). Let P̃ be the row-wise
ℓ2 -normalized version of P . We define the squared-distance RDM: 2 RDMP [i, j] = P̃ [i, :] − P̃ [j, :] 2 . We vectorize the upper-triangular entries, vec(RDMP ), and apply cosine similarity: LRCSA (P, Q) = 1−
denoted
⟨vec(RDMP ), vec(RDMQ )⟩ . (14) vec(RDMP ) 2 vec(RDMQ ) 2
RCSA is invariant to orthogonal transformations, as formulated in Proposition 3. Proposition 3 (Structure invariance of RCSA). Let P, Q ∈ Rn×d . Suppose Q is obtained from P via an orthogonal feature-space transformation Q = P R, ⊤
where R R = I. Then the RDM Cosine structural alignment loss satisfies LRCSA (P, Q) = 0. Remark 1 (Structural alignment versus coordinate alignment). Propositions 2 and 3 show that both GCSA and RCSA impose no penalty under feature-space transformations naturally induced by heterogeneous models. In contrast, coordinate alignment incurs nonzero loss under such transformations, as it enforces pointwise matching in a shared feature space with a fixed basis. Notably, RCSA is invariant to orthogonal transformations but penalizes translations, while GCSA further removes translation effects via centering. Both GCSA and RCSA conform to the unified formulation Lstruct (P, Q) = D(S(P ), S(Q)) and instantiate the proposed structure-level alignment paradigm with different structural descriptors. Next, we show how to deploy this framework in prototype-based HtFL, yielding a simple yet effective twolevel structural alignment objective.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
6
D. Structural Alignment for Prototype-Based HtFL
2) Data Heterogeneity. We consider three types of nonIID settings: label-shift, feature-shift, and real-world non-IID. For label-shift non-IID, we adopt the widely used Dirichlet partitioning scheme, where each client’s data is sampled from a Dirichlet distribution Dir(α). The hyperparameter α controls the degree of data heterogeneity, with a smaller α indicating a higher level of non-IID. For feature-shift non-IID, we use cross-domain datasets including PACS, DomainNet, and Amazon Review, and assign one domain to each client. For real-world non-IID, we use the Shakespeare dataset, whose user-wise partitions are naturally non-IID. 3) Model Heterogeneity. Following HtFLlib [12], we simulate model heterogeneity by instantiating a set of model architectures and assigning them to different clients. We denote a heterogeneous configuration as “HtFEX ” or “HtMX ”, where “FE” means that only the feature extractors are heterogeneous, “M” means the entire models are heterogeneous, and “X” is the number of distinct architectures. Each client i is assigned the (i mod X)-th architecture. For image tasks, we consider five heterogeneous settings: HtFEimg , HtFEimg , HtFEimg , 2 4 9 img img HtM4 , and HtM10 . For text tasks, we use HtFEtxt . Unless 6 otherwise specified, HtFEimg is used as the default heteroge9 neous setting. 4) Comparison Methods. We compare FedSAF with representative HtFL methods from multiple categories: (1) prototype-based methods including FedProto [18], FedTGP [19], and AlignFed [21]; (2) knowledge distillation methods including FedGen [31], FedDistill [50]; (3) mutual learning methods including FML [34], FedMRL [35], and FedKD [16]; and (4) partial parameter sharing methods including LGFedAvg [37] and FedGH [38]. All implementations are based on HtFLlib [12] with identical hyperparameter settings. 5) Training Protocol and Metrics. By default, we conduct experiments in a cross-silo setting with N = 20 clients, and all clients participate in every communication round. The batch size is set to 100 and the number of local epochs is E = 5. For state-of-the-art comparisons, we run each method for 300 global rounds to ensure convergence. At each round, we compute the average test accuracy across all clients and report the best accuracy over all rounds. For FedSAF, we adopt the vanilla aggregation-based prototype construction (i.e., FedProto) by default and report two instantiations of the proposed structural alignment framework, denoted FedSAF-GCSA and FedSAF-RCSA, corresponding to the two structural alignment losses described in Section III-C.
FedSAF Workflow. We follow the standard prototypebased HtFL protocol. In each communication round, the server maintains a global prototype set Pg = {Pgc }C c=1 (constructed by any rule in Sec. III-A) and broadcasts it to clients. Each client then performs local updates using its private data and uploads its local class prototypes to the server for updating Pg . Fig. 4 provides an overview of the overall workflow and the proposed two-level structural alignment used in local training. Local Prototype Construction. For client i, given a minibatch {(xj , yj )}B j=1 and representations zj = fi (xj ), we compute batch-wise local prototypes as the mean feature vector for each class: X 1 zj . (15) Pic = |{j : yj = c}| j:y =c j
Stacking prototypes for classes observed in the batch yields Pi ∈ Rni ×d , where ni is the number of present classes. Prototype-level Structural Alignment. Let Ci denote the set of labels appearing in the current batch. We align the structure of the corresponding local and global prototypes: PiCi = {Pic : c ∈ Ci },
PgCi = {Pgc : c ∈ Ci }, (i) Lproto = D S(PiCi ), S(PgCi ) .
(16) (17)
This term transfers the global inter-class relations without requiring coordinate-wise matching. Instance-level Structural Alignment. Prototype alignment constrains class-wise summaries. To further shape the embedding space at the instance level, we align the structure of batch features to a prototype-induced target. Let z ∈ RB×d be the batch feature matrix whose rows are zj , and define a prototype-substituted matrix z̃ by replacing each row with its corresponding global prototype: z̃j = Pgyj ,
z̃ = [z̃1 ; . . . ; z̃B ].
We then define (i) Linst = D S(z), S(z̃) .
(18)
(i)
Intuitively, Linst steers instance features toward a geometry that is compatible with the global prototype structure, while still leaving the client free to choose its own coordinates. Overall local objective. The final local objective combines supervised learning and the two structural alignment terms: (i)
(i)
(i)
Llocal = L(i) sup + λLproto + γLinst .
(19)
Here λ and γ control the strengths of prototype-level and instance-level alignment, respectively. IV. E XPERIMENTS A. Experimental Setup 1) Datasets. We evaluate FedSAF on both vision and language benchmarks. For image classification, we use CIFAR-10 [43], CIFAR-100 [44], Tiny-ImageNet [45], PACS [46], and DomainNet [47]. For text classification, we use AG NEWS [48], Amazon Review [48], and Shakespeare [49].
B. Comparison with State-of-the-Art Methods In this subsection, we compare the proposed FedSAF with representative state-of-the-art (SOTA) HtFL methods under diverse data- and model-heterogeneity settings to systematically evaluate the effectiveness and performance advantages of our framework. 1) Performance under Different Data Heterogeneity Scenarios. Table I reports results on CIFAR-10/100 and Tiny-ImageNet under Dirichlet label shift with HtFE9 . Across all α values, FedSAF-GCSA/RCSA consistently outperform prototype-based baselines (FedProto, FedTGP, AlignFed) while retaining the vanilla global prototype construction.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
7
TABLE I T EST ACCURACY (%) OF DIFFERENT METHODS UNDER VARIOUS NON -IID SCENARIOS ON CIFAR-10, CIFAR-100, AND T INY I MAGE N ET WITH H T FE9 . T HE TOP THREE RESULTS ARE HIGHLIGHTED AS FIRST , SECOND , AND THIRD , RESPECTIVELY. CIFAR-10 α = 0.1
Methods
CIFAR-100
α = 0.5
α = 1.0
α = 0.1
α = 0.5
Tiny ImageNet α = 1.0
α = 0.1
α = 0.5
α = 1.0
Prototype-based HtFL Methods FedProto FedTGP AlignFed
85.81±0.06 85.73±0.03 85.80±0.34
61.53±0.21 61.60±0.31 62.43±0.10
54.34±0.14 53.96±0.25 56.59±0.30
41.07±0.17 41.37±0.01 41.88±0.05
24.48±0.09 24.43±0.17 24.22±0.23
18.89±0.29 18.33±0.13 18.22±0.26
31.52±0.20 31.16±0.10 30.77±0.13
16.96±0.10 15.70±0.02 14.54±0.15
12.51±0.05 11.80±0.11 10.73±0.02
Ours-GCSA Ours-RCSA
88.03±0.13 88.06±0.24
65.95±0.21 65.63±0.19
57.29±0.18 57.57±0.32
44.06±0.16 44.26±0.11
26.83±0.35 26.97±0.24
21.04±0.22 20.83±0.29
32.87±0.19 33.08±0.15
18.02±0.12 17.86±0.16
14.11±0.08 13.41±0.09
FedGen FedGH LG-FedAvg FML FedDistill FedKD FedMRL
84.42±0.34 83.59±0.16 84.53±0.13 86.79±0.18 85.93±0.11 86.80±0.15 86.70±0.19
61.63±0.31 60.87±0.26 61.31±0.09 63.53±0.34 62.35±0.15 63.86±0.16 64.06±0.14
54.54±0.14 54.83±0.06 55.26±0.11 57.09±0.13 55.75±0.14 57.05±0.40 57.11±0.30
40.64±0.12 40.42±0.17 42.56±0.23 42.49±0.36 42.47±0.16 42.54±0.49 42.82±0.37
17.86±0.34 17.64±0.14 19.25±0.24 19.46±0.12 19.55±0.02 19.20±0.16 19.25±0.29
30.86±0.22 29.95±0.23 32.30±0.19 32.43±0.07 32.22±0.32 32.79±0.08 31.43±0.22
14.57±0.06 12.23±0.18 15.75±0.06 17.99±0.05 17.68±0.09 17.80±0.09 16.22±0.14
10.65±0.15 9.12±0.04 11.79±0.05 13.85±0.03 13.07±0.08 13.72±0.04 12.48±0.19
Other HtFL Methods
TABLE II T EST ACCURACY (%) OF DIFFERENT METHODS UNDER D IR (0.1) PARTITION ON CIFAR-10, CIFAR-100, AND T INY I MAGE N ET ACROSS VARIOUS MODEL HETEROGENEITY SETTINGS . T HE TOP THREE RESULTS ARE HIGHLIGHTED AS FIRST , SECOND , AND THIRD , RESPECTIVELY.
23.20±0.21 22.68±0.21 25.03±0.16 25.32±0.17 25.50±0.09 24.78±0.10 24.87±0.23
TABLE III T EST ACCURACY (%) OF DIFFERENT FEATURE ALIGNMENT LOSS UNDER LABEL SHIFT NON -IID PARTITION ON CIFAR-10 AND CIFAR-100. CIFAR-10
CIFAR-100
Methods α = 0.1 α = 0.5 α = 1.0 α = 0.1 α = 0.5 α = 1.0 CIFAR-10 Methods
HtFE2
CIFAR-100
HtFE4
HtFE2
HtFE4
Tiny ImageNet HtFE2
HtFE4
Prototype-based HtFL Methods FedProto FedTGP AlignFed
86.40±0.11 62.51±0.16 55.10±0.17 41.95±0.16 25.26±0.21 19.54±0.25 86.41±0.15 62.69±0.11 55.73±0.13 42.58±0.21 25.55±0.27 19.97±0.19 88.03±0.13 65.95±0.21 57.29±0.18 44.06±0.16 26.83±0.35 21.04±0.22 88.06±0.24 65.63±0.19 57.57±0.32 44.26±0.11 26.97±0.24 20.83±0.29
87.36±0.15 86.89±0.07 44.99±0.10 42.14±0.13 31.70±0.12 32.40±0.11 88.06±0.13 87.49±0.25 45.81±0.13 42.22±0.10 31.74±0.10 31.98±0.14 87.04±0.06 85.51±0.09 43.43±0.03 41.53±0.13 29.80±0.11 29.42±0.18
Ours-GCSA 88.90±0.13 88.71±0.18 46.64±0.15 44.92±0.18 32.99±0.11 32.72±0.21 Ours-RCSA 88.88±0.17 88.95±0.16 46.67±0.14 45.19±0.11 33.13±0.09 32.93±0.17 Other HtFL Methods FedGen FedGH LG-FedAvg FML FedDistill FedKD FedMRL
MSE Cosine GCSA RCSA
85.18±0.12 83.46±0.24 85.64±0.05 87.12±0.05 87.32±0.03 87.22±0.17 87.51±0.29
84.77±0.08 85.26±0.22 84.95±0.22 86.73±0.08 86.86±0.14 86.38±0.19 87.00±0.09
41.56±0.11 41.81±0.14 43.15±0.06 42.25±0.09 43.46±0.09 45.99±0.09 43.63±0.21
39.64±0.19 40.26±0.19 40.83±0.15 40.01±0.21 42.14±0.17 34.72±0.29 41.31±0.19
29.28±0.03 29.90±0.05 30.09±0.05 29.77±0.15 31.05±0.07 32.95±0.07 29.02±0.20
29.77±0.21 30.44±0.18 30.29±0.11 30.77±0.20 32.01±0.14 27.56±0.23 28.98±0.15
the top performer within this family and is consistently competitive against heavier distillation and mutual-learning methods. Notably, as heterogeneity increases, the degradation of FedSAF is smaller than coordinate-alignment baselines, corroborating our motivation that rigid coordinate matching becomes increasingly restrictive as client feature spaces diverge. C. Comparison with Coordinate Alignment
This indicates that, under heterogeneous feature spaces, improving how prototypes are aligned can be more effective than further complicating how prototypes are constructed. Compared with other HtFL baselines that rely on additional auxiliary models or distillation modules, our method attains competitive, and in several cases superior, accuracy while preserving the communication and computation efficiency of prototype-based schemes. By improving the alignment mechanism rather than the prototype construction pipeline, our framework closes much of the performance gap to heavier HtFL methods while maintaining the lightweight communication pattern that makes prototype-based approaches attractive in practice. 2) Performance under Different Model Heterogeneity Scenarios. Table II fixes Dir(0.1) and varies heterogeneity from HtFE2 to HtFE4 . Prototype-based baselines become weaker under more severe heterogeneity, but FedSAF remains
To isolate the effect of the alignment objective, we conduct controlled comparisons where all components (client models, training protocol, and global prototype construction) are held identical, and only the local alignment loss is varied. We compare two coordinate-alignment baselines (MSE and cosine matching) with two structural instantiations (GCSA and RCSA). 1) Performance under Label Shift non-IID Scenarios. We first evaluate the four alignment objectives under label-shift non-IID settings with varying levels of data heterogeneity. The results on CIFAR-10 and CIFAR-100 with HtFE9 are reported in Table III. Both structural alignment variants (GCSA and RCSA) consistently and substantially outperform coordinate alignment baselines (MSE and Cosine) across all datasets and heterogeneity levels. On CIFAR-10, our methods improve over the best coordinate baseline by up to 3.26% (α = 0.5), with an average gain of 2.25% across all settings. On CIFAR100, the improvement reaches 1.48% (α = 0.1), with an
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
8
TABLE IV E XPERIMENTAL RESULTS ON PACS AND D OMAIN N ET DATASETS . T HE BEST AND SECOND - BEST RESULTS ARE HIGHLIGHTED IN BOLD AND UNDERLINE , RESPECTIVELY.
PACS
DomainNet
Method
Art
Cart.
Photo
Sketch
Avg.
Clip.
Info.
Paint.
Quick.
Real
Sketch
Avg.
MSE Cosine GCSA RCSA
64.11 61.96 64.59 64.92
49.80 46.48 51.37 49.27
61.95 65.70 64.16 64.31
63.68 64.50 66.12 65.51
59.89 59.66 61.56 61.00
8.21 9.49 9.28 8.28
2.08 2.09 2.64 3.64
3.23 3.20 5.98 5.11
41.34 39.60 41.83 45.56
13.86 14.18 16.80 22.29
2.50 2.75 4.84 5.97
11.87 11.89 13.56 15.14
average gain of 1.32%. We also observe that the advantage of structural alignment is more pronounced under moderate data heterogeneity (α = 0.5) than under extreme heterogeneity (α = 0.1). A plausible explanation is that, under severe label imbalance, the limited class coverage per client restricts the expressiveness of inter-class structural relations, partially diminishing the benefit of structure-level alignment. Nevertheless, even in this challenging regime, structural alignment maintains a clear margin over coordinate-based methods. 2) Performance under Feature Shift non-IID Scenarios. Beyond label shift, we further evaluate structural alignment under feature shift non-IID settings, where data heterogeneity arises from domain discrepancy rather than class imbalance. We conduct experiments on two cross-domain benchmarks: PACS (4 domains) and DomainNet (6 domains), assigning one domain to each client. The results are reported in Table IV. On PACS, both GCSA and RCSA outperform coordinate alignment baselines in terms of average accuracy, with GCSA achieving 61.56% and RCSA achieving 61.00%, compared to 59.89% for MSE and 59.66% for Cosine. The improvements are consistent across most individual domains, with GCSA obtaining the best performance on Cartoon and Sketch, and RCSA achieving the highest accuracy on Art. On DomainNet, the advantage of structural alignment becomes even more pronounced. RCSA achieves the best average accuracy of 15.14%, substantially outperforming MSE (11.87%) and Cosine (11.89%) by over 3%. The gains are particularly notable on challenging domains such as Real, where RCSA improves over Cosine by 8.11% (from 14.18% to 22.29%), and Sketch, where the improvement reaches 3.22% (from 2.75% to 5.97%). GCSA also demonstrates strong performance, achieving the best results on Painting and competitive accuracy on other domains. These results indicate that structural alignment is especially beneficial when clients exhibit significant feature distribution shifts. In such scenarios, heterogeneous models trained on visually distinct domains naturally develop diverse feature representations. Coordinate alignment, which forces all clients into a shared global feature space, struggles to accommodate these domain-induced variations. In contrast, structural alignment focuses on preserving inter-class relational geometry, allowing each client to maintain domain-specific feature characteristics while still benefiting from global prototype knowledge. 3) Performance in Textual Modality. To verify the generality of structural alignment beyond visual tasks, we conduct experiments on three text classification benchmarks covering
TABLE V T EST ACCURACY (%) OF DIFFERENT FEATURE ALIGNMENT LOSS UNDER DIFFERENT NON -IID SETTINGS IN TEXTUAL MODALITY. Datasets
AG News
Amazon Review
Shakespeare
Scenarios
Label Shift
Feature Shift
Real-World
MSE Cosine GCSA RCSA
52.49±0.09 54.17±0.18 57.49±0.16 57.48±0.21
87.73±0.03 87.68±0.05 88.15±0.03 88.56±0.08
43.41±0.29 44.04±0.16 45.26±0.16 45.75±0.21
diverse non-IID scenarios: AG News (label shift), Amazon Review (feature shift across product categories), and Shakespeare (real-world non-IID with natural user partitions). The results are reported in Table V. Structural alignment consistently outperforms coordinate alignment across all three datasets and non-IID types. On AG News, GCSA and RCSA achieve 57.49% and 57.48% respectively, improving over the best coordinate baseline (Cosine, 54.17%) by 3.31%. On Amazon Review, RCSA obtains 88.56%, surpassing MSE (87.73%) and Cosine (87.68%) by approximately 0.8%. On the Shakespeare dataset, which exhibits naturally heterogeneous data distributions arising from different writing styles of characters, RCSA achieves the best accuracy of 45.75%, outperforming MSE by 2.34% and Cosine by 1.71%. These results demonstrate that the advantage of structural alignment extends beyond image classification to textual modality as well. 4) Performance in Cross-device Scenarios. Previous experiments focus on cross-silo settings with a moderate number of clients. We now evaluate structural alignment in crossdevice scenarios, where the number of clients is significantly larger and each client holds fewer local samples. To simulate the cross-device setting, we set the client number to 50, 100, and 200, with 20% of clients participating in each round. Considering the limited computational capacity of mobile devices, we set the batch size to 10 and the number of local epochs to 1. The results on CIFAR-100 under Dir(0.1) with HtFE9 are reported in Table VI. Structural alignment consistently outperforms coordinate alignment across all client scales. RCSA achieves the best performance in all settings, improving over Cosine by 3.29% with 50 clients, 2.24% with 100 clients, and 1.17% with 200 clients. As the number of clients increases and local data becomes scarcer, all methods experience performance degradation. However, structural alignment maintains a clear advantage throughout, demonstrating robustness under increased
0.89 0.87 0.27 -0.85 -2.17 -1.94 -1.91 -2.64 -2.56 -3.08 -2.62 1.29 1.29 0.10 -0.67 -1.15 -1.82 -2.31 -2.70 -2.50 -2.68 -2.86 1.40 1.18 0.54 -1.01 -1.68 -1.53 -2.57 -2.42 -2.70 -3.00 -2.92
-1.58 -1.14 -0.88 -0.64 -1.44 -1.78 -1.33 -1.74 -2.34 -2.35 -2.60 -0.98 -0.40 -0.72 -0.60 -0.66 -0.64 -1.54 -1.03 -1.56 -1.98 -2.52 -0.59 0.24 0.41 0.56 0.18 -0.62 -0.33 -1.02 -1.02 -1.86 -1.86 1.02 1.16 1.12 1.37 0.61 0.35 -0.32 -0.08 -0.60 -1.08 -1.75 2.33 1.68 1.42 1.19 0.43 0.25 -0.01 -0.50 -0.68 -0.84 -1.50 2.53 2.03 1.81 1.11 0.50 -0.06 -0.15 -0.63 -0.67 -1.28 -1.53 2.47 1.69 1.63 0.67 0.57 0.30 -0.24 -0.23 -0.64 -0.90 -1.24 2.27 2.31 2.03 0.97 0.13 -0.34 -0.41 -1.07 -0.38 -0.93 -1.06
(a) MSE
(b) Cosine
3.31 3.60 3.77 3.96 3.09 2.85 2.83 2.39 2.53 1.47 1.25
4 3 2 1 0 -1 -2 -3 -4 -5
4.12 3.22 3.68 3.21 3.34 2.70 2.43 2.52 2.54 1.91 1.96 3.29 2.89 3.11 3.34 3.37 3.22 3.17 2.57 2.37 1.42 1.13 3.74 3.51 2.86 3.11 3.27 2.75 2.65 3.20 2.65 1.35 1.79 3.29 2.95 2.83 2.76 2.69 2.85 2.72 2.93 1.95 1.41 1.44 2.75 2.75 2.63 2.77 3.01 2.70 2.37 2.19 1.27 2.18 1.69 2.12 2.08 2.41 2.27 2.56 2.21 2.21 2.53 1.79 1.29 1.56 1.19 1.43 1.72 1.90 1.99 2.09 2.15 2.00 1.31 1.64 1.09 1.22 0.97 0.93 1.85 1.65 2.43 1.73 1.61 2.06 1.45 1.30 1.06 1.53 0.67 1.50 2.28 2.48 2.43 1.91 1.52 0.84 1.92 1.95 1.91 1.25 1.81 1.87 2.18 2.19 1.85 2.36 1.17 1.73
2.99 3.07 3.31 3.19 2.48 3.71 2.43 3.42 2.89 2.64 2.97 3.05 2.63 3.12 3.17 3.11 3.23 3.00 3.05 2.39 2.40 2.66 2.53 3.40 2.65 2.81 3.13 1.99 3.13 2.67 2.67 2.47 2.62 3.31 2.92 2.87 3.13 2.85 3.40 2.71 2.93 2.08 2.19 2.26 2.94 2.75 3.03 3.04 3.16 2.95 3.25 2.36 2.01 2.27 2.20 2.63 2.51 2.31 2.65 2.07 2.61 2.43 2.60 1.97 2.13 1.86 1.61 2.01 2.10 2.55 2.14 1.97 2.50 2.53 1.95 2.13 2.56 1.13 1.35 1.05 1.27 1.55 1.78 1.81 1.85 1.95 2.13 2.27 1.35 1.39 1.05 0.49 0.90 0.95 1.46 1.55 1.25 1.52 1.56 2.10 1.83 1.06 0.23 0.99 0.85 1.19 1.43 1.37 1.23 1.47 2.25 1.73 0.93 0.15 0.95 0.51 1.29 1.74 1.76 1.81 1.85
(c) GCSA
(d) RCSA
4 3 2 1 0 -1 -2 -3 -4 -5
Acc. (%)
-2.50 -1.86 -1.19 -1.69 -1.64 -1.81 -2.17 -2.28 -2.40 -2.55 -2.70
4 3 2 1 0 -1 -2 -3 -4 -5
Acc. (%)
-0.38 -0.10 -0.66 -1.23 -1.24 -1.72 -2.26 -2.49 -2.92 -3.12 -2.50
0.0 0.1 1.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 1 0 0 0 0 0 0 0 0 0 0
-1.29 -1.21 -1.40 -1.12 -2.21 -1.65 -2.46 -2.52 -2.50 -2.55 -2.34
-2.29 -1.33 -1.42 -1.61 -1.94 -2.20 -2.62 -2.50 -2.83 -2.66 -2.44
0.0 1 0.1 0 1.0 0 5.0 10 0 .0 15 0 .0 20 0 .0 25 0 .0 30 0 .0 35 0 .0 40 0 .00
client counts and partial participation. These results suggest that structural alignment is well-suited for large-scale crossdevice deployments. 5) Performance under Different Model Heterogeneity Scenarios. We further compare the two alignment paradigms under varying degrees of model heterogeneity. We consider both feature extractor heterogeneity (i.e., HtFE2 and HtFE4 ) and entire model heterogeneity (i.e., HtM4 and HtM10 ). The results on CIFAR-10 and CIFAR-100 under Dir(0.1) are reported in Table VII. Structural alignment (GCSA and RCSA) consistently outperforms coordinate alignment (MSE and Cosine) across all heterogeneity levels on both datasets. More importantly, the advantage of structural alignment becomes increasingly pronounced as model heterogeneity grows. From HtFE2 to HtM10 , the performance gap between structural and coordinate alignment widens consistently. This trend directly supports our central hypothesis: as architectural diversity increases, client feature spaces become more distinct, and enforcing coordinate matching becomes increasingly restrictive. Structural alignment, by focusing on inter-class relational geometry rather than absolute coordinates, gracefully accommodates this diversity and scales to highly heterogeneous scenarios. 6) Sensitivity to Hyperparameters. We investigate the sensitivity of different alignment methods to hyperparameters λ (prototype-level alignment weight) and γ (instance-level alignment weight). Fig. 5 shows the accuracy improvement over the baseline (λ = 0, γ = 0) across a wide range of hyperparameter combinations on CIFAR-100 with HtFE9 and Dir(0.1). The heatmaps reveal a striking contrast between coordinate alignment and structural alignment. For MSE and Cosine, positive gains are observed only when both λ and γ are small. As the alignment strength increases, performance degrades substantially, with large hyperparameter values leading to significant negative transfer (up to -4.58% for MSE). This phenomenon aligns with our theoretical analysis in Section III-B: when alignment strength is weak, the benefit from structural alignment may outweigh the harm from feature space enforcement. However, as alignment strength increases, the rigid coordinate matching increasingly suppresses model individuality, eventually dominating the overall effect and causing performance degradation. This observation provides empirical evidence that forcing clients into a unified feature space is indeed harmful in HtFL. In contrast, GCSA and RCSA exhibit consistently positive improvements across nearly all hyperparameter combinations,
-1.76 -2.40 -2.07 -2.07 -1.74 -2.52 -2.58 -2.82 -2.75 -3.14 -2.34
0.0 0.1 1.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 1 0 0 0 0 0 0 0 0 0 0
34.50±0.10 34.89±0.17 35.35±0.17 36.06±0.10
-2.67 -2.51 -2.26 -2.88 -2.76 -2.78 -3.09 -2.96 -3.11 -3.08 -2.69
-3.06 -3.02 -2.77 -2.86 -3.32 -3.80 -3.76 -3.91 -3.04 -2.77 -3.28
0.0 1 0.1 0 1.0 0 5.0 10 0 .0 15 0 .0 20 0 .0 25 0 .0 30 0 .0 35 0 .0 40 0 .00
38.43±0.08 38.53±0.11 40.47±0.10 40.77±0.06
-2.95 -2.58 -3.02 -2.28 -2.99 -3.03 -2.56 -3.40 -3.16 -2.92 -3.24
Acc. (%)
40.06±0.18 40.24±0.18 43.17±0.16 43.53±0.13
-2.91 -3.07 -2.78 -2.73 -3.06 -3.20 -3.32 -3.37 -2.78 -3.20 -3.30
Acc. (%)
MSE Cosine GCSA RCSA
4 3 2 1 0 -1 -2 -3 -4 -5
-3.96 -3.58 -3.92 -3.33 -3.80 -4.32 -3.81 -3.31 -4.21 -3.78 -3.34
0.0 1 0.1 0 1.0 0 5.0 10 0 .00 15 .0 20 0 .0 25 0 .0 30 0 .0 35 0 .0 40 0 .00
200
0.0 0.1 1.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 1 0 0 0 0 0 0 0 0 0 0
100
-3.48 -4.27 -3.83 -4.32 -4.51 -4.58 -4.41 -3.72 -3.86 -3.80 -3.77
0.0 1 0.1 0 1.0 0 5.0 10 0 .00 15 .0 20 0 .0 25 0 .0 30 0 .0 35 0 .0 40 0 .00
50
Fig. 5. Accuracy improvement (%) over the no-alignment baseline (λ = 0, γ = 0) under different combinations of λ and γ for each alignment method. Baseline
70
Accuracy
Client #
MSE
Cosine
GCSA
RCSA
60 50 40 Res-4
Res-6
Res-8 Res-10 Res-18 Res-34 Res-50 Res-101 Res-152
Clients
(a) CIFAR-10 Baseline
Accuracy
TABLE VI T EST ACCURACY (%) OF DIFFERENT METHODS IN CROSS - DEVICE SCENARIOS ON CIFAR-100.
9
0.0 0.1 1.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 1 0 0 0 0 0 0 0 0 0 0
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
MSE
Cosine
GCSA
RCSA
30 20 10 Res-4
Res-6
Res-8 Res-10 Res-18 Res-34 Res-50 Res-101 Res-152
Clients
(b) CIFAR-100 Fig. 6. Per-client test accuracy on CIFAR-10 and CIFAR-100 under Dir(0.1) with HtFE9 . Each client is assigned a ResNet variant of different depth.
with gains ranging from approximately 1% to 4%. This robustness stems from the fact that structural alignment decouples semantic structure transfer from feature space enforcement, allowing clients to benefit from global prototype knowledge without sacrificing their feature space individuality. Furthermore, the performance remains stable across a broad region around the optimal hyperparameters, indicating strong practical usability without the need for extensive hyperparameter tuning. 7) Per-Client Performance Analysis. To validate our claim that coordinate alignment suppresses model learning capacity
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
10
TABLE VII T EST ACCURACY (%) OF DIFFERENT FEATURE ALIGNMENT LOSS UNDER LABEL SHIFT NON -IID PARTITION ON CIFAR-10 AND CIFAR-100 WITH DIFFERENT MODEL STRUCTURE . CIFAR-10
CIFAR-100
Methods
HtFE2
HtFE4
HtM4
HtM10
HtFE2
HtFE4
HtM4
HtM10
MSE Cosine GCSA RCSA
87.87±0.13 87.67±0.16 88.90±0.13 88.88±0.17
87.47±0.15 87.42±0.09 88.71±0.18 88.95±0.16
86.59±0.08 86.41±0.13 87.89±0.11 87.93±0.11
85.55±0.06 85.91±0.05 87.69±0.10 87.55±0.08
46.44±0.12 44.10±0.10 46.63±0.15 46.67±0.14
43.18±0.21 43.04±0.07 44.92±0.18 45.19±0.11
40.64±0.17 39.99±0.11 42.13±0.08 42.29±0.10
38.45±0.14 39.03±0.15 41.04±0.09 41.61±0.06
8) t-SNE Visualizations for Different Alignment Methods. To provide an intuitive understanding of the difference between coordinate alignment and structural alignment, we visualize the local prototypes from nine clients using t-SNE on CIFAR-10 with HtFE9 , as shown in Fig. 7. Different colors represent different clients, and different markers denote different classes. The visualization reveals a clear distinction. Under coordinate alignment (MSE), prototypes are grouped by class, with the same class from different clients clustering together. This confirms that coordinate alignment enforces pointwise matching, pulling all clients toward a shared feature space regardless of their architectural differences. In contrast, under structural alignment (GCSA), prototypes are grouped by client, with each client’s prototypes forming a distinct cluster. This demonstrates that structural alignment allows each client to optimize within its own feature subspace while maintaining consistent inter-class relations, corroborating our theoretical analysis and the quantitative results presented above.
6
Clients
Clients
Client 0 Client 1 Client 2 Client 3 Client 4 Client 5 Client 6 Client 7 Client 8
Client 0 Client 1 2 Client 2 Client 3 Client 4 Client 5 Client 6 0 Client 7 Client 8
4
2
0
Classes
Classes
2
Class 0 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9
Class 0 Class 1 Class 2 Class 3 Class 4 4 Class 5 Class 6 Class 7 Class 8 Class 9
2
4
6 3
2
1
0
1
2
3
6
4
12
10
8
(a) MSE
6
4
(b) GCSA
Fig. 7. t-SNE visualization of local prototypes under MSE (coordinate alignment) and GCSA (structural alignment) on CIFAR-10 with HtFE9 . Colors denote clients and markers denote classes.
85
+2.22% 88.03% 85.81%
With GCSA
+2.56% 88.29% 85.73%
+2.33% 88.13% 85.80%
80 75 70
Origin
50
Accuracy (%)
Origin
90
Accuracy (%)
in heterogeneous settings, we analyze the performance of each individual client under HtFE9 , where nine clients are equipped with ResNet variants of increasing capacity (ResNet-4 to ResNet-152). Fig. 6 reports the test accuracy of each client on CIFAR-10 and CIFAR-100 under Dir(0.1). The results reveal a clear pattern. For smaller models (ResNet-4 to ResNet-10), all alignment methods provide similar improvements over the baseline, as these models have limited representation capacity and benefit from any form of global guidance. However, for larger models (ResNet-18 and above), coordinate alignment (MSE and Cosine) yields only marginal gains over the baseline, and in some cases even underperforms it. In contrast, structural alignment (GCSA and RCSA) consistently delivers substantial improvements across all model capacities, with the advantage becoming more pronounced for deeper architectures. This observation directly supports our motivation: larger models possess stronger representational power and naturally develop richer, more distinctive feature spaces. Forcing these models to match a shared global feature space constrains their expressiveness and prevents them from fully exploiting their capacity. Structural alignment, by targeting only on inter-class relational geometry, allows each client to leverage its architectural strength while still benefiting from global semantic knowledge. These results empirically confirm that preserving model individuality is crucial for effective knowledge transfer in HtFL.
+2.99% 44.06% 45 41.07% 40
With GCSA
+4.42% 45.79%
41.37%
+3.00% 44.88% 41.88%
35 30 25
FedProto
FedTGP
(a) CIFAR-10
AlignFed
20
FedProto
FedTGP
AlignFed
(b) CIFAR-100
Fig. 8. Test accuracy (%) of existing methods when combining with GCSA.
D. Combination with Advanced Global Prototype Construction Methods Our structure-level alignment framework is orthogonal to the choice of global prototype construction. In all previous experiments, we deliberately adopted the most basic aggregationbased strategy, where global prototypes are obtained by averaging local prototypes across clients. As discussed in Section III-A, most existing prototype-based HtFL methods focus on designing stronger global prototypes. In this subsection, we evaluate FedSAF as a plug-in that can be combined with these advanced construction schemes. Specifically, we keep the global prototype pipelines of FedProto, FedTGP, and AlignFed unchanged, and only replace their original local alignment losses with GCSA. The results on CIFAR-10 and CIFAR-100 are reported in Fig. 8. Three key observations emerge from these experiments. First, structural alignment consistently improves the accuracy of all three prototype-based baselines, confirming that it transfers prototype knowledge to clients more effectively than their original coordinate-level alignment. Second, the performance gain from integrating GCSA reaches up to 4.42%, which is substantially larger than the
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
11
TABLE VIII E XPERIMENTS ON THE CIFAR-10 AND CIFAR-100 TO ILLUSTRATE THE EFFECTIVENESS OF EACH LOSS ITEM . Settings
Lproto
I II III IV
Linst
CIFAR-10
CIFAR-100
✓ ✓
84.87±0.15 86.45±0.11 87.02±0.13 88.03±0.13
40.55±0.08 42.47±0.28 43.27±0.19 44.06±0.16
✓ ✓
TABLE IX C OMMUNICATION AND C OMPUTATION C OSTS ON CIFAR100 WITH H T FE4 .
Method
Comm. (MB)
Computation (s)
Up.
Server
Down.
Client
Prototype-based HtFL Methods
90.0%
FedProto FedTGP AlignFed
1.69 1.69 3.91
3.90 3.90 3.91
0.118 1.235 0.006
32.800 29.640 28.768
Ours-MSE Ours-GCSA
1.69 1.69
3.90 3.90
0.060 0.060
35.698 36.400
44.0%
Other HtFL Methods
88.0% 42.0%
84.0% 82.0%
FedGen FedKD FedGH FedDistill FML LG-FedAvg FedProto
80.0% 78.0% 76.0% 0
50
100
150
FedTGP AlignFed MSE Cosine GCSA RCSA
200
Communication Rounds
(a) CIFAR-10
250
300
Test Accuracy
Test Accuracy
86.0% 40.0% 38.0% FedGen FedKD FedGH FedDistill FML LG-FedAvg FedProto
36.0% 34.0% 32.0% 30.0%
0
50
100
150
FedTGP AlignFed MSE Cosine GCSA RCSA
200
250
FedGen FedGH LG-FedAvg FML FedDistill FedKD FedMRL
3.91 1.69 3.91 70.54 0.33 62.99 70.54
25.20 3.91 3.91 70.54 0.76 62.99 70.54
4.036 0.955 1.936 0.364 0.089 2.180 0.343
29.600 25.924 26.496 37.428 35.815 37.184 26.780
300
Communication Rounds
(b) CIFAR-100
Fig. 9. The test accuracy curve of different methods on CIFAR-10 and CIFAR-100. Best accuracy is marked as star.
gain from improving prototype construction alone (0.81%). This indicates that local alignment is a crucial yet largely overlooked component in prototype-based HtFL, and further highlights the practical value of the proposed structure-level alignment framework. E. Ablation Study We conduct ablation experiments to evaluate the contribution of each component in our structural alignment framework. We consider four configurations on CIFAR-10 and CIFAR100 under Dir(0.1) with HtFE9 using GCSA as the structural alignment instantiation. The results are reported in Table VIII. Comparing Setting I (baseline without any alignment) to Settings II and III, we observe that both Lproto and Linst individually improve performance. On CIFAR-10, Lproto alone yields a 1.58% improvement, while Linst alone provides a 2.15% gain. On CIFAR-100, the improvements are 1.92% and 2.72%, respectively. This confirms that both prototype-level and instance-level structural alignment contribute positively to knowledge transfer. Comparing Setting IV to Settings II and III, combining both losses achieves the best performance, with 88.03% on CIFAR-10 and 44.06% on CIFAR-100. The two losses provide complementary guidance: Lproto enforces consistency in classlevel semantic geometry, while Linst further refines individual sample representations to conform to the global prototype structure. Together, they deliver comprehensive structural guidance at both the class and instance levels. F. Convergence Speed In this section, we examine the convergence behavior of different methods on CIFAR-10 and CIFAR-100 under the Dir(0.1) label shift setting with HtFE9, as shown in Fig. 9.
Our structural alignment methods (GCSA and RCSA) achieve both fast convergence and high final accuracy. Among prototype-based methods, FedProto, FedTGP, AlignFed, and coordinate alignment baselines (MSE, Cosine) converge at similar rates but plateau at noticeably lower accuracy levels. In contrast, some non-prototype HtFL methods such as FedKD and FML eventually reach competitive accuracy, yet their convergence is considerably slower, reflecting the higher computational overhead of auxiliary models or distillation modules. Our structural alignment combines the communication efficiency of prototype-based methods with improved knowledge transfer, achieving strong final performance without sacrificing convergence speed. G. Training Efficiency We compare the communication and computation costs of different methods on CIFAR-100 with HtFE4 , as shown in Table IX. Our method (Ours-GCSA) inherits the communication efficiency of prototype-based approaches. Compared to other HtFL methods that rely on auxiliary models or full-model distillation (e.g., FML, FedKD, FedMRL), our method requires significantly less communication overhead. Regarding computation, our method introduces a modest increase in client-side training time compared to existing prototype-based methods (e.g., 36.4s vs. 32.8s for FedProto) due to the twolevel alignment design. However, this overhead is marginal and does not become a system bottleneck, especially considering the substantial accuracy gains demonstrated in previous experiments. H. Robustness to Local Epoch E The number of local training epochs E is a key hyperparameter in FL. To evaluate its impact on different methods, we vary E ∈ {1, 5, 10, 20} on CIFAR-100 under Dir(0.5) with HtFE9 . As shown in Fig. 10, we make two main observations: (1) Most methods exhibit performance degradation as E increases,
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Accuracy (%)
30
FedProto FedTGP AlignFed
28
FedGen FedGH LG-FedAvg
FML FedDistill FedKD
12
Ours-GCSA Ours-RCSA
26 24 22
1
5
10
20
Local Epoch
Fig. 10. Effect of local epoch E on different methods.
since less frequent communication reduces the effectiveness of global knowledge exchange. (2) Our structural alignment methods (GCSA and RCSA) remain robust across all settings and consistently outperform other HtFL methods under all local epoch configurations. V. D ISCUSSION A. Relationship with Contrastive Alignment Many prototype-based methods adopt contrastive losses (e.g., InfoNCE) rather than direct cosine matching for prototype alignment. To clarify why we use cosine-based alignment as the coordinate baseline, we analyze the contrastive loss and show that it conflates two orthogonal mechanisms. The standard contrastive loss for aligning a representation zi with its corresponding prototype Pyi is formulated as: exp(sim(zi , Pyi )/τ ) Lcon = − log PC j=1 exp(sim(zi , Pj )/τ )
(20)
where sim(·, ·) denotes cosine similarity and τ is a temperature parameter. This loss can be decomposed into two distinct components: C X 1 exp(sim(zi , Pj )/τ ) Lcon = − sim(zi , Pyi ) + log τ | {z } j=1 | {z } Alignment Term Uniformity Term
(21) (1) Coordinate Alignment Term: The first term is strictly equivalent to cosine-based coordinate alignment scaled by 1/τ . (2) Uniformity Term: The second term acts as a regularizer that encourages separation among inter-class prototypes, thereby improving the uniformity of the embedding space. This term governs the global geometry of prototypes and is orthogonal to the alignment mechanism (i.e., how zi matches Pyi ), which is the primary focus of this work. By substituting contrastive objectives with direct cosine alignment, we isolate the alignment mechanism from uniformity regularization, ensuring that our experimental comparisons strictly evaluate the efficacy of structural versus coordinate alignment without the uniformity effect acting as a confounding variable. We note that our proposed structural alignment (GCSA/RCSA) is fully compatible with uniformity constraints. One can incorporate the uniformity term alongside our structural loss to further refine the global prototype geometry, though this extension is outside the primary scope of our analysis.
VI. C ONCLUSION In this paper, we identify a critical yet overlooked limitation in existing prototype-based HtFL methods: the reliance on coordinate alignment, which forces all clients to map their representations into a shared global feature subspace and suppresses model learning capacity. We reveal that coordinate alignment implicitly couples two distinct objectives: aligning inter-class semantic structure, which is beneficial for knowledge transfer, and enforcing a shared feature basis, which is unnecessary and harmful under model heterogeneity. To decouple these effects, we propose FedSAF, which shifts the alignment objective from absolute coordinates to relational geometry. We provide two concrete instantiations, GCSA and RCSA, that effectively transfer global semantic knowledge while preserving the unique feature spaces of heterogeneous client models. Extensive experiments across diverse datasets, non-IID scenarios, modalities, and model heterogeneity levels demonstrate that FedSAF consistently outperforms state-ofthe-art methods. Furthermore, our framework is orthogonal to prototype construction strategies and can serve as a plugand-play module to enhance existing prototype-based HtFL methods. R EFERENCES [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics. PMLR, 2017, pp. 1273– 1282. [2] W. Huang, M. Ye, Z. Shi, G. Wan, H. Li, B. Du, and Q. Yang, “Federated learning for generalization, robustness, fairness: A survey and benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 9387–9406, 2024. [3] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems, vol. 2, pp. 429–450, 2020. [4] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning. PMLR, 2020, pp. 5132–5143. [5] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 713–10 722. [6] H. Zhang, C. Li, W. Dai, Z. Zheng, J. Zou, and H. Xiong, “Stabilizing and accelerating federated learning on heterogeneous data with partial client participation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 1, pp. 67–83, 2025. [7] T. Zhou, Y. Yuan, B. Wang, and E. Konukoglu, “Federated feature augmentation and alignment,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 11 119–11 135, 2024. [8] X. Wu, X. Liu, J. Niu, G. Zhu, and S. Tang, “Bold but cautious: Unlocking the potential of personalized federated learning through cautiously aggressive collaboration,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 19 375–19 384. [9] X. Wu, X. Liu, J. Niu, H. Wang, S. Tang, G. Zhu, and H. Su, “Decoupling general and personalized knowledge in federated learning via additive and low-rank decomposition,” in Proceedings of the 32nd ACM International Conference on Multimedia, ser. MM ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 7172–7181. [Online]. Available: https://doi.org/10.1145/3664647.3681588 [10] X. Wu, J. Niu, X. Liu, G. Zhu, S. Tang, W. Lin, and J. Cao, “The diversity bonus: Learning from dissimilar clients in personalized federated learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 10, pp. 18 613–18 627, 2025. [11] X. Wu, X. Liu, J. Niu, G. Zhu, M. Shi, S. Tang, and J. Yuan, “Tackling feature-classifier mismatch in federated learning via prompt-driven feature transformation,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=vTJFQu5YXz
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
[12] J. Zhang, X. Wu, Y. Zhou, X. Sun, Q. Cai, Y. Liu, Y. Hua, Z. Zheng, J. Cao, and Q. Yang, “Htfllib: A comprehensive heterogeneous federated learning library and benchmark,” in Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025. [13] D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” arXiv preprint arXiv:1910.03581, 2019. [14] T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” Advances in neural information processing systems, vol. 33, pp. 2351–2363, 2020. [15] J. Zhang, S. Guo, X. Ma, H. Wang, W. Xu, and F. Wu, “Parameterized knowledge transfer for personalized federated learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 10 092–10 104, 2021. [16] C. Wu, F. Wu, L. Lyu, Y. Huang, and X. Xie, “Communication-efficient federated learning via knowledge distillation,” Nature communications, vol. 13, no. 1, p. 2032, 2022. [17] C. He, M. Annavaram, and S. Avestimehr, “Group knowledge transfer: Federated learning of large cnns at the edge,” Advances in neural information processing systems, vol. 33, pp. 14 068–14 080, 2020. [18] Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated prototype learning across heterogeneous clients,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8432–8440. [19] J. Zhang, Y. Liu, Y. Hua, and J. Cao, “Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 38, no. 15, 2024, pp. 16 768–16 776. [20] X. Wu, J. Niu, X. Liu, G. Zhu, J. Zhang, and S. Tang, “Enhancing visual representation with textual semantics: Textual semantics-powered prototypes for heterogeneous federated learning,” 2025. [Online]. Available: https://arxiv.org/abs/2503.13543 [21] G. Zhu, X. Liu, S. Tang, and J. Niu, “Aligning before aggregating: Enabling communication efficient cross-domain federated learning via consistent feature extraction,” IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5880–5896, 2024. [22] J. Xu, X. Tong, and S.-L. Huang, “Personalized federated learning with feature alignment and classifier collaboration,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=SXZr8aDKia [23] L. Wang, J. Bian, L. Zhang, C. Chen, and J. Xu, “Taming crossdomain representation variance in federated prototype learning with heterogeneous data domains,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems. [24] W. Huang, M. Ye, Z. Shi, H. Li, and B. Du, “Rethinking federated learning with domain shift: A prototype view,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023, pp. 16 312–16 322. [25] Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang, “Federated learning from pre-trained models: A contrastive learning approach,” Advances in neural information processing systems, vol. 35, pp. 19 332–19 344, 2022. [26] T. Zhou, J. Zhang, and D. H. K. Tsang, “Fedfa: Federated learning with feature anchors to align features and classifiers for heterogeneous data,” IEEE Transactions on Mobile Computing, vol. 23, no. 6, pp. 6731–6742, 2024. [27] M. Ye, X. Fang, B. Du, P. C. Yuen, and D. Tao, “Heterogeneous federated learning: State-of-the-art and research challenges,” ACM Computing Surveys, vol. 56, no. 3, pp. 1–44, 2023. [28] E. Diao, J. Ding, and V. Tarokh, “Hetero{fl}: Computation and communication efficient federated learning for heterogeneous clients,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=TNkPBBYFkXg [29] S. Alam, L. Liu, M. Yan, and M. Zhang, “Fedrolex: Modelheterogeneous federated learning with rolling sub-model extraction,” Advances in neural information processing systems, vol. 35, pp. 29 677– 29 690, 2022. [30] F. Wu, X. Wang, Y. Wang, T. Liu, L. Su, and J. Gao, “Fiarse: Model-heterogeneous federated learning via importance-aware submodel extraction,” Advances in Neural Information Processing Systems, vol. 37, pp. 115 615–115 651, 2024. [31] Z. Zhu, J. Hong, and J. Zhou, “Data-Free Knowledge Distillation for Heterogeneous Federated Learning,” 2021. [32] W. Huang, M. Ye, Z. Shi, and B. Du, “Generalizable heterogeneous federated cross-correlation and instance similarity learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 2, pp. 712–728, 2023.
13
[33] X. Fang, M. Ye, and B. Du, “Robust asymmetric heterogeneous federated learning with corrupted clients,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2693–2705, 2025. [34] T. Shen, J. Zhang, X. Jia, F. Zhang, Z. Lv, K. Kuang, C. Wu, and F. Wu, “Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives,” Frontiers of Information Technology & Electronic Engineering, vol. 24, no. 10, pp. 1390–1402, 2023. [35] L. Yi, H. Yu, C. Ren, G. Wang, X. Li et al., “Federated model heterogeneous matryoshka representation learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 66 431–66 454, 2024. [36] J. Wang, C. Zhao, L. Lyu, Q. You, M. Huai, and F. Ma, “Bridging model heterogeneity in federated learning via uncertainty-based asymmetrical reciprocity learning,” in Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 52 290–52 308. [37] P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, “Think locally, act globally: Federated learning with local and global representations,” arXiv preprint arXiv:2001.01523, 2020. [38] L. Yi, G. Wang, X. Liu, Z. Shi, and H. Yu, “Fedgh: Heterogeneous federated learning with generalized global header,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023. [39] L. Yi, H. Yu, Z. Shi, G. Wang, X. Liu, L. Cui, and X. Li, “Fedssa: semantic similarity-based aggregation for efficient model-heterogeneous personalized federated learning,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024, pp. 5371– 5379. [40] J. Zhang, Y. Liu, Y. Hua, and J. Cao, “An upload-efficient scheme for transferring knowledge from a server-side pre-trained generator to clients in heterogeneous federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 12 109–12 119. [41] Y. Zhou, X. Qu, C. You, J. Zhou, J. Tang, X. Zheng, C. Cai, and Y. Wu, “Fedsa: A unified representation learning via semantic anchors for prototype-based federated learning,” arXiv preprint arXiv:2501.05496, 2025. [42] Y. Dai, Z. Chen, J. Li, S. Heinecke, L. Sun, and R. Xu, “Tackling data heterogeneity in federated learning with class prototypes,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 6, 2023, pp. 7314–7322. [43] A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 (canadian institute for advanced research),” URL http://www. cs. toronto. edu/kriz/cifar. html, vol. 5, 2010. [44] A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009. [45] Y. Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015. [46] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017. [47] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019. [48] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” Advances in neural information processing systems, vol. 28, 2015. [49] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” arXiv preprint arXiv:1812.01097, 2018. [50] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,” arXiv preprint arXiv:1811.11479, 2018.
A PPENDIX A. Proof of Proposition 1 Proof. Recall the coordinate alignment loss Lcoord (Z, P ) := ∥Ẑ − P̂ ∥2F ,
(22)
and define R∗ ∈ arg min ∥Ẑ − P̂ R∥2F ⇐⇒ R∗ ∈ arg max ⟨Ẑ, P̂ R⟩, R∈O(d)
R∈O(d)
(23)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
14
where ⟨A, B⟩ := tr(A⊤ B). Using the identity ∥A − B∥2F = ∥A∥2F + ∥B∥2F − 2⟨A, B⟩, we have Lcoord (Z, P ) = ∥Ẑ∥2F + ∥P̂ ∥2F − 2⟨Ẑ, P̂ ⟩.
(24)
Similarly, for any orthogonal R ∈ O(d), ∥Ẑ − P̂ R∥2F = ∥Ẑ∥2F + ∥P̂ R∥2F − 2⟨Ẑ, P̂ R⟩ = ∥Ẑ∥2F + ∥P̂ ∥2F − 2⟨Ẑ, P̂ R⟩,
(25)
where we used ∥P̂ R∥F = ∥P̂ ∥F since R is orthogonal. Therefore, Lshape (Z, P ) := min ∥Ẑ − P̂ R∥2F
Hence KQ is a positive scalar multiple of KP . The cosine similarity between KP and KQ equals ⟨KP , KQ ⟩ ⟨KP , α2 KP ⟩ = ∥KP ∥F ∥KQ ∥F ∥KP ∥F · ∥α2 KP ∥F α2 ∥KP ∥2F = ∥KP ∥F · α2 ∥KP ∥F = 1,
(35)
whenever ∥KP ∥F ̸= 0 (the degenerate case ∥KP ∥F = 0 implies all rows of P are identical after centering and the alignment is trivially satisfied). Thus, LGCSA (P, Q) = 1−1 = 0.
(26)
C. Proof of Proposition 3
= ∥Ẑ∥2F + ∥P̂ ∥2F − 2 max ⟨Ẑ, P̂ R⟩
(27)
Proof. Assume Q = P R with R⊤ R = I. Let P̃ and Q̃ denote row-wise ℓ2 normalized versions of P and Q:
= ∥Ẑ∥2F + ∥P̂ ∥2F − 2⟨Ẑ, P̂ R∗ ⟩.
(28)
R∈O(d)
R∈O(d)
Combining the above expressions yields the exact decomposition Lcoord (Z, P ) = Lshape (Z, P ) + 2 ⟨Ẑ, P̂ R∗ ⟩ − ⟨Ẑ, P̂ ⟩ =: Lshape (Z, P ) + Lrigid (Z, P ),
(29)
which proves Eq. (9). Finally, by the optimality of R∗ for the Procrustes maximization, ⟨Ẑ, P̂ R∗ ⟩ ≥ ⟨Ẑ, P̂ I⟩ = ⟨Ẑ, P̂ ⟩, (30) hence Lrigid (Z, P ) ≥ 0, and Lrigid (Z, P ) = 0 if and only if the identity rotation achieves the maximum, i.e., ⟨Ẑ, P̂ ⟩ = maxR∈O(d) ⟨Ẑ, P̂ R⟩.
P̃ [i, :] =
P [i, :] , ∥P [i, :]∥2
Q̃[i, :] =
Q[i, :] . ∥Q[i, :]∥2
Because R is orthogonal, it preserves Euclidean norms, so for every row i, ∥Q[i, :]∥2 = ∥P [i, :]R∥2 = ∥P [i, :]∥2 , and therefore P [i, :]R Q̃[i, :] = = P̃ [i, :]R, ∥P [i, :]∥2
or equivalently Q̃ = P̃ R.
Now consider the squared-distance RDM entries used in RCSA: 2
2
RDMQ [i, j] = Q̃[i, :] − Q̃[j, :] 2 = P̃ [i, :]R − P̃ [j, :]R 2 2
2
= (P̃ [i, :] − P̃ [j, :])R 2 = P̃ [i, :] − P̃ [j, :] 2 B. Proof of Proposition 2
= RDMP [i, j],
Proof. Let P̄ ∈ R1×d denote the row-wise mean of P , and define the centered matrix Pc := P − 1P̄ . Similarly, let Q̄ be the row-wise mean of Q and Qc := Q − 1Q̄. By the assumed transformation Q = αP R + 1b⊤ with α > 0 and R⊤ R = I, we have 1 1 Q̄ = 1⊤ Q = 1⊤ (αP R + 1b⊤ ) = αP̄ R + b⊤ . (31) n n Therefore, Qc = Q − 1Q̄ = (αP R + 1b⊤ ) − 1(αP̄ R + b⊤ ) = α(P − 1P̄ )R = αPc R.
(32)
Now consider the centered Gram matrices: KP = Pc Pc⊤ ,
(33)
KQ = Qc Q⊤ c = (αPc R)(αPc R)⊤ = α2 Pc RR⊤ Pc⊤ = α2 Pc Pc⊤ = α2 KP .
(34)
(36)
where we used norm preservation under orthogonal transforms in the third equality. Thus RDMQ = RDMP , which implies their upper-triangular vectorizations are identical: vec(RDMQ ) = vec(RDMP ). Hence the cosine similarity between these two vectors equals 1 (assuming the vector is nonzero; otherwise the loss is trivially 0), and therefore LRCSA (P, Q) = 1 − 1 = 0.