Boosting Multimodal Federated Learning via Chained Modality Optimization
Zixin Zhang 1 Fan Qi 1 2 Shuai Li 2 Xiaoshan Yang 3 Changsheng Xu 3
arXiv:2606.01856v1 [cs.DC] 1 Jun 2026
Abstract Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability. However, most existing MMFL methods cast multimodal training as a joint optimization problem, overlooking a key bottleneck: modality competition, where dominant modalities suppress weaker ones and lead to suboptimal global models. To address this, we propose F ED MC HAIN, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases. This phasewise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an errorcompensated regularizer. On the server side, we employ a sparse sign-guided aggregation strategy that leverages directional sign agreement for robust intra-modality aggregation, avoids destructive averaging, and supports less frequent synchronization to reduce communication overhead. Extensive experiments on multimodal benchmarks demonstrate that F ED MC HAIN consistently improves predictive performance while requiring less frequent communication than baselines.
𝑟=0
All Modalities Joint Learning
𝑟=𝑅
(a) Traditional Multimodal Federated Training
Phase 1: 𝑟 = 0 Visual Modality
Phase 2: Text Modality
Phase 3: Audio Modality
𝑟=𝑅
(b) The proposed Modality-Chained Federated Training
Figure 1. Comparison of the training pipelines of conventional MMFL and our proposed method. Here, r indexes the server communication rounds.
ment (Bao et al., 2023; Yu et al., 2023), designing personalized aggregation and adaptive optimization to accommodate client-specific modality availability and distributions (Chen & Zhang, 2024; Yang et al., 2024; Gao et al., 2025; Pokharel et al., 2025), or jointly optimizing alignment and aggregation for more coherent multimodal collaboration (Qi & Li, 2024; Phung et al., 2025). However, most existing methods implicitly assume that different modalities follow a relatively balanced optimization process, enabling consistent improvements under joint training.
1. Introduction Multimodal Federated Learning (MMFL) serves as a privacy-preserving paradigm for collaborative training over distributed multimodal data silos, holding substantial value for domains such as autonomous driving (Zheng et al., 2023) and intelligent healthcare (Orzikulova et al., 2024). Existing MMFL works primarily tackle optimization difficulties arising from statistical heterogeneity and modality heterogeneity by improving local cross-modal representation align-
The above assumption is often violated in practice by modality competition (Huang et al., 2022; Peng et al., 2022b; Du et al., 2023b), a phenomenon empirically observed in centralized multimodal learning. During joint multimodal training, this competition biases optimization toward dominant modalities, resulting in insufficient learning of weaker ones. Through extensive experiments, we observe that on multimodal clients in MMFL, weaker modalities often suffer from loss stagnation, while the same modalities converge normally on unimodal clients, indicating a modality dominance effect. More importantly, the dominant modality varies across clients, driven by differences in data distri-
1
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China 2 School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China 3 Institute of Automation, Chinese Academy of Sciences, Beijing, China. Correspondence to: Fan Qi <[email protected]>. Preprint. June 2, 2026.
1
Boosting Multimodal Federated Learning via Chained Modality Optimization
bution, data quality, and model availability. At the global level, modality competition can further undermine aggregation stability. Given that the modality competition outcome differs across clients, local models of the same modality may exhibit markedly different convergence progress. Compared with the commonly discussed conflicting update directions in federated learning (FL), this convergence-progress gap poses a more severe challenge to effective cross-client knowledge integration. BMSFed (Fan et al., 2024a) is the closest work to our motivation. It mitigates modality competition via modality selection, aggregating only selected client–modality branches in each round. However, this may discard potentially useful information from unselected branches.
modal benchmarks demonstrate the superior effectiveness and efficiency of the proposed method.
2. Related Works 2.1. Multimodal Federated Learning Based on whether clients share a consistent modality configuration, MMFL can be categorized as consistent and inconsistent MMFL (Che et al., 2023). We focus on the inconsistent setting, where modality availability varies across clients. Existing approaches for inconsistent MMFL mainly fall into two lines: 1) Modularity-based training and aggregation. These approaches treat a part of the client models as shared modules for federated aggregation, thereby facilitating knowledge sharing among clients (Yang et al., 2022; Zhang et al., 2023b; Yuan et al., 2024; Cho et al., 2022; Qi & Li, 2024; Li et al., 2024; Chen & Zhang, 2024; Phung et al., 2025; Pokharel et al., 2025). 2) Representation-based aggregation. Such approaches utilize prototypes or intermediate representations as carriers of knowledge, aggregating them on the server to guide the local training process (Yu et al., 2023; Zeng et al., 2024; Le et al., 2024; Guo et al., 2024; Gao et al., 2025; Seo et al., 2025). However, most inconsistent MMFL methods still optimize modalities jointly at each client and thus do not explicitly address modality competition. Notably, BMSFed (Fan et al., 2024a) targets modal bias by selecting modality-specific networks for communication and aggregating global prototypes to strengthen weaker modalities; yet its selection mechanism may underutilize complementary information and does not directly resolve inter-modality gradient conflicts during local joint optimization.
To bridge the above gap, we propose F ED MC HAIN, a unified framework for balanced multimodal collaboration in heterogeneous MMFL. As illustrated in Figure 1, unlike conventional MMFL, which jointly optimizes all modalities in each communication round, we propose ModalityChained Federated Training (MCFT), a new MMFL training paradigm. MCFT structures global multimodal optimization in a chain-like manner, alternately training modalities by optimizing one modality-specific local model at a time. Accordingly, MCFT splits MMFL into modality-wise phases, optimizing each modality in a unimodal-like regime with an exclusive optimization window to mitigate clientspecific local modality competition without increasing overall training time. To promote cross-modal complementarity across phases, we design a local error-compensated regularizer that up-weights samples misclassified by preceding modalities. Moreover, to improve the stability of intra-modality aggregation under heterogeneity, we propose Sparse Sign-guided Consensus Aggregation (SSCA). SSCA sparsifies client updates and clusters clients by leveraging directional sign agreement, making clustering less sensitive to large cross-client differences in update magnitude. It then fuses information across clusters only on coordinates with consistent directional consensus, avoiding destructive averaging under direction conflicts. As a side benefit, SSCA remains stable under longer synchronization intervals, enabling a lower communication frequency. In summary, our contributions are threefold:
2.2. Imbalanced Multimodal Learning Multimodal learning often suffers from modality competition (Wang et al., 2020; Peng et al., 2022a), where the dominant modality converges faster and suppresses the learning of weaker ones. Recent approaches address this issue from multiple angles. Some methods (Wang et al., 2020; Peng et al., 2022a; Fan et al., 2023; Li et al., 2023; Wei & Hu, 2024) adjust the gradient magnitudes to slow down the learning of dominant modalities, thereby maintaining a more balanced optimization process across modalities. Others (Wu et al., 2022; Du et al., 2023a; Zhang et al., 2023a) introduce additional auxiliary modules to explicitly reweight or recalibrate the contribution of each modality. Another line of work explores improving training paradigms (Fan et al., 2024b; Hua et al., 2024; Jiang et al., 2024; Zhang et al., 2024) or leveraging data augmentation (Hwang et al., 2025; Ma et al., 2025) to achieve more coordinated multimodal representation learning. However, directly applying these methods to MMFL is challenging: gradient modulation and auxiliary recalibration add computation overhead and rely
• We propose F ED MC HAIN, a novel MMFL framework that performs sequential modality optimization in heterogeneous federated settings, enabling more sufficient modality-specific learning and alleviating modality competition. • We introduce a sparse sign-guided aggregation strategy that leverages directional sign agreement for robust intra-modality integration, with reduced communication overhead. • Extensive experiments on three widely used multi2
Boosting Multimodal Federated Learning via Chained Modality Optimization Class B
Trainable
Server
Clients
𝑇
𝑇 𝑇
𝑉
𝑉
Server
Misclassified Samples
Corrected Samples
𝑇
𝑇
ℓ
SSCA
𝑉
SSCA
Frozen
𝑉
𝑉
Feature Space
Visual Modality Training Stage
𝑇
……
Class A
ℓ
+
ℛ! +
𝑉
ℛ"
Feature Space
Textual Modality Training Stage
Figure 2. Illustration of Modality-Chained Federated Training (MCFT), using visual (V ) and textual (T ) modalities as an example.
on sufficiently rich local data, while alternating training can markedly prolong local training, hurting system efficiency in communication-limited federated settings. Consequently, their gains can be limited and less stable under resource constraints and heterogeneous data.
defined as: Fi (ΘkG ; Di ) ≜
ni 1 X (m,k) ℓ Fi {zi,j }m∈Mi , yi,j , ni j=1
min
{Θk }K G k=1
3. Method
ai,k Fi (ΘkG ; Di ),
(2)
k=1
Prior work (Wang et al., 2020; Peng et al., 2022a; Fan et al., 2023; Du et al., 2023a) attributes modality competition to gradient conflict across modalities under joint training, i.e., simultaneously optimizing multiple modalityf specific branches under a shared objective. Let zi,j =
We consider a MMFL framework consisting of a central server and a set of clients C = {1, 2, . . . , |C|}. Each client i ∈ C holds a local dataset Di = ni (m) ({xi,j }m∈Mi , yi,j ) j=1 , where ni = |Di | is the number P|C| of local samples and N = i=1 ni is the total number of samples. Here, Mi ⊆ M denotes the subset of modalities available to client i, and M is the global modality set. Each (m) xi,j is the input of modality m for sample j on client i, and yi,j is the corresponding label. Clients differ in their modality configurations: some are unimodal (|Mi | = 1), while others are multimodal (|Mi | > 1). Each client i maintains modality-specific encoders and classifiers with parameters (m) (m) (m) Θi = {(θE,i , θC,i ) | m ∈ Mi }. Given input xi,j , the (m)
K X
3.2. Modality-Chained Federated Training (MCFT)
3.1. Problem Formulation
(m)
i=1
wi
where wi = nNi , Fi (·; ·) denotes the empirical risk of client i, and ℓ(·, ·) denotes the per-sample local loss function.
In this section, we propose the F ED MC HAIN framework, comprising Modality-Chained Federated Training (MCFT) and Sparse Sign-guided Consensus Aggregation (SSCA), as shown in Figure 2 and Figure 3, respectively. Further details on F ED MC HAIN will be discussed in the following.
(m)
|C| X
(1)
(m)
Fi ({zi,j }m∈Mi ) denote the fused prediction scores for the j-th sample at client i. Following the analysis proposed by Hua et al. (Hua et al., 2024), the learning behavior can be characterized through a gradient alignment analysis: a modality achieves effective optimization and becomes dominant when the gradient of its modality-specific prediction is directionally consistent with that of the fused prediction, D E ∇z(m) ℓ, ∇zi,j > 0, (3) f ℓ i,j
In the opposite case, a persistent misalignment (< 0) causes the modality to converge to suboptimal solutions, hindering its learning progress.
(m)
modality prediction score is zi,j = fC,i (fE,i (xi,j )). Multimodal clients may fuse predictions from multiple modalities using a fusion operator Fi (·), while unimodal clients rely on a single branch. The central server aggregates local updates to learn K sets of global parameters (m,k) (m,k) k {ΘkG }K k=1 , where ΘG = {(θE,G , θC,E )}m∈M , aiming to achieve robustness under both data heterogeneity and modality heterogeneity. At each communication round, each client i is associated with exactly one global model, represented by an assignment indicator ai,k ∈ {0, 1} satisPK fying k=1 ai,k = 1. The global optimization objective is
In MMFL, modality competition not only undermines clientside optimization, but also carries over to server aggregation, resulting in modality-biased global updates and unstable cross-client collaboration. To address this issue, we propose Modality-Chained Federated Training (MCFT), a modalityby-modality optimization paradigm that updates one modality branch at a time, thereby reducing inter-modality gradient interference. Concretely, MCFT decomposes global training into a sequence of modality-specific phases. In phase m, only parameters associated with modality m are optimized: 3
Boosting Multimodal Federated Learning via Chained Modality Optimization (#)
Client 𝒊
ΔΘ!
+4 -2 +2 -1 -3 -4
① sparsification
+1 -2 +3
+4 -2 +2
average
clustering
③
0 -3 -4
-2
+1
+1
④ average
𝑠!
+2 +1 +4
' (#,&) ΔΘ
+4
+3
-1
( (#,&) -1.5 +1 +2 ΔΘ
-4
-1 +1 -4 𝜌: 1.0 0.67 0.57 1.0 1.0 1.0 1.0 0.67 0.5
② symbolization flatten
-1
+4
+3
④
0 -2 +3
0
- Maintain
M Average
&!# ΔΘ
M
M
+1
+4
⑤
- M M M M M - merging +2 +1 -3 +4
+1 -3
-2
-2
-2
( (#,') -1.5 +1 +2 ΔΘ
' (#,') ΔΘ
Clients Set
-1 +1 +4
Figure 3. Illustration of Sparse Sign-guided Consensus Aggregation (SSCA). We use K = 2 as an example.
a) Unimodal clients with modality m update their modality-m branch locally;
apart mismatched pairs within the mini-batch: (m)
Ra,i (Bi ) =
b) Multimodal clients update only modality-m parameters while freezing all other modality branches; c) The server periodically aggregates updates restricted to modality m from participating clients to maintain modality-consistent global optimization.
X 1 |Pm ∩ Mi | ′ m ∈Pm ∩Mi " # (m) (m′ ) B exp S(h̃i,b , h̃i,b )/τ 1 X − log PB , (m) (m′ ) B b=1 b′ =1 exp S(h̃i,b , h̃i,b′ )/τ (5)
where S(·, ·) denotes cosine similarity, τ is a temperature coefficient, and b, b′ index samples within the mini-batch.
Nevertheless, phase-wise modality-specific optimization alone is insufficient to fully leverage multimodal collaboration. Beyond avoiding gradient interference, effective MMFL requires (i) cross-modal alignment to induce consistent semantics across modalities and (ii) cross-modal complementarity to ensure that distinct modalities contribute non-redundant information. To this end, we augment the client objective in each phase with explicit regularizers targeting both properties. Formally, during the phase with active modality m, client i minimizes the following minibatch objective:
(ii) Cross-modal Complementarity Regularizer. To promote cross-modal knowledge complementarity, we introduce an error-compensation regularizer that increases the loss contribution of samples that are predicted with low confidence by modalities trained in preceding phases. For client i at the phase of modality m, we first aggregate the logits from preceding modalities available locally: X (m′ ) (<m) p̄i,j (yi,j | xi,j ) = softmax( zi,j )yi,j , (6) m′ (<m)
" (m)
Li
=EBi ∼Di
where m′ ∈ Pm ∩ Mi and p̄i,j (yi,j | xi,j ) denotes the aggregated confidence on the ground-truth label. We then define an error-compensation weight from the precedingphase aggregated confidence:
X 1 (m) ℓ zi,b , yi,b |Bi | {z } (xi,b ,yi,b )∈Bi | Task
(m)
(m)
+ I[ |Pm ∩ Mi | > 0 ] λa Ra,i (Bi ) + λc Rc,i,b | {z } | {z } Alignment
#
(m)
,
ei,j
Complementarity
(<m)
= 1 − p̄i,j
(yi,j | xi,j ) ,
(7)
and construct the complementarity regularizer by modulating the supervised loss of the active modality m: (m) (m) (m) Rc,i,j = ei,j ℓ zi,j , yi,j . (8)
(4)
where Pm denotes the set of modalities preceding m in the chain, Ra and Rc are regularizers for cross-modal alignment and complementarity, respectively; I[·] enforces that these terms are applied only when the required preceding modalities are available at client.
3.3. Sparse Sign-guided Consensus Aggregation (SSCA)
(i) Cross-modal Alignment Regularizer. We encourage the active modality m to learn modality-invariant semantics by aligning its representation with those of modalities in (m) (m) (m) Pm ∩Mi . Let hi,b = fE,i (xi,b ) denote the latent feature of modality m for the b-th sample in a mini-batch Bi of size B = |Bi |. We ℓ2 -normalize features as h̃ = h/∥h∥2 and define an alignment regularizer that pulls together crossmodal representations of the same sample while pushing
Although MCFT mitigates modality competition by decoupling modality updates across stages, within-modality aggregation remains challenging. Under statistically non-IID client data, updates for the same modality can be highly inconsistent across clients. To address this issue, we propose Sparse Sign-guided Consensus Aggregation (SSCA), which leverages directional agreement to selectively consolidate intra-modality updates, reducing destructive averaging 4
Boosting Multimodal Federated Learning via Chained Modality Optimization
when client updates are inconsistent under statistical heterogeneity.
coordinate-wise cross-cluster mask µq = I(ρq ≥ π) .
For a given modality m, each participating client i ∈ C com(m) (m) putes a modality-specific client vector ∆Θi = Θi − (m,k) ΘG , i.e., the directional offset between its local modality branch and the corresponding server model. We then apply a sparsification operator T (∆Θ, κ) with retention ratio κ to keep the most influential coordinates and zero out the rest, yielding a sparse signed update whose sign reflects the dominant optimization direction: (m) (9) si = sign T (∆Θi , κ) .
For µq = 1, we first identify the dominant direction dq = sign(pq − nq ),
X
k=1 α
k
P where αk = i∈Ck wi is the total weight of cluster Ck . For µq = 0 (i.e., ρq < σ), we do not enforce cross-cluster merg(m,k) (m,k) ing and simply retain ∆Θ̂q = ∆Θ̄q . Combining the two cases, the final aggregated update can be expressed compactly as ∆Θ̂(m,k) = µq ∆Θ̂(m) + (1 − µq ) ∆Θ̄(m,k) . q q q
(16)
This rule unifies clusters only on coordinates with clear directional dominance, while keeping sign-ambiguous coordinates at the cluster-level consensuses to avoid interference from conflicting optimization directions. Finally, the server updates the modality-specific global parameters via a controlled merge step: (m,k)
(m,k)
ΘG,r+1 = ΘG,r
+ λmerge ∆Θ̂(m,k) ,
(17)
where λmerge ∈ (0, 1] regulates the integration strength of the aggregated update to stabilize optimization under heterogeneous client updates. Notably, SSCA reconciles divergent client updates by emphasizing directional consensus during aggregation, which improves robustness when client updates are misaligned under heterogeneity. Empirically, this robustness makes SSCA less sensitive to longer synchronization intervals (Table 3), allowing the server to aggregate less frequently and thus reducing communication overhead.
We then coordinate the K cluster consensuses through a parameter consistency score ρ, which measures cross-cluster directional agreement at each parameter coordinate and determines whether to unify the corresponding updates or preserve specific differences. Concretely, for each coordi(m,k) nate q, let ∆Θ̄q denote the q-th entry of ∆Θ̄(m,k) . We define ρq as the dominance ratio of the stronger direction: ∆Θ̄(m,k) , nq = q
(m,k) (m,k) ∆Θ̄q I sign(∆Θ̄q ) = dq , (15) PK (m,k) k ) = dq + ε k=1 α I sign(∆Θ̄q
PK ∆Θ̂(m) = q
Within each cluster Ck , we aggregate the member updates to form a cluster-level consensus update that captures their common optimization direction. Given the sparsified modal(m) (m) ity update ∆Θ̃i = T (∆Θi , κ), we compute the cluster consensus by a sample-weighted mean: P (m) i∈Ck wi ∆Θ̃i (m,k) P ∆Θ̄ = . (10) i∈Ck wi
X
(14)
and aggregate only the cluster entries aligned with dq :
The resulting sign vectors {si }i∈C provide compact directional descriptors of client updates. To group clients with coherent update orientations, we build a clustering feature by flattening and concatenating all tensors in si into a single vector, and perform unsupervised clustering in this sign feature space, yielding K directional groups {C1 , . . . , CK }. Sign-based clustering focuses on consistent update directions while attenuating magnitude heterogeneity, thereby reducing destructive averaging in subsequent within-group aggregation.
pq =
(13)
3.4. Convergence Analysis
∆Θ̄(m,k) , q
Here, ε is a small constant for numerical stability. The score ρq ∈ [0, 1] quantifies how strongly one update direction dominates across clusters at coordinate q: values close to 1 indicate that most cluster-level consensuses agree on the same sign, while smaller values imply severe sign conflicts and thus low cross-cluster consistency.
We establish a convergence guarantee for F ED MC HAIN in smooth non-convex federated optimization. For an arbitrary modality stage m and an arbitrary cluster k, we consider the corresponding stage objective F (m,k) induced by the participating clients in Ck . Under standard federated assumptions, Theorem D.13 in Appendix D proves the canonical nonconvex stationarity bound R−1 1 X 1 (m,k) 2 2 E ∥∇F (Θr )∥ ≤ O + O(σagg ), (18) R r=0 R
Based on ρq , SSCA applies a thresholding rule with hyperparameter π ∈ (0, 1) to decide whether to merge cluster updates at coordinate q. Specifically, we construct a
where R is the number of server aggregation rounds and 2 σagg captures the SSCA-induced aggregation noise. The details of the proof are provided in Appendix D.
(m,k)
k:∆Θ̄q
(m,k)
>0
k:∆Θ̄q
<0
(11) ρq =
max(pq , nq ) . pq + n q + ε
(12)
5
Boosting Multimodal Federated Learning via Chained Modality Optimization Table 1. Performance comparison on CREMA-D, AVE, and CMU-MOSEI. We report overall accuracy (ACC, averaged over all clients), unimodal-client accuracies (ACCv /ACCa /ACCt , averaged over clients that only have the corresponding modality), and multimodal-client accuracy (ACCm , averaged over multimodal clients). Best results are in bold and second best are underlined. CREMA-D
Method
AVE
CMU-MOSEI
ACC
ACCv
ACCa
ACCm
ACC
ACCv
ACCa
ACCm
ACC
ACCv
ACCa
ACCt
ACCm
Local FedAvg FedProx SCAFFOLD
48.96 47.11 48.35 48.27
29.64 31.33 30.74 31.41
50.45 49.81 50.69 51.25
50.69 51.75 51.97 50.85
45.14 47.18 48.25 51.09
26.37 29.49 30.26 30.57
43.81 44.06 44.68 48.23
49.25 50.24 51.42 53.75
55.48 55.63 56.59 58.83
40.36 41.22 40.03 41.62
39.15 39.84 40.24 41.47
59.84 58.15 60.11 62.06
54.15 55.39 55.45 57.31
FedMSplit CreamFL HAMFL M3 Fed FedMBridge BMSFed
51.62 53.66 52.53 53.94 54.88 56.14
33.59 35.68 41.16 40.15 39.82 44.25
51.02 52.95 52.21 51.52 51.95 53.65
53.46 55.13 54.60 55.98 56.75 58.69
55.16 55.44 56.17 55.28 57.33 57.62
29.64 32.25 33.51 33.05 35.69 37.96
50.18 53.46 53.78 54.12 56.33 58.25
58.83 58.98 59.45 59.36 61.84 60.22
61.17 62.40 61.53 61.29 63.05
43.74 43.89 45.14 45.96 46.24
44.12 46.08 48.26 50.38 49.76
64.52 65.97 68.46 68.21 68.12
60.92 63.83 63.77 64.68 64.30
Ours
58.36
43.89
54.10
60.68
59.85
38.49
57.95
62.74
64.96
45.86
50.25
68.37
65.75
4. Experiments
MMFL baselines under modality distribution heterogeneity: Local (where each client is trained independently without communication), classic FL methods adapted to multimodal settings (FedAvg (McMahan et al., 2017), FedProx (Li et al., 2020), SCAFFOLD (Karimireddy et al., 2020)), and recent heterogeneous MMFL approaches (HAMFL (Qi & Li, 2024), M3 Fed (Li et al., 2024), CreamFL (Yu et al., 2023), FedMSplit (Chen & Zhang, 2022), FedMBridge (Chen & Zhang, 2024), BMSFed (Fan et al., 2024a)). Among them, BMSFed is specifically designed to alleviate modality competition in MMFL scenarios, serving as a particularly relevant and competitive baseline to our approach.
4.1. Experimental Setup Datasets. We evaluate our proposed method on three commonly used multimodal benchmarks: CREMA-D (Cao et al., 2014), AVE (Tian et al., 2018), and CMUMOSEI (Zadeh et al., 2018). To simulate federated multimodal settings with heterogeneous data distributions, we partition each dataset into multiple clients using a Dirichlet distribution with concentration parameter β ∈ {0.5, 1.0}1 ,where a smaller β indicates higher heterogeneity. Within each client, the local data are split into 80% for training and 20% for testing. Additionally, we simulate heterogeneous modality availability by masking modalities for a subset of clients to create diverse cross-client modality configurations. Further dataset and partitioning details are included in the Appendix B.1.
4.2. Performance Analysis Table 1 summarizes the performance of our method and representative baselines on three datasets. Overall, our approach achieves the best ACC (overall clients) and ACCm (multimodal clients) across the three datasets, suggesting that it more effectively mitigates modality competition and thus better preserves cross-modal complementarity during collaborative training. Although we are not the top method on every unimodal subset (e.g., ACCv on CREMA-D can be slightly lower than BMSFed), we consistently outperform classical FL baselines such as FedAvg/FedProx/SCAFFOLD on unimodal-client metrics, and often exceed Local training. Since unimodal clients cannot leverage crossmodal fusion locally, these gains primarily reflect more reliable server-side aggregation under non-IID updates, providing indirect evidence of the robustness of our aggregation strategy. BMSFed is the most competitive baseline and can yield higher unimodal-client performance in some cases, consistent with its modality-aware coordination that explicitly targets modality competition and better protects unimodal branches from being overwhelmed by multimodal objectives. In contrast, a group of heterogeneous-MMFL
Implementation Details. All methods are implemented in PyTorch (Paszke et al., 2017) and trained on 8 NVIDIA RTX 3090 GPUs. For a fair comparison, all baselines are evaluated under the same data partition protocol and identical backbone/model configurations. The local batch size is set to 128, and each communication round runs one local epoch. We train for 500 communication rounds and perform server aggregation every 20 rounds. For our method, we set the sparsity ratio κ to 0.7, the consensus threshold π to 0.9, the merge learning rate λmerge to 0.9, and the loss weighting coefficients λa and λc to 0.4 and 1.0, respectively. For all datasets, we repeat each experiment three times with different random seeds, and report the mean performance. Further experimental setup details are in Appendix B.2. Baselines. We compare our method with representative 1 Unless otherwise specified, all results reported in this paper are obtained with β = 1.0.
6
Boosting Multimodal Federated Learning via Chained Modality Optimization Table 2. Ablation study of our framework on CREMA-D, AVE, and CMU-MOSEI. Method
CREMA-D
AVE
CMU-MOSEI
w/o MCFT w/o SSCA w/o Ra w/o Rc w/o Ra &Rc
54.96 ↓3.40 56.25 ↓2.11 57.12 ↓1.24 57.03 ↓1.33 56.31 ↓2.05
54.82 ↓5.03 55.43 ↓4.42 58.14 ↓1.71 57.29 ↓2.56 55.60 ↓4.25
60.39 ↓4.57 60.98 ↓3.98 64.25 ↓0.71 63.84 ↓1.12 63.08 ↓1.88
Ours
58.36
59.85
64.96
𝛽 = 1.0
𝛽 = 0.5
M 3F ed
Figure 5. The effect of different data heterogeneity on performance of all competitors.
were misclassified by preceding modalities, thereby promoting complementary evidence rather than redundant signals. Consequently, removing both terms simultaneously breaks alignment and weakens complementarity, leading to the most pronounced performance loss. (a) MIR
(b) K
4.4. Further Analysis
Figure 4. Analysis of modality competition and clustering sensitivity. (a) Modality imbalance ratio (MIR) on CREMA-D. (b) Effect of the number of SSCA clusters K on test accuracy across three datasets under different data heterogeneity levels β ∈ {0.5, 1.0}.
Modality competition. Modality competition often enlarges the performance gap between modalities. To quantify this effect, we report the modality imbalance ratio (MIR), where a larger value indicates more severe modality imbalance (see Appendix C for the formal definition). As shown in Figure 4a on CREMA-D, our method achieves the smallest MIR among all compared approaches (1.31). This reduced imbalance is achieved together with the best ACC/ACCm in Table 1, showing that our method improves modality balance without sacrificing the stronger modality’s performance.
methods (e.g., FedMSplit, CreamFL, HAMFL, M3 Fed, and FedMBridge) achieves weaker overall results. Although these approaches introduce modularization or bridging to cope with missing modalities, they still do not sufficiently suppress inter-modality gradient conflicts, so modality competition persists during training and ultimately limits both ACC and ACCm . Finally, classical FL algorithms may even underperform Local training (e.g., FedAvg 47.11 vs. Local 48.96 on CREMA-D), indicating that naive parameter averaging can amplify conflicting updates and lead to negative transfer rather than effective knowledge sharing.
Effect of the number of clusters. We evaluate our method with different numbers of clusters, setting K ∈ {2, 3, . . . , 10}. Figure 4b reports test accuracy on CREMAD, AVE, and CMU-MOSEI under β ∈ {0.5, 1.0}. Across datasets and both β values, performance improves as K increases from 2 to roughly 3 ∼ 6, and then plateaus or mildly degrades when K ≥ 7. In particular, the best (or near-best) performance is typically achieved around K = 4 ∼ 6, suggesting that a moderate clustering granularity provides a good balance between capturing intra-class diversity and avoiding over-fragmentation. When K is too large, the clustering becomes overly fine-grained and introduces noisy assignments, which can hurt generalization. Overall, the method is relatively insensitive within the moderate range, and we set K = 5 as a default in all experiment.
4.3. Ablation Studies Table 2 isolates the effect of each component. Removing MCFT causes the largest performance degradation, indicating that modality-wise optimization is essential for mitigating modality competition. Disabling SSCA also leads to a clear drop. In this setting, the server aggregation degenerates to a plain averaging scheme applied separately to each modality branch. This indicates that SSCA is necessary to stabilize collaborative optimization and to prevent naive averaging from causing unstable aggregation under heterogeneous client updates. Removing either Ra or Rc results in a noticeable degradation, and ablating both produces an even larger drop. Specifically, Ra promotes cross-modal alignment, which facilitates reliable knowledge transfer among modality branches, whereas Rc reweights training to make each modality focus more on samples that
Degree of Heterogeneity of Data Distribution. We investigate the performance of different methods on CREMA-D under two Dirichlet heterogeneity levels (β ∈ {0.5, 1.0}). As shown in Figure 5, classic FL baselines exhibit pronounced performance degradation as heterogeneity increases. In contrast, MMFL approaches such as FedMBridge improve in 7
Boosting Multimodal Federated Learning via Chained Modality Optimization AVE
CREMA-D
CMU-MOSEI
Time (s)
300
HAMFL
(per round) CreamFL
280 Ours
260
r tte Be
240 220
M3Fed
200 Audio Phase
Visual Phase
Audio Phase
Text Phase
Visual Phase
Audio Phase
Visual Phase
180
FedAvg
(per client)
140
3k 2k 1k
COT (M)
Ours(AVG)
120
(per commutation round)
100 0
(b)
SCAFFOLD
Bubble size: GFLOPs
160
(a)
FedMBridge FedMSplit
BMSFed
FedProx
30
60
90 120 150 180 210 240 270 300
(c)
(d)
Figure 6. (a)-(c) The effect of communication rounds on the performance of the three datasets; (d) Efficiency trade-offs for all competitors on CREMA-D (x-axis: communication cost per round; y-axis: average runtime per round; bubble size: local computation per round). Table 3. The effect of aggregation period on the performance of the three datasets.
the more heterogeneous regime, owing to their personalized parameter aggregation or update mechanisms that mitigate cross-client negative transfer. Notably, our method achieves the best results under both settings and further improves under stronger heterogeneity (+6.03%). We attribute this improvement to SSCA, it groups clients by directional sign agreement and aggregates within groups, which becomes increasingly beneficial as non-IID intensifies.
period (υ)
1
5
10
15
20
30
CREMA-D 57.69 57.93 58.08 58.21 58.36 58.33 58.76 59.38 59.71 59.64 59.85 59.42 AVE CMU-MOSEI 64.02 64.34 64.19 64.22 64.96 64.83
Number of Communication Rounds. Figure 6 shows the average test performance across all clients over different communication rounds. Due to the chained modality training paradigm, our method exhibits a distinct phase-wise performance improvement pattern, with noticeable gains at each modality transition. Through successive modality updates, these phase-wise improvements accumulate, enabling our method to outperform all baselines in later communication rounds. Notably, this advantage is achieved under the same number of communication rounds as the baselines, indicating higher performance for the same overall training duration.
and the lowest per-round communication cost (leftmost position) among all compared methods. This is because MCFT updates only one modality branch per stage while freezing the others, reducing both backpropagation and the transmitted payload. Regarding the per-round time, the dominant cost of our method comes from the global aggregation stage. With SSCA enabling stable aggregation at a lower frequency, we additionally report Ours(AVG), which measures the average runtime over all communication rounds (including both aggregation and non-aggregation rounds). Ours(AVG) achieves the lowest average time, indicating that our method has the shortest overall training time among all competitors.
Sensitivity of the Aggregation Period. Table 3 compares different aggregation periods υ ∈ {1, 5, 10, 15, 20, 30}, where the server aggregates once every υ rounds. Performance improves as υ increases from 1 to 20, and then becomes flat or slightly degrades for larger υ. This trend can be explained by the reliance of SSCA on directional sign consistency. SSCA clusters clients using the sparsified signed updates and enables cross-cluster merging only on coordinates that satisfy the consensus threshold. As υ increases, local optimization yields cleaner and more consistent update directions, making the sign-based grouping more reliable, which in turn improves aggregation quality. Beyond a certain point, however, longer local trajectories under non-IID data amplify drift and reduce cross-client directional agreement, so fewer coordinates exceed the consensus threshold and the benefit of SSCA saturates or mildly declines.
Due to space limitations, additional experimental results are provided in Appendix C.
5. Conclusion & Limitation In this work, we identify modality competition as a key bottleneck in MMFL. We propose F ED MC HAIN, which structures training into modality-wise phases to balance modality learning and employs a sparse sign-guided aggregation for conflict-aware, communication-efficient updates. Extensive experiments demonstrate the effectiveness of our method. A limitation is that the modality training order is currently chosen empirically; developing more adaptive and principled ordering strategies is left for future work.
Impact Statement
Efficiency Analysis. We illustrate the efficiency comparison of different methods in Figure 6d. Our method attains the lowest per-client local computation (smallest bubbles)
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be 8
Boosting Multimodal Federated Learning via Chained Modality Optimization
specifically highlighted here.
Fan, Y., Xu, W., Wang, H., Huo, F., Chen, J., and Guo, S. Overcome modal bias in multi-modal federated learning via balanced modality selection, 2024a. URL https: //arxiv.org/abs/2401.00403.
References Bao, G., Zhang, Q., Miao, D., Gong, Z., Hu, L., Liu, K., Liu, Y., and Shi, C. Multimodal federated learning with missing modality via prototype mask and contrast. arXiv preprint arXiv:2312.13508, 2023.
Fan, Y., Xu, W., Wang, H., Liu, J., and Guo, S. Detached and interactive multimodal learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 5470–5478, 2024b.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390, 2014.
Gao, M., Zheng, H., Feng, X., and Tao, R. Multimodal fusion using multi-view domains for data heterogeneity in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 16736– 16744, 2025.
Che, L., Wang, J., Zhou, Y., and Ma, F. Multimodal federated learning: A survey. Sensors, 23(15):6986, 2023.
Guo, Q., Yao, M., Tian, Z., Qi, S., Qi, Y., Lin, Y., and Dong, J. S. Contribution evaluation of heterogeneous participants in federated learning via prototypical representations. arXiv preprint arXiv:2407.02073, 2024.
Chen, J. and Zhang, A. Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 87–96, 2022.
Hua, C., Xu, Q., Bao, S., Yang, Z., and Huang, Q. Reconboost: Boosting can achieve modality reconcilement. arXiv preprint arXiv:2405.09321, 2024.
Chen, J. and Zhang, A. Fedmbridge: Bridgeable multimodal federated learning. In Forty-first International Conference on Machine Learning, 2024.
Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L. Modality competition: What makes joint training of multimodal network fail in deep learning? (provably), 2022. URL https://arxiv.org/abs/2203.12221.
Cho, Y. J., Manoel, A., Joshi, G., Sim, R., and Dimitriadis, D. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703, 2022.
Hwang, S.-H., Choi, S., and Whang, S. E. Midas: Misalignment-based data augmentation strategy for imbalanced multimodal learning, 2025. URL https: //arxiv.org/abs/2509.25831.
Degottex, G., Kane, J., Drugman, T., Raitio, T., and Scherer, S. Covarep — a collaborative voice analysis repository for speech technologies. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 960–964, 2014. doi: 10.1109/ICASSP. 2014.6853739.
Jiang, Q.-Y., Chi, Z., and Yang, Y. Multimodal classification via modal-aware interactive enhancement. arXiv preprint arXiv:2407.04587, 2024. Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132–5143. PMLR, 2020.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv. org/abs/1810.04805.
Le, H. Q., Thwal, C. M., Qiao, Y., Tun, Y. L., Nguyen, M. N., and Hong, C. S. Cross-modal prototype based multimodal federated learning under severely missing modality. arXiv preprint arXiv:2401.13898, 2024.
Du, C., Teng, J., Li, T., Liu, Y., Yuan, T., Wang, Y., Yuan, Y., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning. In International Conference on Machine Learning, pp. 8632–8656. PMLR, 2023a. Du, C., Teng, J., Li, T., Liu, Y., Yuan, T., Wang, Y., Yuan, Y., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning, 2023b. URL https://arxiv. org/abs/2305.01233.
Li, H., Li, X., Hu, P., Lei, Y., Li, C., and Zhou, Y. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22214–22224, 2023.
Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029–20038, 2023.
Li, S., Qi, F., Zhang, Z., and Xu, C. Cross-modal meta consensus for heterogeneous federated learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 975–984, 2024. 9
Boosting Multimodal Federated Learning via Chained Modality Optimization
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
Stich, S. U. Local SGD converges fast and communicates little. In International Conference on Learning Representations (ICLR), 2019. URL https://openreview. net/forum?id=S1g2JnRcFX. arXiv:1805.09767.
Ma, X., Chen, H., and Deng, Y. Improving multimodal learning balance and sufficiency through data remixing. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=hDH3KfZSsF.
Tian, Y., Shi, J., Li, B., Duan, Z., and Xu, C. Audiovisual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pp. 247–263, 2018. Wang, W., Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12695–12705, 2020.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.
Wei, Y. and Hu, D. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. arXiv preprint arXiv:2405.17730, 2024.
Orzikulova, A., Kwak, J., Shin, J., and Lee, S.-J. Federated learning for time-series healthcare sensing with incomplete modalities. arXiv preprint arXiv:2405.11828, 2024.
Wu, N., Jastrzebski, S., Cho, K., and Geras, K. J. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp. 24043–24055. PMLR, 2022.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. 2017. Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8238–8247, 2022a.
Yang, X., Xiong, B., Huang, Y., and Xu, C. Cross-modal federated human activity recognition via modality-agnostic and modality-specific representation learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 3063–3071, 2022.
Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation, 2022b. URL https://arxiv.org/abs/ 2203.15332.
Yang, X., Xiong, B., Huang, Y., and Xu, C. Cross-modal federated human activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Phung, T. H., Nguyen, D. M., Huynh, T. T., Nguyen, Q. V. H., Hoang, T. N., and Le Nguyen, P. Federated prompttuning with heterogeneous and incomplete multimodal client data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3936–3946, October 2025.
Yu, H., Yang, S., and Zhu, S. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019. URL https://ojs.aaai. org/index.php/AAAI/article/view/4514. arXiv:1807.06629.
Pokharel, A., Rahman, R., Morris, T., and Nguyen, D. C. Quantum federated learning for multimodal data: A modality-agnostic approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 545–554, June 2025.
Yu, Q., Liu, Y., Wang, Y., Xu, K., and Liu, J. Multimodal federated learning via contrastive representation ensemble. arXiv preprint arXiv:2302.08888, 2023. Yuan, L., Han, D.-J., Chellapandi, V. P., Żak, S. H., and Brinton, C. G. Fedmfs: Federated multimodal fusion learning with selective modality communication, 2024. URL https://arxiv.org/abs/2310.07048.
Qi, F. and Li, S. Adaptive hyper-graph aggregation for modality-agnostic federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12312–12321, 2024.
Zadeh, A. B., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236–2246, 2018.
Seo, M., Kim, T., Lee, H., Choi, J., and Tuytelaars, T. Not all clients are equal: Collaborative model personalization on heterogeneous multi-modal clients, 2025. URL https: //arxiv.org/abs/2506.11024. 10
Boosting Multimodal Federated Learning via Chained Modality Optimization
Zeng, H., Yue, Z., and Wang, D. Open-vocabulary federated learning with multimodal prototyping. arXiv preprint arXiv:2404.01232, 2024. Zhang, H., Li, Y., and Li, X. Constrained bipartite graph learning for imbalanced multi-modal retrieval. IEEE Transactions on Multimedia, 26:4502–4514, 2023a. Zhang, R., Chi, X., Liu, G., Zhang, W., Du, Y., and Wang, F. Unimodal training-multimodal prediction: Cross-modal federated learning with hierarchical aggregation. arXiv preprint arXiv:2303.15486, 2023b. Zhang, X., Yoon, J., Bansal, M., and Yao, H. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 27456–27466, 2024. Zheng, T., Li, A., Chen, Z., Wang, H., and Luo, J. Autofed: Heterogeneity-aware federated multimodal learning for robust autonomous driving. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, pp. 1–15, 2023.
11
Boosting Multimodal Federated Learning via Chained Modality Optimization
A. Appendix Overview Given the space limitations in the main paper, we defer additional materials to the appendix. Appendix B provides further implementation and experimental setup details, Appendix C reports additional experimental results, Appendix D contains the convergence analysis and corresponding proofs, and Appendix E summarizes the proposed method with pseudocode.
B. Additional Experimental Setup B.1. Datasets CREMA-D CREMA-D is an audio-visual dataset for speech emotion recognition. It contains 7,442 original clips (2–3 seconds) from 91 actors speaking short utterances, annotated with six emotion classes (anger, disgust, fear, happy, neutral, and sad). Labels are obtained via crowd-sourcing from 2,443 raters. AVE The Audio-Visual Event (AVE) dataset is a benchmark for audio-visual event localization, consisting of 4,143 YouTube videos from 28 event categories. Each video is temporally annotated with audio-visual event boundaries and contains at least one event segment of 2 seconds. The events span diverse domains including human/animal activities, musical performances, and vehicle-related sounds. CMU-MOSEI CMU-MOSEI is a large-scale multimodal sentiment/emotion dataset, comprising 23,453 annotated utterances extracted from over 5,000 videos, with more than 1,000 distinct speakers and roughly 250 topics, offering substantial diversity in both content and speaker characteristics. Human annotators label each sample with a sentiment score from -3 (strongly negative) to +3 (strongly positive). We view this as a three classification problem, with the categories being negative, neutral, and positive. Table 4. Statistics of the datasets used in our experiments. Dataset
Modality
#sample #class #client
Visual Audio Text CREMA-D 7,442 AVE 3,741 CMU-MOSEI 22,346
6 28 3
30 30 70
✓ ✓ ✓
✓ ✓ ✓
✗ ✗ ✓
As shown in Figure 7-Figure 8, we present heatmaps of the client-wise label distributions induced by our Dirichlet-based partitioning under two heterogeneity levels (β ∈ {0.5, 1.0}). For each dataset, we construct a client–class matrix and normalize each client’s label counts into proportions; the heatmap visualizes these proportions with color intensity, where columns denote clients and rows denote classes. In particular, a smaller concentration parameter (β = 0.5) leads to more skewed and client-specific label compositions, whereas a larger value (β = 1.0) yields comparatively more balanced label distributions across clients. As shown in Figure 9, we further illustrate the client-wise modality availability resulting from our modality partitioning strategy. Specifically, we construct a binary client–modality matrix in which each entry indicates whether a modality is available at a given client; the heatmap uses color to distinguish available versus missing modalities, with columns denoting clients and rows denoting modalities. Such modality-level non-IID settings reflect realistic multimodal federated scenarios, where acquisition or system constraints lead to systematic modality missingness across participants. B.2. Training Details With respect to the CREMA-D and AVE datasets, we adopt a ResNet-18 backbone with modality-specific input stems (3-channel for vision and 1-channel for audio). For both datasets, we use one video frame of size 224 × 224 × 3, extracted as the middle frame of each clip (with random resized cropping and horizontal flipping during training). For the audio stream, we load the waveform at 22,050 Hz and ensure a fixed duration of 3 seconds by truncation or repetition. We then compute a log-magnitude STFT spectrogram using n fft = 512 and hop length = 353, resulting in 257 frequency bins (i.e., 1 + n fft/2). The visual and audio backbones output 512-dimensional global features via adaptive pooling, which are fed into a fully connected layer to produce modality-specific predictions. Finally, the audio and visual predictions are fused to obtain the final score. 12
Boosting Multimodal Federated Learning via Chained Modality Optimization
(a) CREMA-D
(b) AVE Figure 7. Client-wise label distributions on CREMA-D and AVE dataset under different heterogeneity levels (β ∈ {0.5, 1.0}).
Figure 8. Client-wise label distributions on CMU-MOSEI dataset under different heterogeneity levels (β ∈ {0.5, 1.0}).
For the CMU-MOSEI dataset, we use pre-extracted language, visual, and acoustic features. The text modality is represented by BERT embeddings (Devlin et al., 2019) with a 768-dimensional feature size, while the visual and audio modalities use FACET features (35 dimensions) and COVAREP features (Degottex et al., 2014) (74 dimensions), respectively. These features are passed through modality-specific encoders to produce 128-dimensional latent representations. In particular, AudioNet and VisualNet adopt a three-layer MLP backbone, where each layer is followed by ReLU, dropout, and layer normalization, and then a linear layer is used to output modality-specific predictions. For the text modality, TextNet uses a bidirectional LSTM to encode the input sequence; the final hidden state is projected to a 128-dimensional vector and fed into a linear prediction head. Finally, we combine the predictions from different modalities to obtain the final output. For all three datasets, we use the Adam optimizer with dataset-specific learning rates of 1 × 10−5 , 1 × 10−4 , and 1 × 10−3 , respectively. In each communication round, we randomly sample a fraction of clients with sampling ratio 0.7 to participate in training. We run federated training for 500 rounds in total, and allocate the same number of training rounds to each modality. For the SSCA, we perform direction-consistency clustering using the standard KMeans algorithm.
13
Boosting Multimodal Federated Learning via Chained Modality Optimization available
missing
Figure 9. Client-wise modality availability on CREMA-D/AVE and CMU-MOSEI.
C. Additional Experimental Results Impact of different modality training orders. Table 5 reports the performance of MCFT under different modality training Table 5. The effect of MCFT modality training order on the performance of the three datasets. orders on CREMA-D/AVE (two modalities) and CMU-MOSEI (three modalities). Empirically, on CREMA-D and AVE, the Audio → Visual 58.36 CREMA-D Audio → Visual schedule consistently outperforms Visual → Visual→Audio 56.09 Audio by a clear margin (58.36 vs. 56.09 on CREMA-D; 59.85 Audio→Visual 59.85 vs. 56.33 on AVE). On CMU-MOSEI, schedules that start with AVE Visual→Audio 56.33 Text perform best, with Text → Audio → Visual achieving the top score (64.96), while starting from Audio/Visual leads to Textual→Audio→Visual 64.96 noticeably worse results (e.g., 62.07–63.24). We attribute this Text→Visual→Audio 64.58 pattern to the design of MCFT: later modalities are trained with Audio→Visual→Text 62.16 CMU-MOSEI explicit cross-modal alignment and complementarity regularizAudio→Text→Visual 63.24 ers that depend on the set of preceding modalities, i.e., earlier Visual→Audio→Text 62.07 stages effectively provide “reference signals” that shape subseVisual→Text→Audio 62.37 quent optimization. Consequently, placing a more semantically reliable modality earlier (Audio in CREMA-D/AVE, Text in MOSEI) yields stronger guidance for later stages, whereas starting from a weaker/less stable modality can propagate suboptimal biases to downstream stages. Sensitivity of hyperparameters (κ, π) in SSCA. Table 6 evaluates the hyperparameter sensitivity of SSCA by varying the retention ratio κ and the consensus threshold π, two factors that directly control how much client update information is preserved and how conservatively cross-cluster updates are reconciled. Specifically, κ retains the top-κ fraction of coordinates (thus sparsifying 1 − κ), while π gates coordinate-wise cross-cluster merging according to the directional dominance criterion. Regarding κ, performance improves markedly as κ increases from 0.1 to the range 0.7–0.8, but then saturates or slightly degrades when κ becomes very large (e.g. 0.9). This behavior suggests a trade-off inherent to SSCA: overly small κ corresponds to aggressive sparsification that discards many informative yet non-dominant coordinates, yielding an underexpressive global update and reducing the reliability of sign-based clustering. Conversely, overly large κ retains almost the full update and therefore re-admits low-magnitude, high-variance coordinates that are most sensitive to client heterogeneity, which blurs the dominant directional structure exploited by SSCA and increases the chance that conflicting local drifts adversely affect the aggregated update. For π, moderate-to-high values consistently outperform small thresholds, with the best results attained around π ≈ 0.8. When π is too small, SSCA merges coordinates even when directional dominance is weak, effectively over-smoothing across clusters and amplifying cross-cluster interference. When π is too large (e.g., 0.9), the criterion becomes overly conservative, suppressing merging even for coordinates with substantial but not overwhelming agreement, and thus leaving shared beneficial signal underutilized. Sensitivity of λa and λc on the three datasets. Figure 10(a–c) shows the effect of λa and λc on CREMA-D, AVE, and CMU-MOSEI. Overall across the three datasets, the strongest performance is achieved when λa is in the range 0.3–0.5 and λc is in the range 0.8–1.2 (with our final choice λa = 0.4 and λc = 1.0 lying near the optimum). This observation suggests that moderate λa provides sufficient cross-modal alignment to transfer stable semantics from preceding modalities, while moderate λc strengthens complementary learning by emphasizing hard/uncertain samples; excessively small values weaken both effects, whereas overly large values over-regularize the stage objective and can amplify noisy reweighting signals, 14
Boosting Multimodal Federated Learning via Chained Modality Optimization
reducing generalization. Sensitivity of λmerge on the three datasets. Figure 10(d) reports the effect of λmerge on the three datasets. Overall, performance peaks when λmerge is in the range 0.8–0.9 (with our final choice λmerge = 0.9). The reason is that λmerge controls how strongly the SSCA-aggregated consensus update is injected into the global model: too small values make the server update overly conservative and slow to accumulate consensus, while too large values make the update overly aggressive and more sensitive to residual cross-client inconsistency, slightly hurting stability. C.1. Modality Imbalance Ratio (MIR) We use the modality imbalance ratio (MIR) to quantify the degree of modality competition, i.e., how unevenly the model performs when restricted to different modalities. For each modality m, we evaluate the model in a unimodal inference manner by enabling only modality m (and disabling/masking other modalities) on all clients (including multimodal clients), and obtain the unimodal accuracy for each client. We then average these unimodal accuracies over all clients to get a ¯ (m) . Finally, MIR is defined as the ratio between the best and the worst modality-wise modality-wise mean accuracy ACC mean accuracies: ¯ (m) maxm ACC MIR = . (19) ¯ (m) minm ACC By construction, MIR ≥ 1, and a larger MIR indicates a larger modality-induced performance imbalance. Table 6. The effect of hyperparameters κ and π on the performance of the three datasets.
κ
κ = 0.1
κ = 0.2
κ = 0.3
κ = 0.4
κ = 0.5
κ = 0.6
κ = 0.7
κ = 0.8
κ = 0.9
CREMA-D AVE CMU-MOSEI
50.16 54.38 60.67
51.84 54.97 61.68
53.12 55.43 62.24
55.25 57.31 62.53
56.34 58.66 63.75
58.01 59.18 64.27
58.36 59.85 64.84
58.14 59.73 64.96
57.79 59.44 64.65
π
π = 0.1
π = 0.2
π = 0.3
π = 0.4
π = 0.5
π = 0.6
π = 0.7
π = 0.8
π = 0.9
CREMA-D AVE CMU-MOSEI
56.43 55.92 59.36
56.29 56.47 59.73
56.82 57.55 60.31
57.02 57.32 60.56
57.33 57.86 61.43
58.39 58.48 63.76
58.21 59.24 63.99
58.36 59.85 64.96
58.14 59.45 64.76
(a) CREMA-D
(b) AVE
(c) CMU-MOSEI
(d) 𝜆!"#$"
Figure 10. (a)-(c): The effect of λa and λc on the performance of the three datasets. (d): The effect of λmerge on the performance of the three datasets.
15
Boosting Multimodal Federated Learning via Chained Modality Optimization
D. Convergence Analysis D.1. Problem Setup Modality-stage view. MCFT proceeds in modality-specific stages by activating one modality at a time. In a fixed stage m, only the parameter block associated with the active modality is updated, while all other modality branches are frozen (see main text). Accordingly, the analysis below focuses on the stage-m parameter block, denoted by Θ ∈ Rdm . Cluster-specific objective. SSCA maintains K cluster-specific global models whose parameter blocks evolve independently. Fix a cluster index k ∈ {1, . . . , K} and let Ck denote the set of clients assigned to the cluster model k in the current stage. We define the stage-m, cluster-k global objective as h i X X (m) F (m,k) (Θ) := pi Eξ∼Di fi (Θ; ξ) , pi ≥ 0, pi = 1, (20) i∈Ck
i∈Ck
(m)
where fi is the stage-m local loss used by client i (Task + Alignment + Complementarity, as defined in the main text). For brevity, we write F (Θ) when (m, k) is clear from context. Local update and SSCA aggregation. Let r = 0, 1, 2, . . . index the server aggregation rounds for the cluster model k in (m,k) stage m. At the beginning of round r, the server broadcasts Θr ≡ Θr to clients in Ck . Each client i ∈ Ck performs E local SGD steps: (m) s s Θs+1 (Θsi,r ; ξi,r ), s = 0, 1, . . . , E − 1, Θ0i,r = Θr , (21) i,r = Θi,r − η gi (m)
where gi
(m)
(·; ξ) is a stochastic gradient of fi
at sample ξ and η > 0 is the local stepsize.
Define the client vector: ∆Θi,r := ΘEi,r − Θr .
(22)
SSCA takes {∆Θi,r }i∈Ck as input and outputs an aggregated update direction for cluster k: b r ≡ ∆Θ b (m,k) ∆Θ ∈ Rdm . r
(23)
The server updates the cluster-k model with merge strength λmerge > 0: b r. Θr+1 = Θr + λmerge ηg ∆Θ
(24)
Notation. The server update in the main text uses a single coefficient λmerge (main). For analysis, we rewrite it as λmerge (main) = ηg λmerge , where ηg is a global stepsize and λmerge denotes the merge strength. This is a purely notational re-parameterization and does not change the algorithm. D.2. Assumptions (m)
Assumption D.1 (Smoothness). For any modality stage m and any client i, the local objective fi (Θ) is L-smooth: for all (m) (m) Θ, Θ′ , ∥∇fi (Θ) − ∇fi (Θ′ )∥ ≤ L∥Θ − Θ′ ∥. Consequently, the corresponding cluster objective F (m,k) (Θ) is also L-smooth. Assumption D.2 (Unbiased stochastic gradients and bounded variance). For all i ∈ Ck and all Θ, the stochastic gradient (m) (m) (m) (m) satisfies E[gi (Θ; ξ)] = ∇fi (Θ) and E∥gi (Θ; ξ) − ∇fi (Θ)∥2 ≤ σ 2 . Assumption D.3 (Client heterogeneity bound). There exists ζ ≥ 0 such that for all Θ, X 2 (m) pi ∇fi (Θ) − ∇F (Θ) ≤ ζ 2 .
(25)
i∈Ck
Assumption D.4 (Expected descent correlation of SSCA update). There exists a constant γ > 0 such that for all rounds r, D E 2 b r | Θr ∇F (Θr ), E ∆Θ ≤ −γ ∇F (Θr ) . (26) 16
Boosting Multimodal Federated Learning via Chained Modality Optimization
h i b r ∥2 | Θr exists and is finite for all Assumption D.5 (Bounded second moment of SSCA update). Assume that E ∥∆Θ 2 rounds r. There exist constants B > 0 and σagg ≥ 0 such that for all rounds r, 2 b r ∥2 | Θr ≤ B ∥∇F (Θr )∥2 + σagg E ∥∆Θ .
(27)
Remark D.6 (Interpretation of Assumptions D.4–D.5). Assumption D.4 requires that the SSCA update is descent-correlated b r | Θr ]⟩ < 0. Assumption D.5 upper-bounds the conditional second moment of the in expectation, i.e., ⟨∇F (Θr ), E[∆Θ 2 SSCA update and permits an “error floor” σagg that captures additional noise introduced by Top-κ sparsification, sign quantization, and coordinate-wise masking/merging. D.3. Local descent for the stage-m objective (m)
Lemma D.7 (One-step expected descent for local SGD). Suppose each fi (m) for a local update Θ+ = Θ − η gi (Θ; ξ) with η ≤ L1 ,
is L-smooth and Assumption D.2 holds. Then
(m) η Lη 2 2 2 (m) (m) E fi (Θ+ ) ≤ fi (Θ) − ∇fi (Θ) + σ . 2 2
(28)
(m)
Proof. By L-smoothness of fi , f (Θ+ ) ≤ f (Θ) + ⟨∇f (Θ), Θ+ − Θ⟩ + L2 ∥Θ+ − Θ∥2 . Substitute Θ+ − Θ = −ηg and 1 take expectation. Use unbiasedness and E∥g∥2 ≤ ∥∇f (Θ)∥2 + σ 2 . Finally, 0 < η ≤ 1/L implies 1 − Lη 2 ≥ 2. Lemma D.8 (E-step local progress bound). Under the conditions of Lemma D.7, for any client i and any aggregation round r, E−1 X 2 (m) 2 (m) (m) E ∇fi (Θsi,r ) ≤ Efi (Θr ) − Efi (ΘEi,r ) + LηEσ 2 . (29) η s=0 Proof. Apply Lemma D.7 to each s = 0, . . . , E − 1 and sum. Rearrange terms. D.4. Local drift under periodic aggregation Define the “round-start ideal” local update: (m)
∆Θideal := −ηE ∇fi i,r
(Θr ),
(30)
edrift := ∆Θi,r − ∆Θideal i,r i,r .
(31)
and the local drift error: (m)
Lemma D.9 (Local drift bound (second moment)). Under Assumptions D.2–D.3 and the L-smoothness of fi universal constants c1 , c2 > 0 (depending only on L) such that for η ≤ L1 , X 2 pi E ∥edrift ≤ c1 η 2 E(E − 1) σ 2 + c2 η 2 E 2 ζ 2 . i,r ∥
, there exist
(32)
i∈Ck
We follow the standard drift analysis for local SGD and periodic model averaging (Stich, 2019; Yu et al., 2019) and adapt it to the stage-m objective. Proof sketch. Expanding the local update gives ∆Θi,r = −η
PE−1
(m) s (Θsi,r ; ξi,r ). s=0 gi
For each s, add and subtract
(m) ∇fi (Θsi,r ) to obtain (m)
gi
(m)
s (Θsi,r ; ξi,r ) − ∇fi
(m)
(Θr ) = gi |
(m)
s (Θsi,r ; ξi,r ) − ∇fi {z stochastic noise
(m)
(Θsi,r ) + ∇fi } |
(m)
(Θsi,r ) − ∇fi {z drift
(Θr ) . }
The noise term forms a martingale difference sequence and, under Assumption D.2, contributes a mean-squared accumulation (m) (m) on the order of η 2 Eσ 2 . For the drift term, L-smoothness yields ∥∇fi (Θsi,r ) − ∇fi (Θr )∥ ≤ L∥Θsi,r − Θr ∥. Moreover, E∥Θsi,r −Θr ∥2 is bounded by recursively unrolling (21) and invoking Assumptions D.2–D.3. Summing over s = 0, . . . , E −1 gives the claimed scaling O(η 2 E(E − 1)σ 2 + η 2 E 2 ζ 2 ). 17
Boosting Multimodal Federated Learning via Chained Modality Optimization
D.5. SSCA as an Inexact Descent Direction b r . We model SSCA From local deltas to SSCA direction. SSCA maps the collection of local deltas {∆Θi,r }i∈Ck to ∆Θ as an aggregation operator that returns a descent-correlated update with a bounded conditional second moment, formalized in Assumptions D.4 and D.5. 2 Optionally, one may decompose σagg into contributions from (i) Top-κ sparsification/sign quantization and (ii) coordinatewise dominance masking, but the analysis below only requires the combined bound in (27).
Assumption D.10 (SSCA stability w.r.t. client deltas). There exists a constant Cssca ≥ 1 such that for any two collections of client deltas {ui }i∈Ck and {vi }i∈Ck , the SSCA mapping satisfies X 2 SSCA({ui }) − SSCA({vi }) ≤ Cssca pi ∥ui − vi ∥2 . (33) i∈Ck
Remark D.11 (On Assumption D.10). Assumption D.10 is a technical abstraction that controls how perturbations in client deltas (e.g., those induced by multi-step local training) are amplified through the SSCA mapping. While SSCA involves non-smooth operations such as Top-κ selection, sign quantization, and discrete clustering, (33) can be viewed as a piecewise stability condition that holds within regions where the selected support and cluster assignment are unchanged, or as an expected stability condition with respect to the inherent randomness in the selection/clustering procedures. Ideal SSCA direction (no local drift). ing ideal SSCA direction:
Using the ideal client deltas {∆Θideal i,r }i∈Ck defined in (30), define the correspond b ideal := SSCA {∆Θideal }i∈C . ∆Θ r i,r k
(34)
b r = SSCA({∆Θi,r }). The actual SSCA direction is ∆Θ Lemma D.12 (Propagation of local drift through SSCA). Under Assumption D.10, X 2 b r − ∆Θ b rideal ∥2 ≤ Cssca E ∥∆Θ pi E ∥edrift i,r ∥ ,
(35)
i∈Ck 1 where edrift i,r is defined in (31). Consequently, by Lemma D.9, for η ≤ L , b r − ∆Θ b rideal ∥2 ≤ Cssca c1 η 2 E(E − 1) σ 2 + c2 η 2 E 2 ζ 2 . E ∥∆Θ
(36)
Proof. Apply Assumption D.10 to ui = ∆Θi,r and vi = ∆Θideal i,r : b r − ∆Θ b rideal ∥2 ≤ Cssca ∥∆Θ
X
2 pi ∥∆Θi,r − ∆Θideal i,r ∥ = Cssca
i∈Ck
X
2 pi ∥edrift i,r ∥ .
i∈Ck
Taking conditional expectation given Θr gives (35), and (36) follows by Lemma D.9. D.6. Main result: Nonconvex convergence to a neighborhood Theorem D.13 (Stage-m, cluster-k nonconvex convergence of FedMediator). Suppose Assumptions D.1–D.3 hold. Assume b r produced by SSCA at round r. Choose the server further that Assumptions D.4–D.5 hold for the actual SSCA direction ∆Θ stepsize ηg > 0 such that γ−
L ηg λmerge B > 0 2
(e.g., ηg <
2γ when B > 0). Lλmerge B
(37)
Then for any number of aggregation rounds R ≥ 1, R−1
1 X 2 E ∇F (Θr ) ≤ R r=0 η λ g
2 L ηg λmerge σagg 2 F (Θ0 ) − F ∗ , + 2γ − Lη λ B R 2γ − Lη λ B merge g merge g merge
where F ∗ := inf Θ F (Θ). 18
(38)
Boosting Multimodal Federated Learning via Chained Modality Optimization
Proof. By L-smoothness (Assumption D.1) and the update (24), F (Θr+1 ) ≤ F (Θr ) + ∇F (Θr ), Θr+1 − Θr + br = F (Θr ) + ηg λmerge ∇F (Θr ), ∆Θ
L 2 Θr+1 − Θr 2 Lηg2 λ2merge b r 2. ∆Θ + 2
(39)
Take conditional expectation w.r.t. Θr and apply Assumptions D.4–D.5: Lηg2 λ2merge 2 E F (Θr+1 ) | Θr ≤ F (Θr ) − ηg λmerge γ∥∇F (Θr )∥2 + B∥∇F (Θr )∥2 + σagg 2 Lηg2 λ2merge 2 Lηg λmerge B ∥∇F (Θr )∥2 + = F (Θr ) − ηg λmerge γ − σagg . 2 2
(40)
Now take total expectation and sum (40) over r = 0, . . . , R − 1: R−1 Lηg2 λ2merge Lηg λmerge B X 2 ηg λmerge γ − E∥∇F (Θr )∥2 ≤ F (Θ0 ) − EF (ΘR ) + Rσagg . 2 2 r=0
Using F (ΘR ) ≥ F ∗ and dividing both sides by ηg λmerge γ −
(41)
Lηg λmerge B R yields (38). 2
Remark D.14 (Choosing stepsizes). Condition (37) ensures a positive descent coefficient. A sufficient choice is ηg = 2γ 1 O λmerge , and in particular ηg < Lλmerge B when B > 0. Remark D.15 (From stage-wise to multi-stage (MCFT cycles)). Theorem D.13 is stage-wise (fixed active modality m). Under a cyclic schedule where each stage is activated regularly, one can sum the per-stage descent inequalities over a full cycle and obtain an analogous bound on the cycle-averaged gradient norm, with constants depending on the number of stages. Remark D.16 (A sufficient condition for Assumption D.5 under periodic local training). Theorem D.13 assumes Assumption D.5 for the actual SSCA direction. When periodic local training is used, one convenient way to verify this assumption b rideal in (34) and control the additional drift-induced error via Assumption D.10 is to start from the ideal SSCA direction ∆Θ b ideal with and Lemma D.12. In particular, if Assumptions D.4–D.5 hold for ∆Θ r D E b ideal | Θr ] ≤ −γ∥∇F (Θr )∥2 , ∇F (Θr ), E[∆Θ (42) r 2 b rideal ∥2 | Θr ≤ B∥∇F (Θr )∥2 + σssca E ∥∆Θ , (43) then the actual SSCA direction satisfies b r ∥2 | Θr ≤ 2B∥∇F (Θr )∥2 + 2σ 2 + 2 Cssca c1 η 2 E(E − 1) σ 2 + c2 η 2 E 2 ζ 2 , E ∥∆Θ ssca
(44)
where η is the local SGD stepsize used in (21), and ηg is the server stepsize used in (24). Thus Assumption D.5 holds for 2 2 b r with updated constants B ← 2B and σagg ∆Θ ← 2σssca + 2 Cssca c1 η 2 E(E − 1) σ 2 + c2 η 2 E 2 ζ 2 . 2 Refinement: Separating SSCA Aggregation Noise and Drift. If desired, one can keep σssca (pure aggregation noise) and 2 2 the drift term separate in (38), by replacing σagg with σssca and adding an extra term proportional to ηg λmerge c1 η 2 E(E − 1)σ 2 + c2 η 2 E 2 ζ 2 . This yields the same qualitative conclusion: periodic aggregation (E > 1) increases the neighborhood radius through the drift term.
19
Boosting Multimodal Federated Learning via Chained Modality Optimization
E. Pseudo-code This appendix provides the pseudo-code of F ED MC HAIN, including the chain-of-modalities training schedule, the client-side L OCAL U PDATE, and the server-side SSCA aggregation used in each modality stage. Algorithm 1 F ED MC HAIN 1: Input: Clients C = {1, . . . , C} with local datasets {Di }i∈C ; global modality set M; modality-specific global parameters (m)
{ΘG }m∈M ; local learning rate η; local steps E; per-modality rounds {Rm }m∈M ; SSCA hyper-parameters κ (sparsification), K (number of clusters), π (threshold), λmerge (merge rate); regularizer weights λa , λc and temperature τ. (m) 2: Output: Trained {ΘG }m∈M . (m)
3: Initialize {ΘG }m∈M 4: for each modality m ∈ M do 5: for r = 1, 2, . . . , Rm do (m) 6: Server samples participating clients Sr ⊆ C (m)
▷ Modality-Chained schedule
(m)
Broadcast ΘG to all i ∈ Sr (m) 8: for each client i ∈ Sr in parallel do (m) (m) 9: ∆Θi ← L OCAL U PDATE(i, m, ΘG , η, E, λa , λc , τ ) (m) 10: Send ∆Θi and wi to server 11: end for b (m) ← SSCA({∆Θ(m) , wi } (m) ; κ, K, π) 12: ∆Θ i i∈Sr (m) (m) b (m) 13: ΘG ← ΘG + λmerge · ∆Θ 14: end for 15: end for 7:
▷ wi as in Eq. 10
(m)
16: L OCAL U PDATE(i, m, ΘG , η, E, λa , λc , τ ): (m)
(m)
17: Θi ← ΘG 18: for e = 1, 2, . . . , E do 19: Sample mini-batch Bi ⊂ Di 20: Freeze all modality branches except modality m 21: Update modality-m parameters by minimizing the stage objective in Eq. 4– 8 (m)
(m)
Θi ← Θi − η∇Θ(m) L(m) (Bi ) i 23: end for (m) (m) (m) 24: ∆Θi ← Θi − ΘG (m) 25: return ∆Θi
22:
(m)
, wi }i∈S (m) ; κ, K, π): (m) 27: Compute sparse sign vectors si = sign T (∆Θi , κ) for all i ∈ S 28: Cluster {si } into {C1 , . . . , CK } 29: Compute cluster consensuses {∆Θ̄(m,k) }K k=1 by weighted mean 30: Compute ρq , µq , and dq coordinate-wisely b (m) using Eq. 15– 16 31: Merge cluster consensuses into ∆Θ (m) b 32: return ∆Θ 26: SSCA({∆Θi
(m)
33: return {ΘG }m∈M
20
▷ Eq. 10 ▷ Eq. 11– 14