Pod-wise Multimodal Federated Learning for Heterogeneous IoT

Pod-wise Multimodal Federated Learning for Heterogeneous IoT Joy Okolo, Zihao Ding, and Jun Huang

arXiv:2611.03273v1 [cs.NI] 2 Nov 2026

Department of Electrical Engineering and Computer Science, South Dakota State University, Brookings, SD 57007, USA Email: {joy.okolo, zihao.ding}@jacks.sdstate.edu, jun.huang@sdstate.edu

Abstract—Federated learning (FL) is well suited to Internet of Things (IoT) sensing. However, devices carry different sensors and come online in periodic patterns, and as such, the sensors among the participants change from round to round. Under naive averaging, devices without a sensor overwrite its parameters, and a single-sensor device never learns how its sensor relates to the others. Existing methods either redesign local training for mismatched sensors (assuming always-available devices) or select clients without regard to their sensors. We propose Tier-aware Intra-pod Decoupled Encoders (TIDE). Specifically, each sensor has its own encoder and auxiliary head, whose parameters are merged only within the “pod” (i.e., the devices that hold the sensor), while the fusion layer and the task head remain shared by all devices. On three multimodal activity recognition datasets against eight baselines, TIDE reaches 78.06% on ActionSense, 7.63 points above the next method, is the most accurate on all three datasets at half its own communication, and at equal communication leads on ActionSense and is ahead within seed variation on the other two. Index Terms—Federated learning, Internet of Things, multimodal sensing, device heterogeneity, client scheduling, human activity recognition.

I. I NTRODUCTION Federated learning (FL) allows many devices to train one shared model without sending their private data to a server [1]– [7]. As such, it is a natural fit for the Internet of Things (IoT), where wearables and mobile devices continuously collect private data through several sensors at once [8]–[20]. As an example, a smartwatch may carry an accelerometer and a gyroscope, while a patch worn next to it carries only one of them. In real deployments, however, the sensors among the participants change from round to round, for two reasons. (1) Devices carry different sensors, and FL degrades when modalities are missing or mismatched [8], [21], [22]. (2) Devices train only when they are idle, charging, and on an unmetered network, so that participation follows periodic patterns [23]–[25]. As such, a sensor held by few devices may be absent for several rounds. Averaging its parameters with those of devices that never trained them then dilutes what its holders have learned (see Fig. 1). Moreover, a single-sensor device never observes how its sensor relates to the others in the first place. Existing heterogeneity-aware methods fall into two categories, namely, (i) redesigning local training for mismatched sensors and (ii) deciding which devices train. In the first category, Harmony [26] and FedHKD [27] train the sensor encoders and the fusion in separate stages, FedMVD [28]

round

round

round

1

1

1

devices online

2

2

2

faded: offline

3

3

3

plain averaging

intra-pod merging

trained

diluted

one copy has no camera

pod {1, 2}

pod {2}

each pod merges alone

trained

overwritten

no camera device

pod {1}

kept

keeps its value

diluted

trained

one copy has no IMU

pod {2}

pod {2, 3}

each pod merges alone

Fig. 1. Naive averaging and intra-pod merging over three rounds.

aligns representations across sensor views, and MCARN [29] learns a classifier for each sensor with a cross-modal alignment objective. In the second category, CLAMP [30] lets slow devices train fewer layers rather than excluding them, and FLPCS-MRF [31], PFHAR [32], and GraMFedDHAR [33] select or train clients for activity recognition. Note that the former are typically evaluated with devices that are always available (or sampled at random), whereas the latter ignore which sensors a device carries or simply assume a complete set. The sensors that a device carries, however, tell the server which parameters its update can inform. To see this, notice that only the devices holding a sensor train its encoder on real data, while the others merely return copies that never saw it. As such, the parameters of each sensor can be averaged within its “pod” (i.e., the set of devices that hold it), and the shared layers over all devices. Note that RELIEF [34] already aggregates the modality blocks of the fusion layer within the devices that hold each modality, and MA-CLAMP [35] aggregates each layer only over the clients that trained it. Furthermore, class prototypes [36], alignment across sensor views [28], and auxiliary classifiers for each modality [29] can carry knowledge to the devices that see a single sensor. To this end, we design Tier-aware Intra-pod Decoupled Encoders (TIDE). Specifically, unlike RELIEF, TIDE gives each sensor its own encoder and auxiliary classification head, and averages them only within the pod of that sensor, while the fusion layer and the task head remain shared by all devices. In this way, the multi-sensor devices of a pod train the encoders

that its single-sensor devices rely on. Class prototypes kept at the server further give every device the class centers of the whole federation, and a scheduler ranks the online devices in terms of speed, data size, and the rarity of their sensor combination. The contributions of this paper are fourfold. (1) We formulate multimodal FL under periodic availability as maximizing the accuracy reached within a communication budget. (2) We propose TIDE, which trains a classifier on exactly the sensors that each device carries and updates each encoder and its auxiliary head only within its pod. (3) We show in an incremental ablation that the auxiliary heads and intrapod merging account for most of the gain, that the prototype and cross-sensor losses change accuracy within seed variation, and that score-based selection matches random selection at K = 20 but trails it at small K on WEAR. (4) On three activity recognition datasets, TIDE is the most accurate at half its own communication, leads at equal communication on ActionSense and is ahead within seed variation on the other two, and has the second-smallest drop when a sensor is missing. II. S YSTEM M ODEL AND P ROBLEM F ORMULATION A. System Model

overwrites θ m entirely with copies from devices that do not hold sensor m. B. Problem Formulation Let Ψ[t] denote the total model traffic between the server and the devices in both directions up to round t (counting every training stage of multi-stage methods), and let Acc[t] denote the test accuracy of the global model evaluated in round t. The accuracy reached within a communication budget B is then  Acc@B = max Acc[t] : Ψ[t] ≤ B , (3) which compares methods at equal cost rather than at their own stopping points. Note that accuracy with every sensor present does not reveal how well the devices with fewer sensors are served. To this end, we also measure (1) the tier-restricted F1, i.e., the macro F1 over the test inputs restricted to the sensors of each less-equipped tier, pooled across those tiers, and (2) the worst-case drop ∆, i.e., the largest accuracy loss when any one sensor is removed from every test input. Now, under the availability of Eq. (1) (which the server does not control), the server chooses the selected sets {S[t]} and an aggregation rule G by solving (P) :

We consider a server and a set N of N devices that train a global model over rounds indexed by t, as illustrated in Fig. 1. The federation uses a set M of sensors. Device i holds a local dataset Xi and a subset Mi of the sensors, with the inputs of the other sensors set to zero. We stipulate the following conventions. Devices with the same sensors are said to form a tier. For each sensor m, we call the set of devices that hold it the pod Cm of sensor m. The global model consists of the sensor-specific parameters θ m of each sensor m and the shared parameters θ sh . Device i is online in round t when its availability indicator ai [t] equals one. Availability is periodic, namely,   ai [t] = I (t + ϕi ) mod Di < di , (1) where Di is the period, di is the number of online rounds per period, and ϕi is an offset that differs across devices (so that devices with the same pattern are not synchronized). Note that an always-on device has di = Di , and that the server knows none of Di , di , and ϕi . In round t, the server observes the set A[t] of online devices, selects a set S[t] of at most K of them, and broadcasts the global model θ[t]. Subsequently, each selected device i trains the model on Xi and returns θ i [t]. Standard averaging then sets X |Xi | P θ i [t]. (2) θ[t + 1] = ′ ′ i ∈S[t] |Xi | i∈S[t]

Note that only the members of Cm provide information about θ m . Under Eq. (2), however, θ m [t + 1] also averages the copies θ i,m [t] returned by the selected devices outside Cm , which have never seen real data for sensor m. Worse still, the sensors that the selected devices hold vary from round to round. As such, if no member of Cm is selected, Eq. (2)

max

{S[t]}, G

Acc@B

(4) s.t. (C1 ) : S[t] ⊆ A[t], |S[t]| ≤ K, ∀t,
(C2 ) : θ[t + 1] = G {θ i [t]}i∈S[t] , ∀t, where G maps the copies returned in a round to the next global model (Eq. (2) being one such rule). A solution to (P) should, in addition, keep the tier-restricted F1 high and ∆ small. III. TIDE: T IER - AWARE I NTRA - POD D ECOUPLED E NCODERS Problem (P) in Eq. (4) cannot be solved directly, since the server knows none of Di , di , and ϕi and hence cannot plan which sensors a round will represent. TIDE therefore fixes G so that θ m depends only on the members of Cm (Section III-C), which removes the overwriting of Section II-A in every round, and chooses S[t] by a heuristic score (Section III-A) that Section IV-B compares with uniform random selection. Figs. 2 and 3 show the overall framework and where each module runs. A. Client Selection The purpose of this module is to choose S[t] subject to (C1 ). Specifically, in every round, TIDE scores each online device i by ui = α vi + β ηi + γ ρi , (5) where α, β, and γ weight the speed score vi , the data score ηi , and the rarity score ρi of the device, respectively, namely, vi =

1 , 1 + τ̄i

ηi =

|Xi | , maxi′ |Xi′ |

ρi =

mini′ Ni′ . Ni

(6)

Here, τ̄i is the mean wall-clock time (in seconds) of the local training of device i over the last 10 rounds, with vi set to 0.5 before its first round, and Ni is the number of devices whose

1 Rarity-aware selection

online now 1

score, select

upload, merge in pods pod 1

selected

2

pod

t

0.77

camera

same sample

selected

t

2

adapter on stays local

device 1 has no camera

uploads class means of

all of 1

0.64

device with the camera only

embedding space of

2

below top-

camera 4

train on selected device

0.85

IMU + camera 3

3 Decoupled intra-pod merging

IMU

t

IMU 2

2 Unimodal heads and prototype guidance

device 4: not scored

t

zero input to

top-

1

next round:

and

camera, offline

2

go to the devices selected from

sent only after

warm-up rounds

Fig. 2. The framework of TIDE with its three modules.

sensors equal Mi , so that ρi is one for the devices with the rarest sensor combination. TIDE then selects the K online devices with the largest ui (or all of them if fewer than K are online).

Each sensor m has an encoder Em that maps its input to an embedding em , where a sensor that a device lacks (or that is dropped in training) receives an all-zero input. A fusion layer F maps the concatenated embeddings to a fused representation z, from which a task head H predicts the class. In addition, each sensor has an auxiliary head Hm that predicts the class from em alone, so that every encoder learns to be discriminative in its own right. As such, the parameters θ m are those of Em and Hm , and θ sh holds those of F and H. Note that each device also keeps a small residual adapter on z that it never uploads, which holds device-specific adjustments locally (the global model is evaluated without it). A selected device trains for a fixed number of local epochs on the loss (7)

where Ltask is the cross-entropy of the task head, Lprox is the squared distance to θ[t], and Lh is the cross-entropy of the auxiliary heads averaged over the sensors present. The crosssensor loss Lcs aligns the embeddings of the same sample across sensors. More precisely, for sensors m and m′ and a batch of J samples, the scaled similarity between sample j from sensor m and sample j ′ from sensor m′ is

ωjj ′ =

cos(em,j , em′ ,j ′ ) , κc

J

Lcs m,m′ = −

exp(ωjj ) 1 Xh log P ′ 2J j=1 j ′ exp(ωjj )

(9)

exp(ωjj ) i + log P , ′ j ′ exp(ωj j )

B. Unimodal Heads and Prototype-Guided Training

L = Ltask + µ Lprox + λcs Lcs + λp Lp + λh Lh ,

where em,j is the embedding of sample j from sensor m and κc is a temperature. The loss of the pair is then

(8)

and Lcs is the mean of Lcs m,m′ over all pairs of sensor branches of the model. Note that a zero-input branch has the same embedding for every sample in the batch, and as such, its pairs carry no alignment signal between two real sensors, so Eq. (9) aligns sensors only on devices that hold at least two. The prototype loss Lp pulls each fused representation toward the server prototype py [t] of its class y and away from the prototypes of the other classes. Specifically, over the classes that already have a prototype,
J exp cos(zj , pyj [t])/κp 1X p
, log P L = − (10) p J j=1 ′ y ′ exp cos(zj , py [t])/κ where zj and yj are the fused representation and the label of sample j, respectively, and κp is a temperature. Note that the prototypes are class means of z over all participants. Hence, Eq. (10) pulls a single-sensor device toward class centers shaped largely by devices with other sensors. In addition, on a multi-sensor device, one random sensor is zeroed in a fraction of the batches (so-called modality dropout), so that the model also sees inputs with a missing sensor. C. Decoupled Intra-Pod Merging This module serves as the aggregation rule G in (C2 ). Rather than Eq. (2), TIDE averages the parameters θ m of each sensor only over the selected devices in its pod, weighted by data size, namely, P i∈S[t]∩Cm |Xi | θ i,m [t] P , (11) θ m [t + 1] = i∈S[t]∩Cm |Xi |

TABLE I ACCURACY (%), D ROP ( POINTS ), AND C OMM (G I B) PER METHOD AND DATASET. A DAGGER MARKS A RUN NEAR CHANCE ; METHODS THAT CANNOT TAKE A MISSING SENSOR ARE EVALUATED WITH IT SET TO ZERO . S ELECTION ROWS USE K = 20 ON UCI-HAR, WHERE SCORE - BASED SELECTION IS TIDE ITSELF, AND 3 AND 6 ON ACTION S ENSE AND WEAR. A CELL MARKS THE BEST AND A CELL THE SECOND - BEST VALUE IN EACH BAND , EXCLUDING RUNS NEAR CHANCE ; – MARKS NO EVALUATION WITHIN THE BUDGET. Method (a) Comparison with baselines RELIEF [IoTJ’26] FedMVD [AAAI’25] FLPCS-MRF [Access’25] MCARN [TPAMI’24] FedHKD [SenSys’24] Harmony [MobiSys’23] PFHAR [TMC’26] GraMFedDHAR [ESWA’27] TIDE (b) Ablation of TIDE Core + Prototype alignment + Participation term + Cross-sensor alignment + Auxiliary heads + Intra-pod merging TIDE Evidence-weighted merging Uniform random selection Score-based selection, same K

1

device 1

@B↑

UCI-HAR @B/2↑ F1↑

Drop↓ Comm↓

92.59(±0.41) 88.99(±1.48) 87.39(±0.98) 82.63(±1.80) 84.38(±3.11) 75.72(±6.48) 89.87(±1.15) 22.42(±5.90)† 91.17(±1.03)

90.70 87.23 86.77 48.35 84.38 18.15 76.70 22.42 90.76

89.29 83.56 83.78 39.01 84.38 18.15 64.55 22.42 90.36

0.80 0.68 0.39 0.83 0.71 0.49 0.53 0.09† 0.80

21.90 38.41 59.30 6.19 32.20 44.70 59.09 5.22† 21.58

280.2 142.9 93.8 1523.1 27.7 118.8 340.9 41.9 75.2

90.01(±2.08) 89.84(±1.19) 89.96(±0.45) 90.16(±0.56) 91.30(±0.52) 92.10(±0.83) 91.17(±1.03) 91.72(±0.51) 91.17(±1.49) 91.17(±1.03)

90.01 89.54 89.96 89.95 91.30 91.36 90.76 91.13 91.17 90.76

89.04 88.49 88.81 89.23 90.14 90.35 90.36 90.39 90.38 90.36

0.69 0.71 0.73 0.73 0.79 0.82 0.80 0.82 0.76 0.80

33.30 35.01 31.90 32.47 21.58 16.14 21.58 16.14 27.61 21.58

75.1 95.6 78.1 79.0 70.3 112.3 75.2 111.3 63.5 75.2

3

device 3

Server running TIDE

local report: online, get , send ,

2

Final↑

Availability tracker

local

Intra-pod merger

report: online, get , send ,

Drop↓

Comm↓

70.43(±1.78) 70.30 68.57(±2.47) 66.49 40.16(±2.53) 40.16 60.61(±0.30) 45.87 8.78(±3.46)† 9.17 62.41(±1.47) 62.41 6.91(±2.43)† 6.25 5.11(±0.00)† 5.40 78.06(±0.93) 78.06

69.42 59.35 40.16 33.91 9.17 62.41 5.94 5.40 76.02

0.61 0.61 0.26 0.62 0.07† 0.64 0.01† 0.01† 0.71

41.58 28.82 30.76 7.84 2.04† 47.16 2.12† 0.19† 15.47

45.7 77.9 17.5 228.1 9.6 7.9 41.4 16.0 45.2

69.64(±2.03) 69.64 70.70(±1.00) 69.46 70.70(±2.74) 70.70 71.54(±0.87) 70.39 70.48(±2.62) 70.48 78.06(±0.93) 78.06 78.06(±0.93) 78.06 77.08(±0.20) 76.46 76.11(±1.56) 76.11 76.42(±1.91) 76.42

64.54 64.01 67.42 65.34 69.77 76.02 76.02 73.98 75.98 76.33

0.67 0.69 0.67 0.69 0.65 0.71 0.71 0.71 0.69 0.69

47.83 50.04 45.79 39.45 32.09 15.47 15.47 12.41 17.69 15.11

51.5 66.3 49.2 62.4 33.9 45.2 45.2 63.3 22.4 23.5

Scorer local

Topselector

device 2

ActionSense @B/2↑ F1↑

Final↑

report: online, get , send ,

Pod registry 1

: devices 1, 2

2

: devices 2, 3, 4

4

device 4

Prototype store sent after

rounds

local

offline in round : no messages

Fig. 3. The architecture of TIDE at the server and a device.

and keeps the previous θ m if no selected device holds sensor m. The shared parameters θ sh , on the other hand, are averaged over all selected devices as in Eq. (2). In this sense, each encoder is updated only by the devices with real data for it, so the multi-sensor members of a pod train the encoder that its single-sensor members rely on, while the shared layers still pool knowledge from all devices. Each selected device also returns the mean of z for each class in its data. The server averages these means into z̄y [t] for each class y and updates the associated prototype by py [t + 1] = 0.5 py [t] + 0.5 z̄y [t],

(12)

so that no single round can move a prototype abruptly. Note that the prototypes accumulate from the first round, whereas the server sends them to the devices only after a warmup of tw rounds. IV. P ERFORMANCE E VALUATION A. Experimental Setup We evaluate on three multimodal human activity recognition datasets. From UCI-HAR [37], every method uses the total acceleration (including gravity) and the angular velocity of a waist-mounted smartphone. Its training split is partitioned into 100 clients by a Dirichlet distribution over labels with concentration 0.5, and its official test split (whose subjects differ) is used for testing. ActionSense [38] holds eye-tracking,

@B↑

WEAR @B/2↑ F1↑

Drop↓

Comm↓

73.23(±1.04) 73.23 79.85(±0.74) 79.47 69.20(±0.95) 69.20 79.15(±0.18) 76.54 72.98(±1.02) 3.46 70.19(±1.03) 70.19 61.20(±0.28) 61.20 55.96(±2.61) 47.82 80.42(±0.77) 80.42

73.23 78.58 69.20 74.46 3.46 – 61.20 44.29 80.07

0.55 0.67 0.28 0.79 0.53 0.33 0.35 0.35 0.69

20.86 13.46 52.70 1.95 23.96 55.35 19.93 13.64 10.77

24.7 71.3 18.6 215.3 55.4 24.4 28.2 159.5 41.0

79.14(±1.30) 79.06 79.28(±1.00) 79.09 79.74(±1.04) 79.06 79.33(±0.43) 79.33 79.28(±0.55) 79.28 80.42(±0.77) 80.42 80.42(±0.77) 80.42 80.43(±1.10) 80.43 80.74(±0.31) 80.74 72.52(±2.09) 72.52

78.75 78.67 78.59 78.97 79.05 80.07 80.07 80.24 80.74 72.52

0.65 0.66 0.68 0.67 0.68 0.69 0.69 0.69 0.68 0.60

13.41 12.01 11.83 11.35 10.64 10.77 10.77 10.73 12.04 8.33

43.5 54.6 77.8 45.7 44.4 41.0 41.0 39.1 21.0 15.6

Final↑

@B↑

muscle-activity, and body-tracking data from kitchen tasks, whereas WEAR [39] holds inertial data and pre-extracted I3D features of first-person video from outdoor workouts. On these two, each subject is one client, and the windows of each subject are split by time, identically for every method, with the test parts pooled: the last 20% of the windows of each activity are used for testing, and every training window that shares a sample with a test window is removed. Unlike on UCI-HAR, the test subjects thus also appear in training, but every test window lies later in time. The clients form three tiers by their sensors. On UCI-HAR, 20 clients hold both sensors, 50 the accelerometer only, and 30 the gyroscope only. On ActionSense, three of the nine clients hold all three modalities, three eye tracking and muscle activity, and three muscle activity and body tracking. On WEAR, six of the 18 clients hold both modalities, six inertial data only, and six video only. Each device follows one of three availability patterns of the form of Eq. (1), assigned independently of its sensors. Specifically, “business-hours” devices are online 40 of every 60 rounds, “overnight” devices 20 of every 40, and the remaining devices always, with ϕi drawn uniformly at random per device. Every method uses the same assignment and selects only among online devices. FedMVD, GraMFedDHAR, and PFHAR use the scores of Eq. (5) plus a participation term of weight 0.15 that rewards recently selected devices, FLPCS-MRF uses its own Pareto-based selection, RELIEF and MCARN train every online client, and Harmony and FedHKD sample uniformly at random. Note that ActionSense and WEAR have fewer clients than K = 20 (hence the scores matter only on UCI-HAR), and that all clients share hardware without injected slowdowns (hence the speed score reflects workload rather than hardware). All methods train from scratch for at most 1000 rounds with five local epochs and batches of 32 samples, evaluate every ten rounds, and stop after ten evaluations without improvement (counted in the final stage of the staged Harmony and FedHKD). Each method uses whichever of the learning rates

B 0

50

100

150

200

Cumulative communication (GiB)

RELIEF FedMVD FLPCS-MRF

100 80 60 40 20 0

MCARN Harmony TIDE

B 0

30

60

90

120

Cumulative communication (GiB)

Test accuracy (%)

100 80 60 40 20 0

FedHKD Harmony PFHAR TIDE

Test accuracy (%)

Test accuracy (%)

RELIEF FedMVD FLPCS-MRF MCARN

100 80 60 40 20 0

RELIEF FedMVD FLPCS-MRF MCARN FedHKD

Harmony PFHAR GraMFedDHAR TIDE

B 0

25

50

75

100

Cumulative communication (GiB)

(a) UCI-HAR. (b) ActionSense. (c) WEAR. Fig. 4. Test accuracy versus cumulative communication in one run of each method, with the budget B dotted.

0.01 and 0.001 lets it train stably, namely, 0.01 for FedMVD, FLPCS-MRF, and MCARN and 0.001 for the others. TIDE sets α, β, and γ to 0.25, 0.35, and 0.25, respectively, tw to 10 rounds, µ to 0.01, λcs and λp to 0.1, λh to 0.3, and both temperatures κc and κp to 0.3, with a dropout rate of 0.2. PFHAR is reproduced from random initialization without its large-scale pretraining, and hence its results reflect its finetuning alone. GraMFedDHAR clips updates at norm 1.0 and adds noise with multiplier 1.0 (or 0.5 on WEAR, where 1.0 diverged). Table I uses the seeds 42, 123, and 2024, and the sensitivity runs use seed 42. In Table I, Final is the best evaluation of each run on the pooled test data with every sensor present. In Eq. (3), B is set to the mean total communication of TIDE, i.e., 75.2, 45.2, and 41.0 GiB on UCI-HAR, ActionSense, and WEAR, respectively, which is one point on the curves of Fig. 4 and compares every method at the cost that TIDE pays, and we also report the accuracy at B/2. Comm is the total communication until termination, converted from rounds at a constant cost per round within each run.1 Note that FLPCSMRF trains only fully equipped clients, and that our Harmony and FedHKD on UCI-HAR cannot take a missing sensor, so their F1 and Drop set it to zero. B. Experimental Results In Table I, TIDE is the most accurate method at termination on ActionSense with 78.06%, i.e., 7.63 points above RELIEF, and on WEAR with 80.42%, i.e., 0.57 points above FedMVD (within the seed variation) and 1.27 above MCARN. On UCIHAR, however, RELIEF leads by 1.42 points with 92.59%. At the budget B, TIDE has the highest mean accuracy on all three datasets, by 7.76 points on ActionSense and within the seed variation on UCI-HAR and WEAR. At B/2, it is the most accurate on all three, by 1.07 points over RELIEF on UCI-HAR, 6.60 over RELIEF on ActionSense, and 1.49 over FedMVD on WEAR. In our reproduction, GraMFedDHAR did not converge on UCI-HAR (at either noise multiplier) or ActionSense, and PFHAR, without its pretraining, and FedHKD stay near chance on ActionSense. Of the eight baselines, five thus train on every dataset, and the margins above are against these five. 1 This conversion is approximate for the staged Harmony and FedHKD.

Fig. 4 reveals that the method finishing above TIDE on UCI-HAR needs far more communication. Specifically, RELIEF, which trains every online client rather than K = 20, uses 280.2 GiB on UCI-HAR (3.7 times that of TIDE) for 1.42 points more, so part of this gap is due to participation size. On ActionSense and WEAR, no baseline finishes above TIDE on average, and MCARN, which trains every online client, uses 5.0 and 5.3 times its communication. Conversely, RELIEF communicates about as much as TIDE on ActionSense and less on WEAR, but stops at 70.43% and 73.23%. TIDE has the highest tier-restricted F1 on ActionSense (0.71) and is within 0.03 of the best on UCI-HAR, but trails MCARN on WEAR (0.69 against 0.79). Its worst-case drop is the second smallest on all three datasets among the methods that train, after MCARN, whose per-sensor classifiers make it the most robust. The ablation in Table I(b) starts from a “core” configuration with the scores of Eq. (5), the averaging of Eq. (2), the proximal term, and modality dropout. It turns out that the auxiliary heads and intra-pod merging account for most of the gain. Specifically, on top of all alignment losses, the heads add 1.14 points on UCI-HAR but cost 1.06 and 0.05 on ActionSense and WEAR, and intra-pod merging adds a further 0.80, 7.58, and 1.14 points, respectively. Together, they cut the worst-case drop from 32.47 to 16.14 points on UCI-HAR and from 39.45 to 15.47 on ActionSense, while on WEAR it changes from 11.35 to 10.77. Prototype alignment, the participation term, and the cross-sensor loss change the accuracy by at most 1.06 points, i.e., within the seed variation. Note that TIDE omits the participation term, whose removal changes selection only on UCI-HAR, where it lowers the accuracy by 0.93 points (within the seed variation). Weighting the devices of a pod by their head loss (i.e., evidence-weighted merging) gives no consistent gain either. Changing one of λh ∈ {0.1, 0.5, 1.0}, λp , λcs ∈ {0.05, 0.3}, or tw ∈ {0, 50} at a time keeps the accuracy within 1.74, 2.40, and 0.79 points of the default mean on UCI-HAR, ActionSense, and WEAR, respectively (within the seed variation on UCI-HAR and WEAR), and the worst-case drop at 14.5–18.8, 12.4–15.4, and 8.9–12.3 points, except that λp = 0.3 raises it to 36.3 on WEAR. On UCI-HAR, the data and rarity scores do not change during training, and as such, the 20 clients with both sensors

Gyro-only acc. (%)

Accuracy (%)

93 92 91 90 5

10

20

TIDE RELIEF

80

40

Selected devices K

60

100

Naive averaging

UCI-HAR ActionSense

60 40 20 5 10

20

35

Devices with gyroscope (%)

50

WEAR

TIDE, inertial Naive, inertial

TIDE, video Naive, video

15

Accuracy (%)

No rarity

TIDE − FedMVD (points)

TIDE Random

10 5 0 20

40

60

80

Availability (%)

100

70

65

60

0

100

Round

200

300

(a) Selection size. (b) Sensor scarcity. (c) Availability. (d) Training on WEAR. Fig. 5. Single runs with seed 42. (a) Accuracy versus K and (b) gyroscope-only accuracy versus the share of devices with a gyroscope on UCI-HAR. (c) Accuracy of TIDE minus FedMVD versus the share of each 60-round cycle that every device is online. (d) Per-sensor head accuracy on WEAR.

receive 50% to 60% of all selections, while 64 to 67 of the 100 clients are never selected. Interestingly, random selection gives the same accuracy within the seed variation, but a lower tier-restricted F1 (0.76 against 0.80) and a larger worst-case drop (27.6 against 21.6), since under score-based selection the encoders of the less-equipped devices are trained within their pods by the clients that hold both sensors. With few selected devices, however, the fixed scores keep choosing the same ones. Specifically, with K = 6 on WEAR, score-based selection trails random selection by 8.22 points, whereas with K = 3 on ActionSense the two give the same accuracy within the seed variation, and random selection stays within 1.95 points of full participation with 50% and 51% of its communication. Finally, Fig. 5 varies one condition at a time with seed 42. On UCI-HAR, all three selection rules reach 89.45% to 92.81% for K from 5 to 100, and no rule is consistently ahead. With a gyroscope on only 10% of devices, naive averaging reaches a gyroscope-only accuracy of 16.42% against 51.58% for TIDE, whereas RELIEF is the most accurate at 5%. When every device is online for the same share of each cycle, TIDE stays within 1.7 and 2.4 points of its best accuracy on UCIHAR and ActionSense down to 20%, and above FedMVD at every level on all three datasets (by 0.54 to 2.37, 10.11 to 16.09, and 0.03 to 2.62 points, respectively), although the two are level on WEAR at 20% and 40%. Also on WEAR, the head of each sensor ends within 2.25 points of its accuracy under naive averaging. V. C ONCLUSION In this paper, we have studied multimodal federated learning for IoT devices that carry different sensors and come online in periodic patterns. To this end, we proposed TIDE, which merges the encoder and auxiliary head of each sensor only within its pod (i.e., the devices that hold the sensor), while the fusion layer and task head remain shared by all devices. On three activity recognition datasets, TIDE is the most accurate at half its own communication, leads at equal communication on ActionSense, is ahead within seed variation on UCI-HAR and WEAR, and has the second-smallest worst-case drop when a sensor is missing, after MCARN. Note that its selection scores do not change during training. As such, when few devices are selected, it unfortunately trails random selection

on WEAR. Adapting the scores during training and evaluating on devices with different hardware are left for future work. R EFERENCES [1] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. 20th Int. Conf. Artif. Intell. Statist. (AISTATS), vol. 54, 2017, pp. 1273–1282. [2] B. Wu, Z. Ding, and J. Huang, “A Review of Continual Learning in Edge AI,” IEEE Transactions on Network Science and Engineering, vol. 13, pp. 6571–6588, 2026. [3] X. Che, D. Xu, and X. Wu, “StraightDP: Geometry-aware differential privacy for rectified-flow transformers,” arXiv preprint arXiv:2607.29100, 2026. [4] Z. Ding, B. Wu, and J. Huang, “SCALE: Sensitivity-Aware Federated Unlearning with Information Freshness Optimization for Mobile Edge Computing,” in Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS), 2026, to appear. [5] Z. Ding and J. Huang, “Toward Trustworthy Federated Unlearning for Mobile Autonomous Systems,” IEEE Network, pp. 1–9, 2026. [6] U. Pudasaini, Z. Ding, and J. Huang, “Securing smart agriculture with communication-efficient federated unlearning,” in 2026 IEEE 27th International Conference on High Performance Switching and Routing (HPSR), 2026, pp. 1–7. [7] Z. Ding, J. Huang, and L. Dong, “When unlearning fails: Reliable data deletion under post-training in agent networks,” arXiv preprint arXiv:2607.28829, 2026. [8] Y. Zhao, P. Barnaghi, and H. Haddadi, “Multimodal federated learning on IoT data,” in Proc. IEEE/ACM 7th Int. Conf. Internet-Things Des. Implement. (IoTDI), 2022, pp. 43–54. [9] B. Wu, J. Huang, and S. Yu, ““X of Information” Continuum: A Survey on AI-Driven Multi-Dimensional Metrics for Next-Generation Networked Systems,” IEEE Communications Surveys & Tutorials, vol. 28, pp. 5307–5344, 2026. [10] B. Wu, J. Huang, and Q. Duan, “Real-time intelligent healthcare enabled by federated digital twins with AoI optimization,” IEEE Network, vol. 40, no. 2, pp. 184–191, 2026. [11] B. Wu, J. Huang, Q. Duan, L. Dong, and Z. Cai, “Enhancing Vehicular Platooning With Wireless Federated Learning: A Resource-Aware Control Framework,” IEEE Transactions on Networking, vol. 34, pp. 1479–1494, 2026. [12] Z. Ding, J. Huang, Q. Duan, C. Zhang, Y. Zhao, and S. Gu, “A DualLevel Game-Theoretic Approach for Collaborative Learning in UAVAssisted Heterogeneous Vehicle Networks,” in 2025 IEEE International Performance, Computing, and Communications Conference (IPCCC), 2025, pp. 1–8. [13] Z. Ding, J. Huang, Y. Zhao, and Z. Cai, “Combating knowledge diversity and catastrophic forgetting in UAV-assisted collaborative vehicular learning: A game-theoretic approach,” ACM Transactions on Autonomous and Adaptive Systems, Jun. 2026, just Accepted. [14] Z. Ding, J. Huang, and J. Qi, “Learning to defend: A multi-agent reinforcement learning framework for Stackelberg security game in mobile edge computing,” in 2026 International Conference on Computing, Networking and Communications (ICNC), 2026, pp. 769–774.

[15] J. Huang, C.-C. Xing, and M. Guizani, “Power allocation for D2D communications with SWIPT,” IEEE Transactions on Wireless Communications, vol. 19, no. 4, pp. 2308–2320, 2020. [16] J. Huang, Y. Zhou, Z. Ning, and H. Gharavi, “Wireless power transfer and energy harvesting: Current status and future prospects,” IEEE Wireless Communications, vol. 26, no. 4, pp. 163–169, 2019. [17] X. Che, D. Xu, and X. Du, “Comparative analysis of formula and structure prediction from tandem mass spectra,” in Data Science: Foundations and Applications (PAKDD 2026), ser. Lecture Notes in Computer Science, vol. 16618. Singapore: Springer, 2027, pp. 215–227. [18] B. Wu, Z. Ding, and J. Huang, “Not all experience belongs in the weights: Component routing for self-improving GUI agents,” arXiv preprint arXiv:2610.01787, 2026. [19] B. Wu, J. Huang, and Y. Zhao, “REBASE: Device-cloud experience coherence for GUI agents across app updates,” arXiv preprint arXiv:2609.32130, 2026. [20] B. Wu and J. Huang, “CrystalMem: Elastic memory for selfevolving LLM agents via knowledge crystallization,” arXiv preprint arXiv:2608.00303, 2026. [21] Z. Ding, B. Wu, and J. Huang, “EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure,” arXiv preprint arXiv:2605.00733, 2026. [22] B. Wu, Z. Ding, and J. Huang, “PRISM: Exposing and Resolving Spurious Isolation in Federated Multimodal Continual Learning,” arXiv preprint arXiv:2605.01061, 2026. [23] Y. J. Cho, P. Sharma, G. Joshi, Z. Xu, S. Kale, and T. Zhang, “On the convergence of federated averaging with cyclic client participation,” in Proc. 40th Int. Conf. Mach. Learn. (ICML), vol. 202, 2023, pp. 5677– 5721. [24] B. Wu, J. Huang, and Y. Zhao, “From Alpha to Omega: Lifecycleaware forgetting defense in federated continual learning for planetary exploration,” in 2026 IEEE 46th International Conference on Distributed Computing Systems (ICDCS), 2026, pp. 1–11. [25] B. Wu and J. Huang, “Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems,” arXiv preprint arXiv:2604.20745, 2026. [26] X. Ouyang, Z. Xie, H. Fu, S. Cheng, L. Pan, N. Ling, G. Xing, J. Zhou, and J. Huang, “Harmony: Heterogeneous multi-modal federated learning through disentangled model training,” in Proc. 21st Annu. Int. Conf. Mobile Syst., Appl., Services (MobiSys), 2023, pp. 530–543. [27] X. Wang, H. Wang, F. Wu, T. Liu, Q. Cao, and L. Su, “Towards efficient heterogeneous multi-modal federated learning with hierarchical knowledge disentanglement,” in Proc. 22nd ACM Conf. Embedded Netw. Sensor Syst. (SenSys), 2024, pp. 592–605.

[28] M. Gao, H. Zheng, X. Feng, and R. Tao, “Multimodal fusion using multi-view domains for data heterogeneity in federated learning,” Proc. AAAI Conf. Artif. Intell., vol. 39, no. 16, pp. 16 736–16 744, 2025. [29] X. Yang, B. Xiong, Y. Huang, and C. Xu, “Cross-modal federated human activity recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5345–5361, 2024. [30] J. N. Okolo and J. Huang, “CLAMP: Client-oriented adaptive model pruning for stragglers in federated learning-enabled mobile edge computing,” in Proc. 41st ACM/SIGAPP Symp. Appl. Comput. (SAC), 2026, pp. 2048–2054. [31] B.-Q. Wang, F. Yang, Y. Wang, F. Zhao, Y.-F. Han, and Y.-P. Ma, “Federated learning for fall detection with multimodal residual fusion and Pareto-optimized client selection,” IEEE Access, vol. 13, pp. 54 148– 54 167, 2025. [32] Z. Zhang, D. Zhao, G. Zhu, X. Li, C. Li, B. Liu, Y. Li, and H. Ma, “PFHAR: Practically adopting multi-modal foundation model for human activity recognition through edge-cloud collaborative learning,” IEEE Trans. Mobile Comput., vol. 25, no. 8, pp. 12 764–12 779, 2026. [33] L. Halder, S. Palit, and T. Sen, “GraMFedDHAR: Graph based multimodal differentially private federated human activity recognition,” Expert Syst. Appl., vol. 334, p. 134463, 2027. [34] B. Wu, Z. Ding, and J. Huang, “RELIEF: Turning missing modalities into training acceleration for federated learning on heterogeneous IoT edge,” IEEE Internet Things J., 2026. [35] J. Okolo, “MA-CLAMP: Mask-aware client-oriented adaptive model pruning for stragglers in federated learning-enabled mobile edge computing,” SIGAPP Appl. Comput. Rev., vol. 26, no. 1, pp. 20–30, May 2026. [36] Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “FedProto: Federated prototype learning across heterogeneous clients,” Proc. AAAI Conf. Artif. Intell., vol. 36, no. 8, pp. 8432–8440, 2022. [37] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, “A public domain dataset for human activity recognition using smartphones,” in Proc. Eur. Symp. Artif. Neural Netw., Comput. Intell. Mach. Learn. (ESANN), Bruges, Belgium, 2013, pp. 437–442. [38] J. DelPreto, C. Liu, Y. Luo, M. Foshey, Y. Li, A. Torralba, W. Matusik, and D. Rus, “ActionSense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, 2022, pp. 13 800–13 813. [39] M. Bock, H. Kuehne, K. Van Laerhoven, and M. Moeller, “WEAR: An outdoor sports dataset for wearable and egocentric activity recognition,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 8, no. 4, pp. 1–21, 2024.

Related documents

• Opinion Dynamics-based Coalition Formation for Federated Learning in Heterogeneous IoT Systems#1014283
• Federated Learning: From Theory to Practice#1402938
• Federated Art Chase: Privacy Leakage and Data Poisoning in Federated Fine-Tuning#1356883
• Federated Art Chase: Privacy Leakage and Data Poisoning in Federated Fine-Tuning#1348768
• FedCSGA: Client Selection Based on Genetic Algorithm for Federated Learning#1527895