Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning

Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning Alireza Moayedikia

amoayedikia@swin.edu.au

Department of Business Technology and Entrepreneurship Swinburne University of Technology Melbourne, Australia

Alicia Troncoso Lora

atrolor@upo.es

arXiv:2608.07157v1 [cs.LG] 7 Aug 2026

Data Science & Big Data Lab Universidad Pablo de Olavide Seville, Spain

Editor:

Abstract Sub-model federated learning enables resource-constrained clients to train width-reduced versions of a global model, but existing methods allocate capacity according to device resources alone. A natural next step, allocating capacity according to each client’s data heterogeneity as estimated from the updates the server already observes, has been repeatedly suggested in recent literature. This paper asks whether that step is actually possible, using HAS-FL, an adaptive capacity-allocation framework, as a test case for a systematic analysis. Our findings are threefold. First, when validated against ground-truth label-distribution divergence on reproducible partitions, update-divergence estimates of client heterogeneity turn out to be dominated by capacity rather than by data. Across two corrected estimators, multiple datasets, and all seeds, the estimates correlate strongly and negatively with device capacity, and no data signal remains once capacity is controlled for. This is a previously undocumented confound that affects any method estimating client statistics from sub-model updates. Second, adaptive allocation has a hidden failure mode. When every client is capped below full width, the uncovered parameters stay at their random initialization and progressively corrupt the global model. A simple coverage guarantee removes the failure and also explains why uniform allocation collapses. Third, a matched-budget control settles what adaptivity contributes. Allocating capacity at random to the same average budget performs no differently from the heterogeneity-aware policy on both image benchmarks, and on the naturally partitioned text benchmark the adaptive policy is the weakest of the three allocation strategies while consuming the most capacity. Sub-model training remains valuable because it admits resource-constrained clients at a quadratically reduced cost, but the accuracy it gives up relative to full-model training is substantial, and what protects that accuracy is parameter coverage rather than allocation intelligence. Together these results change how heterogeneity-aware sub-model allocation should be understood. Its apparent benefits come from capacity budgeting and parameter coverage rather than from heterogeneity estimation, and future designs will need heterogeneity signals that can be separated from capacity effects. Keywords: federated learning, sub-model training, system heterogeneity, statistical heterogeneity, empirical analysis

1

Moayedikia and Troncoso Lora

1 Introduction Federated learning (FL) has emerged as a promising paradigm for privacy-preserving distributed machine learning, enabling multiple clients to collaboratively train a shared model without exchanging raw data (McMahan et al., 2017; Kairouz et al., 2021). This approach addresses critical privacy concerns in sensitive domains such as healthcare (Dayan et al., 2021; Rieke et al., 2020), mobile computing (Hard et al., 2019), and Internet of Things (IoT) applications (Nguyen et al., 2021; Khan et al., 2021). However, the practical deployment of FL systems faces two fundamental challenges that significantly hinder their effectiveness: statistical heterogeneity arising from non-independent and identically distributed (non-IID) data across clients, and system heterogeneity stemming from diverse computational capabilities among participating devices (Li et al., 2020a; Wang et al., 2024). Statistical heterogeneity manifests when local data distributions differ substantially across clients, a phenomenon commonly referred to as client drift (Karimireddy et al., 2020; Wang et al., 2026a). This heterogeneity can take multiple forms, including covariate shift where feature distributions vary while label distributions remain consistent, concept drift where the relationship between features and labels changes across clients, and quantity skew where data volumes differ dramatically (Li et al., 2022; Wang et al., 2026a). The consequences are severe: gradient updates computed on heterogeneous local data may point in conflicting directions, degrading the global model’s convergence rate and final accuracy. Furthermore, features learned by shared model components must simultaneously adapt to heterogeneous local distributions while serving the global collaborative objective, creating an inherent optimization conflict that complicates the training process (Wang et al., 2026b). System heterogeneity presents an orthogonal but equally critical challenge. In realistic FL deployments, participating devices exhibit significant disparities in computational power, memory capacity, and communication bandwidth (Diao et al., 2021; Caldas et al., 2018). Resource-constrained clients such as mobile phones, IoT sensors, and edge devices may be unable to train or even store full-sized neural network models, leading to their exclusion from the federation. This exclusion is not merely inefficient—it fundamentally biases the learned model toward data distributions present only on more capable devices, undermining the core premise of collaborative learning. Recent surveys emphasize that addressing these disparities in client computational capabilities alongside non-IID local data distributions is essential for both personalization of local models and generalization of the global model (Xu et al., 2026; Wang et al., 2024). Existing approaches have addressed these challenges largely in isolation. For statistical heterogeneity, methods such as FedProx (Li et al., 2020b) add proximal regularization to limit local model drift, while SCAFFOLD (Karimireddy et al., 2020) employs control variates to correct gradient bias. More recent work explores representation decoupling strategies that explicitly separate global-shared features from client-specific representations to mitigate interference between collaborative and personalized learning objectives (Wang et al., 2026b; Collins et al., 2021). Divergence-aware aggregation mechanisms have also been proposed to dynamically adjust client contributions based on semantic divergence, preventing critical task information from being diluted in highly heterogeneous scenarios (Wang et al., 2026b; Xu et al., 2026). For system heterogeneity, sub-model training approaches such as HeteroFL (Diao et al., 2021) and FjORD (Horvath et al., 2021) allow resource-constrained 2

Capacity Confounds in Adaptive Sub-model Federated Learning

clients to train smaller subsets of the global model. However, a fundamental limitation of such sub-model partitioning is that neurons not selected by any client remain unupdated during aggregation, potentially degrading global model quality (Xu et al., 2026). Despite this progress, a critical gap remains: no existing method jointly addresses both heterogeneity challenges in an adaptive and principled manner. Uniform sub-model allocation ignores the varying degrees of data heterogeneity across clients, while heterogeneity-aware optimization methods assume all clients can train full models. This paper investigates that gap through Heterogeneity-Aware Sub-model Federated Learning (HAS-FL), an adaptive capacity-allocation framework, and asks a question prior to any such design: can client data heterogeneity be estimated at all from the signals a submodel server observes? The motivating hypothesis is that clients with highly non-IID data require larger model capacity to capture their distinctive local patterns, while clients with data more representative of the global distribution can effectively train smaller sub-models. This intuition aligns with recent findings that adaptive mechanisms outperform static allocation strategies in heterogeneous environments (Wang et al., 2026b; Xu et al., 2026; Li et al., 2026). Our analysis, however, shows that operationalizing this hypothesis is far harder than it appears: the observable signal is dominated by capacity itself. We operationalize this insight through normalized gradient divergence—the squared norm of the difference between local and global updates, normalized by the magnitude of the global update—as an observable proxy for data heterogeneity that can be computed without additional privacy cost. Our skepticism is grounded in a recurring pattern: careful, budget-matched evaluation has repeatedly levelled or overturned the claimed advantages of adaptive machinery. In centralized learning, well-tuned SGD matches or exceeds adaptive optimizers (Wilson et al., 2017), and a large-scale neutral comparison of GAN variants found no consistent winner once tuning budgets were equalized (Lucic et al., 2018). In federated learning specifically, heterogeneity-aware re-evaluation on large-scale smartphone data erodes the reported gains of state-of-the-art aggregation and compression schemes (Yang et al., 2021), and an empirical study spanning some 1,500 configurations shows device and behavioral heterogeneity silently driving both model quality and fairness (Abdelmoniem et al., 2023). This paper applies the same discipline to adaptive sub-model allocation. The contributions of this work are threefold. First, we report a systematic analysis— validated against ground-truth label-distribution divergence on reproducible partitions— showing that update-divergence estimates of client heterogeneity are dominated by capacity heterogeneity: small-capacity clients diverge because their sub-models represent the task differently, not because their data is more skewed. The confound persists across two de-biased estimators, two datasets, and all seeds, and to our knowledge has not been documented before. Second, we identify and analyze a failure mode of capped sub-model allocation— parameters left uncovered by every client remain at their initialization and progressively degrade the global model—and introduce a coverage guarantee, together with coordinate-wise aggregation, that eliminates it; the same mechanism explains the previously unexplained collapse of uniform allocation. Third, through a unified multi-seed evaluation on CIFAR-10, EMNIST, and a naturally partitioned Shakespeare benchmark, we show that a matchedbudget random control performs no differently from heterogeneity-aware allocation on the image benchmarks and better than it on the text benchmark, and that sub-model training 3

Moayedikia and Troncoso Lora

as a whole gives up a substantial accuracy margin against full-model training in exchange for its reduced cost. The benefits that adaptive allocation does deliver come from capacity budgeting and parameter coverage rather than from heterogeneity awareness. An implementation of every method and experiment reported here, together with the configuration files needed to reproduce each table and figure, is available at https:// github.com/amoayedikia/hasfl.

2 Related Work Federated learning research has produced a rich landscape of methods addressing the challenges of distributed, privacy-preserving machine learning. We organize the most relevant prior work into four categories: federated optimization under heterogeneity, systemheterogeneous federated learning, personalized federated learning, and communicationefficient approaches. 2.1 Federated Optimization Under Statistical Heterogeneity The foundational FedAvg algorithm (McMahan et al., 2017) established the basic framework of local stochastic gradient descent (SGD) followed by model averaging, demonstrating empirical success in independent and identically distributed (IID) settings but exhibiting degraded performance under statistical heterogeneity. Subsequent work has sought to understand and mitigate this degradation through various mechanisms. FedProx (Li et al., 2020b) addresses client drift by adding a proximal term to the local objective, constraining local updates to remain close to the global model. While effective in moderately heterogeneous settings, this approach uniformly penalizes all clients regardless of their actual divergence from the global distribution. SCAFFOLD (Karimireddy et al., 2020) takes a different approach by introducing control variates that estimate and correct for the drift between local and global gradients. This method achieves variance reduction and improved convergence rates but requires additional communication of control variates and assumes clients can maintain state across rounds. Recent advances have explored more sophisticated mechanisms for handling heterogeneity. FedRDA (Wang et al., 2026b) proposes representation decoupling combined with divergence-aware aggregation, explicitly separating global-shared and client-specific feature representations while dynamically adjusting client contributions based on semantic divergence measured through task vectors. This work demonstrates that static weighting mechanisms based on sample size cannot adequately address scenarios where critical task information may be diluted across highly heterogeneous clients. Similarly, FLex&Chill (Li et al., 2026) introduces temperature scaling during local training, where lower temperatures amplify gradient magnitudes and accelerate convergence under non-IID conditions. Their theoretical analysis shows that gradient magnitude scales inversely with temperature, providing a complementary optimization technique to our capacity-based approach. The challenge of non-stationary data in federated settings has also received attention. Wang et al. (2026a) provide a comprehensive treatment distinguishing between covariate shift, concept drift, and client drift, offering formal definitions and visualization of how these phenomena manifest in the loss landscape. Their analysis reveals that existing methods often conflate these distinct sources of heterogeneity, potentially applying inappropriate corrections. Our work differs from these approaches by focusing on capacity 4

Capacity Confounds in Adaptive Sub-model Federated Learning

allocation rather than gradient correction or aggregation weighting. Rather than uniformly constraining or correcting all clients, HAS-FL adaptively determines how much model capacity each client should train based on observable heterogeneity measures, addressing both statistical and resource heterogeneity simultaneously. 2.2 System-Heterogeneous Federated Learning System heterogeneity in federated learning arises from the diverse computational capabilities of participating devices. Early approaches addressed this through client selection strategies that exclude slower devices, but this introduces bias toward data present only on more capable clients. HeteroFL (Diao et al., 2021) pioneered sub-model training by allowing clients to train width-reduced versions of the global model according to their computational capacity. The method aggregates sub-models through channel-wise averaging, enabling heterogeneous participation while maintaining a single global model. FjORD (Horvath et al., 2021) extended this approach using ordered dropout, training nested sub-models that share parameters hierarchically. Both methods demonstrate that partial model training can enable broader participation without catastrophic accuracy loss. However, these approaches employ fixed or random sub-model allocation strategies that do not account for data heterogeneity. Recent analysis reveals a fundamental limitation: when sub-model partitioning is employed, neurons that are not selected by any client during a communication round remain unupdated, potentially degrading global model quality over time (Xu et al., 2026). This observation motivates the heterogeneity-aware allocation strategy we study, which would give clients with more distinctive data larger sub-models; Section 4.4 examines whether the required estimate is obtainable in the first place. FLSC (Xu et al., 2026) addresses model heterogeneity through semantic-guided aggregation with cascading heterogeneity compensation, where lightweight broad models are sequentially adapted for knowledge transfer across non-IID clients. While effective, this approach requires consistent model structures within client clusters and involves additional computational overhead from the cascading compensation mechanism. PFedGate (Chen et al., 2023) introduces learnable gating mechanisms for structured block sparsity, allowing personalized sparse architectures that adapt to local data characteristics. This architectural personalization approach is complementary to our capacity-based allocation. The method we analyze, HAS-FL, extends these schemes by coupling sub-model allocation to heterogeneity estimation—the natural next step suggested in this literature, and the one this paper puts to the test. Rather than treating system and data heterogeneity as separate concerns, the method uses gradient divergence as a signal to inform capacity decisions, allocating larger sub-models to clients whose data appears to deviate more from the global distribution—the premise Section 4.4 tests directly. A parallel systems literature treats device heterogeneity as a scheduling and selection problem while keeping every client’s model full-size. Oort selects participants by a utility score that deliberately blends statistical utility (local training loss) with system utility (measured latency) (Lai et al., 2021), and PyramidFL refines this joint profiling at finer granularity (Li et al., 2022); AutoFL schedules participants and per-device execution settings for energy efficiency (Kim & Wu, 2021), and FedBalancer selects informative samples under device-speed-aware deadlines (Shin et al., 2022). Because these systems train equal-size models, their published results are untouched by sub-model effects—but their utility signals 5

Moayedikia and Troncoso Lora

already fuse data and capacity by design. A second group sizes the sub-model itself from training-derived signals: FedMP chooses per-worker pruning ratios with a capability-driven bandit (Jiang et al., 2023), Hermes prunes channels by local-data importance (Li et al., 2021), FIARSE composes each client’s sub-model from parameter-magnitude importance up to the device budget (Wu et al., 2024), and FedLPS explicitly learns per-client sparse ratios from the superimposed effect of device capability and non-IID data (Xue et al., 2025). By contrast, FedRolex rotates which slice trains purely by round index—a capacity-only rule involving no data-derived signal (Alam et al., 2022). Our analysis (Section 4.4) shows why this distinction matters: once sub-model width varies with device capacity, update-based signals conflate data and capacity, and the data component becomes unidentifiable. 2.3 Personalized Federated Learning Personalized federated learning (pFL) aims to produce models tailored to individual clients rather than a single global model. This paradigm acknowledges that in highly heterogeneous settings, a one-size-fits-all model may underperform compared to client-specific adaptations. Per-FedAvg (Fallah et al., 2020) applies model-agnostic meta-learning (MAML) principles to federated learning, treating the global model as an initialization for rapid local adaptation. Ditto (Li et al., 2021) maintains separate global and local models with regularization connecting them, while FedRep (Collins et al., 2021) explicitly partitions the model into shared representation layers and personalized classifier heads. These methods demonstrate the value of structured personalization but require all clients to train full-sized models. More recent work has explored sophisticated representation learning strategies for personalization. pFedSD (Jin et al., 2022) employs self-knowledge distillation to prevent catastrophic forgetting during local adaptation, while pFedKT (Zhang et al., 2023a) introduces dual-transfer mechanisms combining hypernetworks with contrastive learning for improved knowledge transfer. FedCP (Zhang et al., 2023b) separates feature information through conditional policies, and FedRDA (Wang et al., 2026b) uses variational contrastive log-ratio upper bound (vCLUB) to minimize mutual information between global and personalized representations, enforcing clean separation. Prototype-based methods offer another avenue for personalization under heterogeneity. FedProto (Tan et al., 2022) aggregates class prototypes rather than model parameters, enabling knowledge sharing even across architecturally heterogeneous clients. ProtoFedGAN (Sun et al., 2026) extends this concept by combining prototype learning with generative adversarial networks for enhanced representation learning in non-IID settings. Heterogeneity-aware allocation would, in principle, provide implicit personalization by giving clients with distinctive data larger sub-models; our results indicate that the premise this rests on does not hold under capacity heterogeneity. This capacity-based personalization is complementary to explicit representation separation techniques and could be combined with them in future work. 2.4 Communication Efficiency and One-Shot Federated Learning Communication efficiency is critical for practical FL deployment, as iterative communication between clients and server can become prohibitively expensive. Various approaches have been proposed to reduce communication overhead while maintaining model quality. Gradient compression techniques reduce the size of transmitted updates through quantiza6

Capacity Confounds in Adaptive Sub-model Federated Learning

tion (Alistarh et al., 2017), sparsification (Aji & Heafield, 2017), or low-rank approximation (Vogels et al., 2019). These methods are orthogonal to sub-model training and can be combined with HAS-FL for additional communication savings. One-shot federated learning represents the extreme end of communication efficiency, aiming to complete model training in a single communication round. Recent surveys (Zhang et al., 2026) categorize approaches into statistical information-based methods, knowledge distillation-based methods, and advanced aggregation techniques. FedDistr (Ma et al., 2026) provides theoretical foundations for when one-shot FL can succeed, introducing the concept of entangled coefficients to measure distribution overlap between clients. Their analysis shows that near-disentangled distributions enable one-shot learning with probabilistic utility guarantees. However, oneshot FL fundamentally trades convergence guarantees for communication efficiency—the performance ceiling depends entirely on the diversity and accuracy of single-round contributions without possibility of iterative refinement (Zhang et al., 2026). Sub-model training occupies a middle ground: by enabling resource-constrained clients to participate at reduced width, it lowers per-round computation while retaining the multi-round refinement that ensures convergence. This efficiency is obtained without sacrificing the iterative optimization that distinguishes federated learning from ensemble methods; whether it can additionally be made heterogeneity-aware is the question we take up.

3 The Method Under Study This section specifies HAS-FL, the adaptive sub-model method whose premises this paper tests. We present it in full because the analysis in Section 4 depends on the exact form of its heterogeneity estimator, its allocation rule, and its aggregation scheme; the design follows the natural extension of resource-only sub-model training that recent literature has repeatedly suggested. We begin by formalizing the problem setting in Section 3.1, then introduce our heterogeneity measure in Section 3.2, describe the complete algorithm with its adaptive allocation mechanism in Section 3.3, and finally analyze communication and computation complexity in Section 3.4. 3.1 Problem Formulation Consider a federated learning system with N clients collaboratively training a shared model. Each client i ∈ [N ] := {1, 2, . . . , N } possesses a local dataset Di drawn from a potentially distinct distribution Pi , and the global objective is to minimize the aggregate loss: N

1 X min F (w) := fi (w), N w∈Rd

where

fi (w) := Eξ∼Pi [ℓ(w; ξ)],

(1)

i=1

with ℓ(w; ξ) denoting the loss function evaluated at parameters w on data sample ξ, and d representing the model dimensionality. In practical federated deployments, clients exhibit heterogeneity along two orthogonal dimensions. First, resource heterogeneity manifests as varying computational and memory capacities across devices. We formalize this through a capacity constraint pmax ∈ (0, 1] for each client i, representing the maximum fraction of i model parameters that client i can store and update. Second, data heterogeneity arises from 7

Moayedikia and Troncoso Lora

non-identical local distributions Pi ̸= Pj for i ̸= j, causing local objectives fi to diverge from the global objective F . The sub-model training paradigm addresses resource heterogeneity by having each client i train only a subset of parameters determined by a binary mask mi ∈ {0, 1}d . Following the nested (ordered) sub-model construction of HeteroFL (Diao et al., 2021) and FjORD (Horvath et al., 2021), we instantiate mi as a structured width mask : a client with capacity pi trains the leading ⌈pi Cℓ ⌉ channels of every layer ℓ, together with the corresponding input connections, so that sub-models of increasing capacity are nested. Here pi := ∥mi ∥0 /d denotes the fraction of parameters updated by client i. The masked local update becomes: (t+1)

wi

(t)

(t)

(t)

= w(t) − η · mi ⊙ ∇fi (mi ⊙ w(t) ; ξi ),

(2)

where ⊙ denotes element-wise multiplication and η is the learning rate. The central challenge we address is: How should the sub-model allocation {pi }N i=1 be determined when clients exhibit both resource constraints and heterogeneous data distributions? Existing approaches either ignore data heterogeneity when allocating sub-models or treat resource and data heterogeneity independently. The premise under test is that these two factors cannot be treated independently: the clients that would benefit most from additional capacity are not necessarily those that possess it. The allocation rule we study, formalized in Equation (6), distributes capacity according to heterogeneity estimates while respecting the constraints pmax . i 3.2 Heterogeneity Measure To enable adaptive sub-model allocation, we require a quantitative measure of each client’s data heterogeneity that is both theoretically grounded and practically computable. The measure under study is the normalized gradient divergence. Definition 1 (Normalized Gradient Divergence) For client i at round t, the normalized gradient divergence is defined as: (t)

Hi

:=

∥∇fi (w(t) ) − ∇F (w(t) )∥2 , ∥∇F (w(t) )∥2 + ϵ

(3)

where ϵ > 0 is a small constant ensuring numerical stability. This measure captures how much client i’s local gradient deviates from the global gra(t) dient direction, normalized by the global gradient magnitude. Clients with Hi ≈ 0 have (t) data distributions aligned with the population, while large Hi indicates significant distributional shift. Importantly, this quantity directly appears in convergence bounds for federated optimization (Li et al., 2020c; Karimireddy et al., 2020), making it theoretically meaningful for guiding algorithmic decisions. In practice, the server cannot directly com(t) pute Hi as it lacks access to local gradients. However, it can be estimated from the observable client updates. After each communication round, the server receives updates (t) (t+1) ∆i := wi − w(t) from participating clients. The estimator under study is: (t)

(t)

Ĥi

:=

(t)

¯ (t) )∥2 ∥mi ⊙ (∆i − ∆ , (t) ¯ (t) ∥2 + ϵ ∥m ⊙ ∆ i

8

(4)

Capacity Confounds in Adaptive Sub-model Federated Learning

¯ (t) denotes the coordinate-wise aggregated update computed by the server (Algowhere ∆ rithm 1, line 18) over the set S (t) of clients participating in round t. This estimator leverages the relationship between gradients and parameter updates under gradient descent and serves (t) as a practical server-side proxy for Hi ; its absolute scale depends on the learning rate and the number of local steps, but only the relative ordering of clients enters the allocation rule. (t) Restricting the computation to the coordinates in mi is essential: without this restriction, every coordinate a low-capacity client did not train would contribute the full magnitude of ¯ (t) to its divergence, systematically inflating the heterogeneity estimates of low-capacity ∆ clients and creating a spurious feedback loop between capacity and estimated heterogeneity. Whether any update-based statistic can recover data heterogeneity under capacity heterogeneity is an empirical question; Section 4.4 answers it with ground-truth validation. To reduce variance in the heterogeneity estimates, we employ exponential moving average smoothing: (t) (t−1) (t) H̃i := β · H̃i + (1 − β) · Ĥi , (5) where β ∈ [0, 1) is the smoothing parameter controlling the trade-off between respon(0) siveness and stability, and H̃i = 1 for all clients as the initial estimate before any observations are available. Larger β values produce more stable estimates but slower adaptation to changing heterogeneity patterns. To prevent unbounded accumulation of the exponential moving averages over long training runs, we apply periodic normalization every Tnorm rounds, rescaling all estimates to maintain bounded values while preserving relative ordering among clients. 3.3 Adaptive Sub-model Allocation The central mechanism of HAS-FL, and the object of our analysis, is an adaptive allocation rule that assigns sub-model capacities from both resource constraints and estimated data heterogeneity. Our allocation rule is motivated by the following principle: clients with higher data heterogeneity require larger sub-model capacity to adequately capture their local data distribution, subject to their resource constraints. Definition 2 (HAS-FL Allocation Rule) Given resource constraints {pmax }N i i=1 , hetero(t) N geneity estimates {H̃i }i=1 , base capacity pmin ∈ (0, 1), and sensitivity parameter γ > 0, the adaptive allocation for client i at round t is: ! (t) H̃ (t) pi := min pmax , pmin + γ · (t)i , (6) i H̄ + ϵ where H̄ (t) := N1

(t) j=1 H̃j is the average heterogeneity across all clients.

PN

The allocation rule in (6) has several properties that motivate it a priori. First, the minimum operator ensures that resource constraints are never violated, respecting the physical limitations of each device. Second, the term pmin guarantees a baseline participation level for all clients, preventing any client from being effectively excluded. Third, the heterogeneity(t) dependent term γ · H̃i /H̄ (t) allocates additional capacity proportionally to relative heterogeneity, ensuring that clients deviating most from the global distribution receive larger 9

Moayedikia and Troncoso Lora

sub-models. The normalization by H̄ (t) makes the allocation robust to the absolute scale of heterogeneity measures across different training stages. The sensitivity parameter γ controls the strength of heterogeneity-aware allocation. When γ = 0, all clients receive identical base capacity pmin (subject to resource constraints), recovering uniform allocation. As γ increases, the allocation becomes more responsive to heterogeneity differences. The rule in (6) alone harbors a subtle failure mode. If every client—including those able to train the full model—is capped below full width, the coordinates beyond the largest allocation are trained by no client: they remain at their random initialization yet continue to participate in inference, and the global model degrades progressively as the trained coordinates drift away from them. Related observations about unselected parameters have been made for static sub-model schemes (Xu et al., 2026; Alam et al., 2022). HAS-FL therefore augments the allocation rule with a coverage guarantee: after each adaptation step, clients whose resource constraint equals the largest constraint in the federation are assigned their full capacity pmax . This ensures that every parameter of the global model i keeps receiving updates whenever such a client is sampled, while leaving the heterogeneityaware allocation of all remaining clients untouched. Section 4.3 quantifies the effect: without the guarantee, accuracy collapses; with it, HAS-FL matches static full-coverage allocation at lower average capacity. The failure mode admits an exact statement. Proposition 3 (Frozen coordinates under capped allocation) Suppose every alloca(t) tion satisfies pi ≤ p̄ < 1 for all clients i and rounds t, with the nested width masks of Algorithm 2. Then every coordinate outside the leading p̄-fraction slice of its layer satisfies [w(T ) ]j = [w(0) ]j for every T : it remains at its random initialization for the entire run while participating in every forward pass. (t)

Proof By nestedness, [mi ]j = 0 for every client i whenever j lies outside the leading p̄ ¯ (t) ]j = 0 slice of its layer. The coordinate-wise aggregation rule (Algorithm 1, line 18) sets [∆ whenever no participant covers j, hence [w(t+1) ]j = [w(t) ]j for all t, and induction from w(0) completes the argument. The proposition turns the empirical collapse into a structural fact: an allocation policy avoids permanently frozen coordinates only if the union of client masks keeps covering every coordinate, and under nested masks the cheapest way to enforce this is to keep the largest-capacity clients at full width—precisely the coverage guarantee. We now present the complete HAS-FL algorithm as outlined in Algorithm 1, which integrates the heterogeneity estimation and adaptive allocation mechanisms into a federated optimization procedure. The algorithm operates in rounds, with periodic updates to heterogeneity estimates and sub-model allocations. The algorithm proceeds as follows. At each round t, the server samples a subset S (t) of clients to participate (line 5). For each selected client i, the server extracts the nested (t) sub-model consisting of the leading pi fraction of each layer’s channels (line 7). The client performs K local SGD steps on its local data (lines 9–13) and returns the parameter update (t) ∆i to the server (lines 14–15). The server aggregates the received updates coordinate-wise (lines 17–18): each parameter is averaged over exactly the clients that trained it, weighted by their local sample counts. This replaces the scalar normalization used in earlier sub-model 10

Capacity Confounds in Adaptive Sub-model Federated Learning

schemes; coordinates covered by few clients are neither diluted nor over-amplified, which is essential when allocation fractions differ across clients. Every Tadapt rounds, the server updates heterogeneity estimates and reallocates sub-model capacities (lines 19–29). The (t) heterogeneity estimate Ĥi is computed from the divergence between client i’s update and the aggregated update, restricted to the coordinates that client i actually trained (line 21), then smoothed using an exponential moving average (line 22). Sub-model allocations are updated according to the allocation rule in Definition 2, followed by the coverage guarantee that keeps the highest-capacity clients at full width (lines 25–28). Algorithm 2 details the nested sub-model extraction procedure. Figure 1 provides a graphical overview of the HAS-FL framework. The server maintains a global model w(t) and distributes client-specific sub-models based on adaptive allocations (t) {pi }. Each client performs local training on its sub-model and returns updates ∆i to the server. The adaptation module, shown on the right, periodically estimates heterogeneity from update divergence, applies exponential smoothing, and computes new allocations that account for both resource constraints and data heterogeneity. As illustrated in the example, Client 1 has high heterogeneity (H1 = 2.1) but limited resources (pmax = 0.3), so it receives 1 the maximum allowed capacity. Client 2 has low heterogeneity (H2 = 0.3); the base rule would assign it a small sub-model, but since it is the highest-capacity client in the federation, (t) the coverage guarantee keeps it at full width (p2 = 0.8) so that every parameter of the global model continues to receive updates. Client N has moderate heterogeneity (HN = 1.2) (t) and receives its maximum allowed capacity (pN = 0.5 = pmax N ).

m1 ⊙ w

∆1

Global Model w(t)

Adaptation Module

Server

Estimate Ĥi

∆2

(t)

∆N mN ⊙ w

m2 ⊙ w

···

(t)

Smooth H̃i

(t+1)

Client 1

Client 2

pmax = 0.3 1

pmax = 0.8 2

pmax = 0.5 N

H1 = 2.1 (high)

H2 = 0.3 (low)

HN = 1.2 (med)

(t)

p1 = 0.3

(t)

Client N

Allocate pi

(t)

p2 = 0.8

pN = 0.5

Figure 1: Overview of the HAS-FL framework. Figure 2 illustrates how the adaptive allocation responds to different combinations of resource constraints and data heterogeneity. The plot shows the allocation rule pi = min(pmax , pmin + γHi /H̄) with the parameters used in our experiments, pmin = 0.4 and i γ = 0.25 (heterogeneity shown relative to the population average). Each curve represents a different resource constraint level. The allocated capacity increases linearly with heterogeneity until hitting the resource constraint, at which point it plateaus. This demonstrates the 11

Moayedikia and Troncoso Lora

Algorithm 1 HAS-FL: Heterogeneity-Aware Adaptive Sub-model Federated Learning Require: Number of clients N , resource constraints {pmax }N i i=1 , total rounds T , local steps K, learning rate η, adaptation interval Tadapt , smoothing parameter β, base capacity pmin , sensitivity γ, client sampling rate s Ensure: Trained global model w(T ) 1: Initialize global model w(0) (0) 2: Initialize heterogeneity estimates H̃i ← 1 for all i ∈ [N ] (0) 3: Initialize allocations pi ← pmax for all i ∈ [N ] i 4: for t = 0, 1, . . . , T − 1 do 5: S (t) ← SampleClients(N, s) ▷ Sample subset of clients 6: for each client i ∈ S (t) in parallel do (t) (t) (t) 7: mi ← ExtractSubmodel(d, pi ) ▷ Nested width mask: leading pi fraction of each layer (t,0) (t) 8: wi ← mi ⊙ w(t) ▷ Extract sub-model 9: for k = 0, 1, . . . , K − 1 do (t,k) 10: Sample mini-batch ξi ∼ Di (t,k) (t,k) (t,k) 11: gi ← ∇ℓ(wi ; ξi ) (t,k+1) (t,k) (t) (t,k) 12: wi ← wi − η · m i ⊙ gi 13: end for (t) (t,K) (t) 14: ∆i ← wi − mi ⊙ w(t) ▷ Compute masked update (t) 15: Send ∆i to server 16: end for 17: Server aggregation: P (t) (t) P (t) ni [mi ]j [∆i ]j (t) ¯ (t) ]j ← i∈S 18: [∆ if i [mi ]j > 0, else 0 ▷ Coordinate-wise; ni : P (t) i∈S (t)

ni [mi ]j

sample count ¯ (t) 19: w(t+1) ← w(t) + ∆ 20: if t mod Tadapt = 0 then 21: for each client i ∈ S (t) do (t) (t) (t) ¯ (t) )∥2 /(∥m(t) ⊙ ∆ ¯ (t) ∥2 + ϵ) 22: Ĥi ← ∥mi ⊙ (∆i − ∆ i (t)

(t−T

)

(t)

H̃i ← β · H̃i adapt + (1 − β) · Ĥi 24: end for P (t) 25: H̄ (t) ← N1 N i=1 H̃i 26: for each client i ∈   [N ] do (t) (t+1) (t) + ϵ) , p + γ · H̃ /( H̄ 27: pi ← min pmax min i i 28: end for (t+1) 29: pi ← pmax for all i with pmax = maxj pmax i j i 30: end if 31: end for 32: return w(T )

23:

▷ Periodic adaptation

12

▷ Coverage guarantee

Capacity Confounds in Adaptive Sub-model Federated Learning

Algorithm 2 ExtractSubmodel: Nested Sub-model Extraction Require: Layer widths {Cℓ }L ℓ=1 , capacity fraction p Ensure: Binary mask m ∈ {0, 1}d 1: for each layer ℓ = 1, . . . , L do 2: cℓ ← ⌈p · Cℓ ⌉ ▷ Number of retained channels 3: Set mj = 1 for the parameters of the leading cℓ channels of layer ℓ (and their input connections from the retained channels of layer ℓ−1); mj = 0 otherwise 4: end for 5: return m = (m1 , . . . , md ) ▷ Masks are nested: p ≤ p′ implies m ⊆ m′

Allocated Capacity pi

key principle: high-heterogeneity clients receive larger sub-models when resources permit, while resource-constrained clients are capped regardless of their heterogeneity level. 1

0.5 pmax = 1.0 i pmax = 0.6 i pmax = 0.3 i pmin = 0.4

0

0

0.5

1 1.5 2 Heterogeneity Hi

2.5

3

Figure 2: HAS-FL allocation rule under different resource constraints.

3.4 Communication and Computation Complexity We analyze the communication and computation costs of HAS-FL compared to standard federated learning approaches. In each round, client i receives a sub-model of expected size (t) pi · d parameters and sends back an update of the same size. The total communication per round is: X (t) X (t) (t) Ccomm =2 pi · d = 2d pi . (7) i∈S (t)

i∈S (t)

Compared to full-model training where Cfull = 2|S (t) | · d, HAS-FL achieves communication P (t) reduction factor of |S (t) |/ i∈S (t) pi . When average allocation p̄ < 1, this represents significant bandwidth savings, particularly important for resource-constrained edge devices. (t) The local computation cost for client i scales with sub-model size as O(pi · d · K · B), where K is the number of local steps and B is the batch size. Resource-constrained clients with small pmax thus perform proportionally less computation, enabling their participation in the i training process. The adaptation module introduces additional server-side computation for 13

Moayedikia and Troncoso Lora

(t)

heterogeneity estimation and allocation updates. Computing Ĥi requires O(d) operations per client, and the allocation rule evaluation is O(N ). Since adaptation occurs every Tadapt rounds, the amortized overhead per round is O((d · |S| + N )/Tadapt ), which is negligible compared to the aggregation cost of O(d · |S|). Table 1 instantiates these costs for the three architectures used in our experiments. Because width slicing shrinks both the input and output channels of every layer, per-client cost scales quadratically rather than linearly with p: a quarter-width client trains only 6– 7% of the full model’s parameters and multiply–accumulate operations, and a half-width client roughly 25%. This quadratic law is what makes sub-model training attractive for edge deployments—the weakest capacity tier pays an order of magnitude less computation and upload bandwidth than the strongest—but it is also the root of the estimation problem studied in Section 4.4: sub-model updates differ across capacity tiers by an order of magnitude in dimensionality and norm, dwarfing any data-driven variation. Table 1: Per-client cost as a function of sub-model width p. Model

p

Params (M)

MACs/sample (M)

Upload (MB)

CIFAR-10 CNN

0.25 0.50 0.75 1.00

0.21 0.82 1.83 3.25

10.0 39.2 87.4 154.9

0.82 3.26 7.32 13.00

EMNIST CNN

0.25 0.50 0.75 1.00

0.21 0.83 1.85 3.28

7.5 29.7 66.6 118.2

0.85 3.31 7.40 13.10

Shakespeare LSTM

0.25 0.50 0.75 1.00

0.06 0.22 0.47 0.82

4.1 16.1 35.9 63.6

0.24 0.86 1.88 3.30

4 Experiments We conduct extensive experiments to evaluate HAS-FL across diverse federated learning scenarios. Our evaluation addresses three primary research questions: (1) Does heterogeneityaware adaptive allocation improve convergence and final accuracy compared to resource-only allocation strategies? (2) Can client-level data heterogeneity be estimated from the submodel updates the server observes, and do the resulting allocations track it? (3) How does HAS-FL compare against state-of-the-art federated optimization methods designed for statistical heterogeneity? The following subsections present our experimental setup, followed by systematic investigation of each research question with evidence drawn from multiple benchmark datasets. 4.1 Experimental Setup We evaluate HAS-FL on two benchmark datasets that span different scales and heterogeneity characteristics. The CIFAR-10 dataset (Krizhevsky, 2009) consists of 60,000 color 14

Capacity Confounds in Adaptive Sub-model Federated Learning

images of size 32 × 32 pixels distributed across 10 mutually exclusive classes, with 50,000 images designated for training and 10,000 for testing. We partition the training data across clients using a Dirichlet distribution with concentration parameter α = 0.3, creating a challenging non-IID setting where each client possesses a highly skewed subset of the label space. This dataset serves as our primary benchmark due to its widespread adoption in federated learning research, enabling direct comparison with existing methods while allowing controlled experimentation with varying degrees of label skew. The EMNIST dataset comprises 697,932 grayscale images of handwritten characters distributed across 62 classes including digits, uppercase letters, and lowercase letters. This represents a 14-fold increase in data volume and 6-fold increase in class count compared to CIFAR-10, testing scalability and performance under increased classification complexity. We apply the same Dirichlet partitioning strategy (α = 0.3) to maintain consistency with our CIFAR-10 experiments while evaluating whether HAS-FL’s mechanisms generalize to substantially larger and more complex classification tasks. To complement the label-skew heterogeneity induced by Dirichlet partitioning, we additionally evaluate on a benchmark with natural client partitions: next-character prediction on the complete works of Shakespeare, following the LEAF benchmark’s construction (Caldas et al., 2018). Each client corresponds to one speaking role, so heterogeneity arises from genuine stylistic and lexical differences between characters rather than from a synthetic sampling procedure; the twenty roles with the largest amount of dialogue form the federation. Text is modeled at the character level (vocabulary of 95 symbols) with sequences of 80 characters predicting the following character, and the model is a two-layer LSTM with 256 hidden units per layer and an 8-dimensional character embedding. Sub-model extraction for the LSTM slices the leading fraction of hidden units within each gate block, preserving the recurrent structure of nested sub-models; the embedding and output bias are shared in full by every client. Training uses K = 1 local epoch, η = 0.1, and gradient-norm clipping at 5; all other protocol elements are identical to the image experiments. Table 2 summarizes the key characteristics of these datasets. Table 2: Datasets used in the experiments. Dataset

Task

Classes

CIFAR-10 EMNIST Shakespeare

Image Image Text

10 62 95

Train Size

Test Size

Input Dim

50,000 10,000 697,932 116,323 20 role-clients

32 × 32 × 3 28 × 28 × 1 80 chars

The federated learning setup consists of N = 20 clients, with 10 clients randomly sampled to participate in each communication round. Resource heterogeneity is simulated by assigning each client a maximum capacity constraint pmax ∈ {0.25, 0.5, 0.75, 1.0} according i to a realistic distribution where resource-constrained devices are more prevalent, yielding an average maximum capacity of p̄max = 0.525. Training proceeds for T = 200 rounds with learning rate η = 0.01, decay factor 0.995, and K = 5 local epochs for CIFAR-10 (reduced to K = 1 for EMNIST following standard practice for large federated datasets). Local optimization uses SGD with momentum 0.9, weight decay 10−4 , and gradient-norm clipping at 10. All experiments are repeated over three random seeds and reported as mean ± standard deviation. The HAS-FL hyperparameters are set to pmin = 0.4, γ = 0.25, and 15

Moayedikia and Troncoso Lora

β = 0.9, with adaptation occurring every Tadapt = 5 rounds following a warmup period of 10 rounds, and the coverage guarantee of Section 3.3 enabled. We set the normalization interval to Tnorm = 20 rounds. We compare HAS-FL against seven baseline algorithms spanning sub-model and fullmodel approaches. Among sub-model methods, HeteroFL (Diao et al., 2021) assigns submodel widths based solely on resource constraints without considering data heterogeneity, FjORD employs ordered dropout with knowledge distillation to train nested sub-networks, and the Uniform baseline assigns fixed allocation pi = min(0.5, pmax ) to all clients rei gardless of their characteristics. A Random-budget control assigns each client a random time-varying allocation drawn to match HAS-FL’s average capacity budget, isolating the contribution of heterogeneity-awareness from mere adaptivity. Among full-model methods, FedAvg (McMahan et al., 2017) serves as the standard baseline, SCAFFOLD (Karimireddy et al., 2020) employs control variates to correct client drift, and FedProx (Li et al., 2020b) adds proximal regularization to constrain local updates. All methods use identical data partitions, resource constraints, participation ratios, and optimization hyperparameters to ensure fair comparison; in particular, SCAFFOLD is evaluated under the same N = 20 client configuration as every other method. 4.2 Heterogeneity-Aware vs. Resource-Only Allocation Our first research question examines whether incorporating data heterogeneity estimates into capacity allocation decisions improves performance compared to strategies that consider only resource constraints. We address this by comparing HAS-FL against two resourceonly approaches: the Uniform baseline, which assigns identical capacity to all clients within their resource limits, and FjORD, which assigns capacity tiers randomly from the available options without considering client-specific data characteristics. This comparison isolates the contribution of heterogeneity-aware allocation by controlling for overall resource utilization. Table 3 presents the comparison, and its message is central to this paper’s thesis: heterogeneity-aware allocation does not separate from random tier assignment. On CIFAR-10, HAS-FL reaches 74.5% ± 5.6 final accuracy while FjORD reaches 75.5% ± 3.8—statistically indistinguishable means—even though FjORD uses less average capacity (0.41 vs. 0.52). On EMNIST the pattern repeats: 77.7% ± 4.6 for HAS-FL against 79.6% ± 3.3 for FjORD at 0.40 average capacity. Both coverage-preserving methods dominate the Uniform baseline, and the margin widens dramatically with task complexity: Uniform ends at 54.5% ± 2.3 on CIFAR-10 but collapses to 9.4% ± 3.8 on EMNIST—a fall of 70–75 points from its own peak of 82.2% ± 1.4, to barely above the 1.6% chance level of the 62-class task—confirming that the decisive design element is coverage, not the allocation criterion. Following recent recommendations on evaluation practice, we emphasize final rather than peak accuracy and report mean ± standard deviation over three seeds; seed-toseed variation is substantial (up to six percentage points) because each seed redraws both the data partition and the device capacity assignment, which underscores the importance of multi-seed reporting. The convergence trajectories in Figure 3 reveal qualitatively different behaviors behind these aggregates. HAS-FL and FjORD improve throughout training, with FjORD oscillating visibly because its random tier assignment changes each client’s sub-model width 16

Capacity Confounds in Adaptive Sub-model Federated Learning

Table 3: Sub-model allocation strategies (mean ± std over three seeds). Dataset

Algorithm

Best Acc.

Final Acc.

Avg. p

CIFAR-10

HAS-FL FjORD Uniform

76.7 ± 4.9% 78.6 ± 3.9% 63.6 ± 1.6%

74.5 ± 5.6% 75.5 ± 3.8% 54.5 ± 2.3%

0.52 0.41 0.44

EMNIST

HAS-FL FjORD Uniform

82.0 ± 1.8% 84.0 ± 0.5% 82.2 ± 1.4%

77.7 ± 4.6% 79.6 ± 3.3% 9.4 ± 3.8%

0.52 0.40 0.44

from round to round. The Uniform baseline follows a characteristic rise-and-decline pattern on every seed: it reaches the low-to-mid 60% range by round ∼110 and then deteriorates steadily, ending 6–11 points below its own peak (65.4%→54.2%, 62.4%→52.3%, and 62.9%→56.9% across the three seeds). This is the stale-parameter failure mode analyzed in Section 4.3: under the fixed allocation pi = min(0.5, pmax ) no client ever trains the outer i half of each layer’s channels, and the untrained coordinates progressively corrupt inference as the trained portion of the network drifts away from them. Notably, the collapse is milder here than under naive aggregation (the same baseline previously fell below 40% on CIFAR-10): coordinate-wise aggregation partially contains the damage, but only an allocation policy that preserves coverage eliminates it. On EMNIST even this containment fails: after peaking above 80%, Uniform decays to near-chance accuracy with the test loss settling at ln 62 ≈ 4.1—the signature of outputs degraded to random guessing—showing that the severity of the coverage failure grows with the number of classes. CIFAR-10

EMNIST

Test Accuracy (%)

Test Accuracy (%)

80 60 40 HAS-FL FjORD Uniform

20 0

0

50

100

150

80 60 40

0

200

HAS-FL FjORD Uniform

20

0

50

100

150

200

Communication Round

Communication Round

(b) EMNIST

(a) CIFAR-10

Figure 3: Convergence of sub-model allocation strategies on CIFAR-10 (a) and EMNIST (b). The Uniform baseline’s failure has a precise mechanistic explanation. Under the fixed allocation pi = min(0.5, pmax ), no client ever trains more than half of each layer’s channels: i the remaining channels are updated by no client, stay at their random initialization, and continue to participate in every forward pass. As training progresses, the trained portion 17

Moayedikia and Troncoso Lora

of the network drifts away from these stale coordinates and their contribution becomes increasingly destructive—gradual degradation on CIFAR-10 and outright collapse on the 62-class EMNIST task, where the mismatch compounds faster. The coverage-guarantee ablation in Section 4.3 reproduces this failure mode in a controlled setting and shows that preserving full parameter coverage eliminates it. Which parameters an allocation policy leaves untrained thus matters as much as how much total capacity it uses. 4.3 Ablation: The Coverage Guarantee To isolate the effect of the coverage guarantee introduced in Section 3.3, we compare three configurations under identical data partitions, seeds, and training budgets: (i) full HAS-FL; (ii) HAS-FL with the coverage guarantee disabled, i.e., the allocation rule of Definition 2 applied verbatim; and (iii) HeteroFL, whose static resource-only allocations provide full coverage by construction. Table 4 reports the results over three seeds on both image datasets. Table 4: Coverage-guarantee ablation (mean ± std over three seeds). Dataset

Configuration

Best Acc.

Final Acc.

Peak→final ∆

Avg. p

CIFAR-10

HAS-FL, guarantee on HAS-FL, guarantee off HeteroFL (full coverage)

76.7 ± 4.9% 57.2 ± 2.4% 76.3 ± 4.9%

74.5 ± 5.6% 40.4 ± 10.2% 75.0 ± 5.0%

−2.2 −16.8 −1.2

0.52 0.48 0.56

EMNIST

HAS-FL, guarantee on HAS-FL, guarantee off

82.0 ± 1.8% 82.4 ± 2.3%

77.7 ± 4.6% 70.6 ± 9.4%

−4.4 −11.8

0.52 0.47

Disabling the guarantee leaves the largest allocation in the federation at 0.58–0.69 rather than 1.0, so roughly a third of every layer’s channels are trained by no client. The consequence separates cleanly into two distinct effects, and the two datasets isolate them. The first is a capacity effect, visible in peak accuracy. On CIFAR-10, best accuracy falls from 76.7% to 57.2%: a two-thirds-width network simply cannot represent the task as well. On EMNIST, by contrast, peak accuracy is unchanged (82.4% without the guarantee versus 82.0% with it), because a two-thirds-width model already saturates that task. Lost coverage costs accuracy only to the extent that the stranded capacity was needed. The second is a stability effect, visible in the gap between peak and final accuracy, and it appears on both datasets regardless of whether capacity binds. With the guarantee, models end training within a few points of their peak (−2.2 on CIFAR-10, −4.4 on EMNIST, and −1.2 for fully covered HeteroFL). Without it, the decay grows to −16.8 and −11.8 respectively, and the seed-to-seed spread of final accuracy roughly doubles (±10.2 and ±9.4, against ±5.6 and ±4.6). This is the mechanism of Proposition 3 acting over time: coordinates frozen at initialization drift ever further from the trained portion of the network, and their contribution to inference becomes progressively more destructive. The magnitude is also strongly seed-dependent, which is worth stating plainly. Removing the guarantee costs 19, 34, and 49 points of final accuracy on the three CIFAR-10 seeds, and 0.3, 4.7, and 16.1 points on EMNIST. A single-seed ablation on EMNIST could therefore have shown almost no effect at all; only the three-seed protocol reveals that the failure is severe but intermittent, triggered by particular combinations of data partition and capacity assignment. 18

Capacity Confounds in Adaptive Sub-model Federated Learning

Taken together, the ablation supports a more precise claim than “coverage is necessary.” Uncovered parameters always destabilise training, and they additionally cost accuracy when the task needs the width they occupy. This also reconciles the Uniform baseline’s behaviour across benchmarks: it caps every client at half width, stranding both more parameters and—critically—the outer channels of the classifier, which is why it collapses outright on the 62-class EMNIST task (Section 4.2) yet survives on Shakespeare, where the output projection is shared in full by every client (Section 4.6). 4.4 Can Update Divergence Measure Data Heterogeneity? Our second research question is whether the divergence estimates of Section 3.2 actually measure data heterogeneity. Because the Dirichlet partitions are generated from fixed seeds, we can reconstruct each client’s exact data and compute a ground-truth heterogeneity measure: the total-variation (TV) distance between the client’s label distribution and the global label distribution. Table 5 correlates the final smoothed estimates H̃i with this ground truth, with the capacity constraints pmax , and—via partial correlations—with the ground truth i after linearly removing the capacity effect. The answer is unambiguous, and negative. The estimates correlate overwhelmingly with capacity (r between −0.72 and −0.84 on all six runs across both datasets), and once capacity is controlled for, no consistent data signal remains: r(H̃, TV | pmax ) is near zero or negative throughout. Consequently the allocations themselves do not track true heterogeneity (r(p, TV | pmax ) ≈ 0). The mechanism is structural rather than an estimator artifact—we verified that both the coordinate-restricted and the common-core variants of the estimator exhibit the coupling. A client that permanently trains a quarter-width sub-model produces updates whose shared coordinates compensate for its missing capacity, so its divergence from the aggregate reflects its width, not its data; heavily skewed clients may even diverge less, since their few-class local tasks are easier and their gradients shrink faster. We conclude that in system-heterogeneous federations, update divergence is confounded by capacity regardless of how carefully it is measured—a caution that applies to any method estimating client statistics from sub-model updates. Whether the adaptive component contributes beyond its capacity budget is therefore tested directly by the random-budget control, which assigns each client a random time-varying width scaled to match HAS-FL’s average capacity. On CIFAR-10 the two are statistically indistinguishable: random-budget reaches 74.6% ± 5.1 final accuracy against HAS-FL’s 74.5% ± 5.6 (mean per-seed difference 0.1), while using slightly less average capacity (0.49 vs. 0.52). The result replicates on EMNIST (77.4% ± 5.7 for random-budget vs. 77.7% ± 4.6 for HAS-FL, again at lower capacity), so it is not an artifact of a single task. Heterogeneity-aware allocation therefore adds nothing over random allocation at an equal budget on either dataset—the direct experimental counterpart to the confound: an estimator that cannot recover data heterogeneity cannot inform allocation, so acting on it is no better than chance. Figure 4 illustrates the temporal evolution of heterogeneity estimates and the resulting per-client allocations. The left panels show the average heterogeneity estimate H̄ exhibiting a characteristic sawtooth pattern on both datasets, rising gradually between normalization intervals before being reset to maintain bounded values. This periodic normalization every 20 rounds prevents unbounded accumulation while preserving the relative ordering of client 19

Moayedikia and Troncoso Lora

Table 5: Correlations of the final estimates H̃i with ground-truth label divergence (TV) and device capacity (pmax ). r(H̃, TV) r(H̃, pmax ) r(H̃, TV|pmax ) r(p, TV|pmax )

Run CIFAR-10, s42 CIFAR-10, s43 CIFAR-10, s44 EMNIST, s42 EMNIST, s43 EMNIST, s44

−0.44 −0.21 0.21 −0.43 −0.24 −0.02

−0.72 −0.72 −0.84 −0.75 −0.72 −0.81

−0.60 −0.47 −0.40 −0.26 −0.28 0.10

0.02 −0.36 −0.46 0.07 0.05 0.19

CIFAR-10 2

1.5

0.53 0.52 0.51

1 0

50

Mean cap p̄max HAS-FL p̄

0.5

150

0.49 200

100

Communication Round

(a) CIFAR-10

Avg. heterogeneity H̄ (green) / allocation p̄ (purple)

Avg. heterogeneity H̄ (green) / allocation p̄ (purple)

heterogeneity estimates that drives allocation decisions. The heterogeneity values remain well-bounded throughout training, oscillating between 1.0 and approximately 1.8–2.0 on both datasets. The average allocation p̄ remains stable near 0.5 throughout training.

EMNIST 0.56 2 0.54

1.5

0.52 HAS-FL p̄ Mean cap p̄max

1 0

50

100

150

0.5 200

Communication Round

(b) EMNIST

Figure 4: Average heterogeneity estimate and allocation during training on CIFAR-10 (a) and EMNIST (b). Figure 5 shows the final per-client allocations. The allocations are visibly differentiated— but, consistent with Table 5, the differentiation follows the capacity tiers rather than the underlying data: low-capacity clients accumulate high divergence estimates and sit at their caps, while high-capacity clients appear homogeneous and receive reduced widths (subject to the coverage guarantee). The practical takeaway is an honest one: adaptive allocation operates at a lower average capacity than static resource-only assignment while matching its accuracy, but the analysis above attributes this to disciplined capacity budgeting under a coverage guarantee rather than to successful heterogeneity targeting. 20

Capacity Confounds in Adaptive Sub-model Federated Learning

EMNIST

pmax i pi (allocated)

1

Width Multiplier (p)

Width Multiplier (p)

CIFAR-10

0.5

0

pmax i pi (allocated)

1

0.5

0 0

2

4

6

8

10 12 14 16 18

0

2

4

6

8

10 12 14 16 18

Client ID

Client ID

(a) CIFAR-10

(b) EMNIST

Figure 5: Final per-client allocations pi (blue) vs. resource constraints pmax (gray). i

4.5 Comparison with State-of-the-Art Methods Our third research question examines how HAS-FL compares against state-of-the-art federated optimization methods designed specifically for statistical heterogeneity. We compare against FedAvg as the standard baseline, SCAFFOLD which employs control variates to correct client drift, and FedProx which adds proximal regularization to constrain local updates. This comparison contextualizes HAS-FL’s contribution relative to full-model approaches that address data heterogeneity through optimization techniques rather than capacity allocation. Table 6 places the sub-model methods against full-model training, and the corrected implementation revises the picture substantially. The full-model ceiling is far higher than previously reported: FedAvg reaches 88.2% ± 0.6 final accuracy on CIFAR10. Training at roughly half capacity therefore carries a real, quantifiable cost—HAS-FL (74.5% ± 5.6) gives up about 13 percentage points relative to that ceiling in exchange for a ∼4× reduction in per-client computation and upload volume (Table 1). The random-budget control lands exactly on HAS-FL (74.6% ± 5.1 at slightly lower capacity), confirming at the system level that the adaptive component contributes nothing beyond its capacity budget. FedProx with the standard µ = 1.0 collapses to 66.4% ± 1.0 on CIFAR-10 but is healthy on EMNIST (83.9% ± 1.3): the proximal term throttles local progress precisely under the severe ten-class label skew, and is comparatively benign on the milder 62-class partition— itself a small heterogeneity-sensitivity signal. On EMNIST the full-model ceiling is closer: SCAFFOLD (87.2% ± 0.3, with the corrected control-variate configuration) and FedAvg (85.5% ± 1.7) lead HAS-FL (77.7%) by roughly eight points rather than the thirteen seen on CIFAR-10, so the capacity–accuracy trade-off softens as the task grows. SCAFFOLD is the strongest full-model baseline on both datasets once its control variates are run under vanilla local SGD (86.3% ± 1.2 on CIFAR-10, 87.2% ± 0.3 on EMNIST); under the momentum-based optimizer shared by the other methods it diverges to chance accuracy, so we report it in its canonical configuration and note the incompatibility explicitly. 21

Moayedikia and Troncoso Lora

Table 6: Comparison with full-model methods on CIFAR-10 and EMNIST. Dataset

Algorithm

Type

Best Acc.

Final Acc.

Avg. p

CIFAR-10

FedAvg SCAFFOLD FedProx HAS-FL Random-budget

Full-model Full-model + CV Full-model + prox Sub-model (adaptive) Sub-model (random)

88.5 ± 0.5% 86.7 ± 0.9% 66.4 ± 1.0% 76.7 ± 4.9% 75.9 ± 4.9%

88.2 ± 0.6% 86.3 ± 1.2% 65.5 ± 2.0% 74.5 ± 5.6% 74.6 ± 5.1%

1.00 1.00 1.00 0.52 0.49

EMNIST

FedAvg SCAFFOLD FedProx HAS-FL Random-budget

Full-model Full-model + CV Full-model + prox Sub-model (adaptive) Sub-model (random)

87.4 ± 0.3% 87.5 ± 0.2% 85.0 ± 0.0% 82.0 ± 1.8% 82.4 ± 2.1%

85.5 ± 1.7% 87.2 ± 0.3% 83.9 ± 1.3% 77.7 ± 4.6% 77.4 ± 5.7%

1.00 1.00 1.00 0.52 0.49

80

60

EMNIST 100

FedAvg (full) SCAFFOLD (full) FedProx (full) HAS-FL Random-budget FjORD HeteroFL Uniform

0.4

0.6

Final Accuracy (%)

Final Accuracy (%)

CIFAR-10

0.8

50

0

1

Average per-round capacity p̄

SCAFFOLD (full) FedAvg (full) FedProx (full) FjORD HAS-FL Random-budget Uniform

0.4

0.6

0.8

1

Average per-round capacity p̄

(a) CIFAR-10

(b) EMNIST

Figure 6: Accuracy–capacity frontier: final accuracy versus average per-round capacity (mean ± std over three seeds).

Figure 7 shows the corrected convergence trajectories. On CIFAR-10, FedAvg’s fullmodel curve rises smoothly and saturates near 88%, visibly separated from the sub-model band throughout training; HAS-FL matches its early pace up to roughly round 15 before flattening near 75%, and FedProx climbs slowly but monotonically toward 65%, its proximal term damping both oscillation and progress. On EMNIST every method exceeds 80% within five rounds and the separation between full- and half-capacity training narrows to roughly eight points, with FedProx recovering to within two points of FedAvg by the end of training. The loss trajectories in Figure 8 add a calibration perspective. FedAvg attains and holds the lowest loss on both datasets (about 0.45 on CIFAR-10 and 0.35 on EMNIST). FedProx’s loss decreases monotonically to the very end of training on both datasets, consistent with an under-converged but stable proximal trajectory. HAS-FL reaches its loss minimum early (about 1.24 on CIFAR-10 and 0.50 on EMNIST) and then drifts upward even as its 22

80

Test Accuracy (%)

Test Accuracy (%)

Capacity Confounds in Adaptive Sub-model Federated Learning

60 40

HAS-FL FedAvg SCAFFOLD FedProx

20 0

0

50

100 Round

150

80 60 40

HAS-FL SCAFFOLD FedProx FedAvg

20 0

200

0

50

100 Round

150

200

(b) EMNIST

(a) CIFAR-10

Figure 7: Convergence comparison with full-model methods on CIFAR-10 (a) and EMNIST (b).

accuracy stays flat—a mild miscalibration that coordinate-wise averaging of heterogeneous sub-models does not remove. 2.5

Test Loss

2 Test Loss

4

HAS-FL FedAvg SCAFFOLD FedProx

1.5 1

HAS-FL SCAFFOLD FedProx FedAvg

3 2 1

0.5

0 0

50

100 Round

150

200

0

50

100 Round

150

200

(b) EMNIST

(a) CIFAR-10

Figure 8: Test-loss trajectories on CIFAR-10 (a) and EMNIST (b). Taken together, the comparison reframes what sub-model training buys. It does not match full-model accuracy—the corrected baselines show a consistent double-digit gap to FedAvg on CIFAR-10—but it purchases the participation of resource-constrained devices at a quadratically reduced compute and communication cost. Within the sub-model regime, the decisive design choices are parameter coverage and total capacity budget rather than allocation intelligence: HAS-FL, FjORD, and the random-budget control are statistically indistinguishable, while the coverage-violating Uniform policy fails catastrophically. Methods addressing statistical heterogeneity through full-model optimization and methods addressing system heterogeneity through sub-model training therefore occupy different points 23

Moayedikia and Troncoso Lora

on an accuracy–cost frontier (Figure 6), and combining gradient correction with coveragepreserving sub-model training remains an open direction. 4.6 Natural Partitions: Shakespeare The Dirichlet partitions used above impose heterogeneity by construction. We therefore repeat the allocation comparison on the naturally partitioned Shakespeare benchmark, where each client is one speaking role and heterogeneity arises from genuine stylistic differences. Table 7 reports next-character prediction accuracy over three seeds. Table 7: Allocation strategies on the naturally partitioned Shakespeare benchmark (mean ± std over three seeds). Algorithm

Best Acc.

Final Acc.

Avg. p

HAS-FL FjORD Uniform

29.5 ± 2.2% 33.5 ± 0.9% 35.0 ± 0.5%

25.5 ± 0.8% 32.6 ± 1.3% 34.4 ± 0.6%

0.53 0.41 0.44

The ordering inverts relative to the image benchmarks, and does so in the direction least favourable to adaptive allocation. HAS-FL is the weakest of the three policies despite consuming the largest average capacity, and it is the only one whose accuracy degrades between its peak and the end of training (a four-point decline, against one point for FjORD and half a point for Uniform). On the one benchmark whose heterogeneity is not synthetic, allocating capacity by estimated divergence is therefore not merely uninformative—as the matched-budget control of Section 4.4 established—but actively harmful. The behaviour of the Uniform baseline is equally instructive: it does not collapse here, even though it caps every client at half width exactly as in the image experiments. The explanation lies in which parameters the cap leaves untrained. In the convolutional models, capping every client freezes the outer channels of every layer including the classifier, so the frozen coordinates sit directly on the output path and corrupt predictions (Proposition 3). In the recurrent model, sub-model extraction slices hidden units within each gate block, but the character embedding and all rows of the output projection are shared in full by every client; the untrained coordinates are interior hidden units whose effect is to reduce effective capacity rather than to distort the output mapping. Parameter coverage thus matters most where it intersects the output layer, which sharpens rather than weakens the conclusion of Section 4.3: what must be preserved is coverage of the parameters that determine predictions, not merely of the parameter count. We note two limitations of this benchmark. Absolute accuracies (25–35%) are well below published LEAF results, reflecting our deliberately small two-layer LSTM, the 200-round budget, and a single local epoch; the comparison across allocation policies is nevertheless internally consistent, since every policy trains the identical architecture under the identical protocol. Second, with twenty role-clients the federation is small, so the natural partition is heterogeneous but not large-scale. 24

Capacity Confounds in Adaptive Sub-model Federated Learning

4.7 Who Inherits the Confound? Implications for Update-Based Estimation The capacity confound of Section 4.4 partitions the literature into two populations. Methods that infer data properties from client updates while holding model size fixed are unaffected: clustered federated learning groups clients by the cosine similarity of equal-size gradients (Sattler et al., 2021), IFCA assigns cluster identities by lowest local loss under candidate models (Ghosh et al., 2020), FedGroup clusters by decomposed optimization-direction similarity (Duan et al., 2021), and loss- or gradient-norm-based client selection (Cho et al., 2022; Marnissi et al., 2021) likewise compares like with like. In the equal-model regime these signals are legitimate data proxies. They lose that meaning the moment sub-model width varies with device capacity: a quarter-width client mechanically produces smaller, differently oriented updates regardless of its data—precisely the coupling we measure (r between −0.72 and −0.84 across all runs). This places methods that size or shape sub-models from training-derived signals—capability-driven pruning ratios (Jiang et al., 2023), datadriven channel importance (Li et al., 2021), magnitude-based sub-model composition (Wu et al., 2024), and learned sparse ratios that explicitly superimpose capability and data effects (Xue et al., 2025)—in the inherited-risk population: any data-property inference drawn from their updates mixes data with capacity. Our results suggest a concrete reporting practice for this setting: whenever client statistics are estimated from heterogeneous sub-model updates, validate them against a capacity-stratified analysis (as in Table 5) before attaching a data-heterogeneity interpretation.

5 Conclusion This paper used Heterogeneity-Aware Adaptive Sub-model Federated Learning (HAS-FL) as a test case for studying a question that comes before any such design. Can client data heterogeneity be estimated from the sub-model updates a federated server observes, and does allocating capacity by that estimate help? The hypothesis we started from was that clients with divergent data need more capacity, while clients whose data matches the population can contribute through smaller sub-models. Our evidence does not support that hypothesis, and it shows why. We report three findings. First, when validated against ground-truth label divergence on reproducible partitions, update-divergence estimates of heterogeneity are dominated by capacity rather than by data. The correlation with capacity falls between −0.72 and −0.84 on all six runs, and no data signal remains once capacity is controlled for. The cause is structural. Sub-models of different widths differ by an order of magnitude in the size and norm of their updates, which is far larger than any variation caused by data, and the effect persists across two corrected estimators. Second, capped allocation has a failure mode that is easy to miss and can be proven rather than merely observed. If every client is capped below full width, the uncovered coordinates never move from their initialization while still taking part in every prediction (Proposition 3). The damage ranges from gradual decay on CIFAR-10 to a fall to nearchance accuracy on the 62-class EMNIST task, and a coverage guarantee removes it. The Shakespeare results make the claim more precise. What must stay covered is the parameters on the output path, because a recurrent model whose embedding and output projection are shared in full can tolerate uncovered hidden units without collapsing. 25

Moayedikia and Troncoso Lora

Third, a matched-budget control settles the practical question. Random allocation to the same average budget performs no differently from the heterogeneity-aware policy on both image benchmarks. On the naturally partitioned Shakespeare benchmark, the adaptive policy is the weakest of the three strategies tested while using the most capacity. Sub-model training itself remains useful, since half-width clients train and communicate at roughly a quarter of the cost, and this admits devices that could not otherwise take part. Even so, it gives up roughly eight to thirteen accuracy points against full-model training, and what governs that trade-off is parameter coverage and total budget rather than allocation intelligence. Several limitations apply. Our federations use twenty clients in simulation rather than physical devices, which is a scale typical of the sub-model literature we build on but small compared with production deployments. Absolute accuracies on Shakespeare are well below published LEAF results, because we use a deliberately small recurrent model and a short training budget. The comparisons remain valid within the study, since every policy trains an identical architecture under an identical protocol. The heterogeneity measure we study is one natural choice, and a negative result about it does not rule out estimators built from other signals. Any such estimator, however, faces the same difficulty of separating data effects from capacity effects while sub-model width varies with the device. Working out which data statistics can still be recovered in that setting, whether by estimating within capacity groups, by using probe batches of a fixed width, or by having clients report statistics directly, is the natural next step. The wider implication is a warning about a design pattern rather than about one algorithm. Inferring client properties from training signals is standard practice in federated learning, for client selection, weighting, and clustering, and it is sound when every client trains a model of the same size. Once sub-model training makes model size depend on device capacity, that inference changes meaning without any warning, and methods that set sub-model size from training signals inherit the same confound. We therefore recommend two practices. Any such method should check its estimates against an analysis that groups clients by capacity before treating them as a measure of data heterogeneity. Adaptive allocation schemes should be reported against a matched-budget random control, which in our experiments was enough to explain the benefits they report.

References B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communicationefficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics, 2017. P. Kairouz, H. B. McMahan, B. Avent, et al., “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021. I. Dayan, H. R. Roth, A. Zhong, et al., “Federated learning for predicting clinical outcomes in patients with COVID-19,” Nature Medicine, vol. 27, no. 10, pp. 1735–1743, 2021. 26

Capacity Confounds in Adaptive Sub-model Federated Learning

N. Rieke, J. Hancox, W. Li, et al., “The future of digital health with federated learning,” npj Digital Medicine, vol. 3, no. 1, pp. 1–7, 2020. A. Hard, K. Rao, R. Mathews, et al., “Federated learning for mobile keyboard prediction,” arXiv preprint arXiv:1811.03604, 2019. D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. V. Poor, “Federated learning for smart healthcare: A survey,” ACM Computing Surveys, vol. 55, no. 3, pp. 1–37, 2021. L. U. Khan, W. Saad, Z. Han, E. Hossain, and C. S. Hong, “Federated learning for Internet of Things: Recent advances, taxonomy, and open challenges,” IEEE Communications Surveys & Tutorials, vol. 23, no. 3, pp. 1759–1799, 2021. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020. X. Wang, C. Li, F. Zhang, and W. Liu, “A comprehensive survey on federated learning: Architecture, applications, and emerging trends,” Computers, Materials & Continua, vol. 73, no. 1, pp. 519–565, 2024. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning, 2020. C. Wang, X. Li, W. Zhang, and Y. Liu, “Federated learning under non-stationary data: A comprehensive survey,” Neurocomputing, vol. 618, p. 129115, 2026. Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-IID data silos: An experimental study,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 12, pp. 4341–4354, 2022. P. Wang, Z. Mi, Z. Shen, Z. Yin, B. Guo, and J. Dong, “FedRDA: Representation decoupling and divergence-aware aggregation for personalized federated learning,” Knowledge-Based Systems, vol. 333, p. 115084, 2026. E. Diao, J. Ding, and V. Tarokh, “HeteroFL: Computation and communication efficient federated learning for heterogeneous clients,” in International Conference on Learning Representations, 2021. S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečný, H. B. McMahan, V. Smith, and A. Talwalkar, “LEAF: A benchmark for federated settings,” arXiv preprint arXiv:1812.01097, 2018. J. Xu, Y. Chen, L. Wang, and X. Zhang, “FLSC: A semantic-guided federated learning framework for heterogeneous clients with cascading heterogeneity compensation,” Computer Networks, vol. 262, p. 111095, 2026. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Conference on Machine Learning and Systems, 2020. 27

Moayedikia and Troncoso Lora

L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” in International Conference on Machine Learning, 2021, pp. 2089–2099. S. Horvath, S. Laskaridis, M. Alber, and N. D. Lane, “FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout,” in Advances in Neural Information Processing Systems, 2021. W. Li, C. Wang, Y. Zhang, and F. Liu, “FLex&Chill: Improving local training in heterogeneous federated learning via temperature scaling,” Advanced Engineering Informatics, vol. 65, p. 103219, 2026. D. Chen, L. Yao, D. Gao, B. Ding, and Y. Li, “Efficient personalized federated learning via sparse model-adaptation,” in International Conference on Machine Learning, 2023, pp. 5234–5256. A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,” in Advances in Neural Information Processing Systems, 2020. T. Li, S. Hu, A. Beirami, and V. Smith, “Ditto: Fair and robust federated learning through personalization,” in International Conference on Machine Learning, 2021. J. Jin, J. Ren, Y. Zhou, L. Lyu, J. Liu, and D. Dou, “Personalized federated learning with feature alignment and classifier collaboration,” in International Conference on Learning Representations, 2022. L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y. Duan, “FedKD: Communication efficient federated learning via knowledge distillation,” in Advances in Neural Information Processing Systems, 2023. J. Zhang, Y. Hua, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan, “FedCP: Separating feature information for personalized federated learning via conditional policy,” in ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 3249–3261. Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “FedProto: Federated prototype learning across heterogeneous clients,” in AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8432–8440. Y. Sun, W. Chen, F. Li, and Y. Zhang, “ProtoFedGAN: Prototype-based federated generative adversarial networks for non-IID data,” Future Generation Computer Systems, vol. 165, p. 107644, 2026. D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communicationefficient SGD via gradient quantization and encoding,” in Advances in Neural Information Processing Systems, vol. 30, 2017. A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” in Conference on Empirical Methods in Natural Language Processing, 2017, pp. 440–445. 28

Capacity Confounds in Adaptive Sub-model Federated Learning

T. Vogels, S. P. Karimireddy, and M. Jaggi, “PowerSGD: Practical low-rank gradient compression for distributed optimization,” in Advances in Neural Information Processing Systems, vol. 32, 2019. Y. Zhang, L. Wang, H. Chen, and M. Li, “A comprehensive survey on one-shot federated learning: Advances and challenges,” Neurocomputing, vol. 617, p. 129014, 2026. Y. Ma, C. Zhang, X. Wang, and J. Liu, “FedDistr: One-shot federated learning via data distribution disentanglement,” Artificial Intelligence, vol. 342, p. 104283, 2026. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of FedAvg on non-IID data,” in International Conference on Learning Representations, 2020. S. Alam, L. Liu, M. Yan, and M. Zhang, “FedRolex: Model-heterogeneous federated learning with rolling sub-model extraction,” in Advances in Neural Information Processing Systems, 2022. A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., University of Toronto, 2009. Y. Deng, M. M. Kamani, and M. Mahdavi, “Adaptive personalized federated learning,” arXiv preprint arXiv:2003.13461, 2020. Y. Deng, F. Qiao, and M. Mahdavi, “On the convergence and stability of distributed submodel training,” arXiv preprint arXiv:2511.06132, 2025. C. T. Dinh, N. H. Tran, and T. D. Nguyen, “Personalized federated learning with Moreau envelopes,” in Advances in Neural Information Processing Systems, 2020. A. Ghosh, J. Chung, D. Yin, and K. Ramchandran, “An efficient framework for clustered federated learning,” in Advances in Neural Information Processing Systems, 2020. F. Haddadpour, M. M. Kamani, A. Mokhtari, and M. Mahdavi, “Federated learning with compression: Unified analysis and sharp guarantees,” in International Conference on Artificial Intelligence and Statistics, 2021. N. Hyeon-Woo, M. Ye-Bin, and O. Tae-Hyun, “FedPara: Low-rank Hadamard product for communication-efficient federated learning,” in International Conference on Learning Representations, 2021. Y. Jiang, S. Wang, V. Valls, B. J. Ko, W.-H. Lee, K. K. Leung, and L. Tassiulas, “Model pruning enables efficient federated learning on edge devices,” IEEE Transactions on Neural Networks and Learning Systems, 2022. T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” in Advances in Neural Information Processing Systems, 2020. F. Qiao, X. Xu, and M. Mahdavi, “Communication-efficient distributed learning via lowrank aggregation,” arXiv preprint, 2021. 29

Moayedikia and Troncoso Lora

S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” in International Conference on Learning Representations, 2021. A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization,” in International Conference on Artificial Intelligence and Statistics, 2020. D. Rothchild, A. Panda, E. Ullah, N. Ivkin, I. Stoica, V. Braverman, J. Gonzalez, and R. Arber, “FetchSGD: Communication-efficient federated learning with sketching,” in International Conference on Machine Learning, 2020. J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” in Advances in Neural Information Processing Systems, 2020. C. Wu, F. Wu, L. Lyu, Y. Huang, and X. Xie, “Communication-efficient federated learning via knowledge distillation,” Nature Communications, vol. 13, no. 1, p. 2032, 2022. B. Yuan, C. R. Wolfe, C. Dun, Y. Tang, A. Kyrillidis, and C. M. Jermaine, “Distributed learning of deep neural networks using independent subnet training,” arXiv preprint arXiv:1910.02120, 2019. A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, 2017. M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are GANs created equal? A large-scale study. In Advances in Neural Information Processing Systems, 2018. C. Yang, Q. Wang, M. Xu, Z. Chen, K. Bian, Y. Liu, and X. Liu. Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data. In Proceedings of the Web Conference (WWW), 2021. A. M. Abdelmoniem, C.-Y. Ho, P. Papageorgiou, and M. Canini. A comprehensive empirical study of heterogeneity in federated learning. IEEE Internet of Things Journal, 10(16):14071–14083, 2023. F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury. Oort: Efficient federated learning via guided participant selection. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2021. C. Li, X. Zeng, M. Zhang, and Z. Cao. PyramidFL: A fine-grained client selection framework for efficient federated learning. In Proceedings of the 28th Annual International Conference on Mobile Computing and Networking (MobiCom), 2022. Y. G. Kim and C.-J. Wu. AutoFL: Enabling heterogeneity-aware energy efficient federated learning. In 54th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2021. 30

Capacity Confounds in Adaptive Sub-model Federated Learning

J. Shin, Y. Li, Y. Liu, and S.-J. Lee. FedBalancer: Data and pace control for efficient federated learning on heterogeneous clients. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services (MobiSys), 2022. Z. Jiang, Y. Xu, H. Xu, Z. Wang, C. Qiao, and Y. Zhao. Computation and communication efficient federated learning with adaptive model pruning. IEEE Transactions on Mobile Computing, 22, 2023. A. Li, J. Sun, P. Li, Y. Pu, H. Li, and Y. Chen. Hermes: An efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (MobiCom), 2021. F. Wu, X. Wang, Y. Wang, T. Liu, L. Su, and J. Gao. FIARSE: Model-heterogeneous federated learning via importance-aware submodel extraction. In Advances in Neural Information Processing Systems, 2024. J. Xue, S. Sun, M. Liu, Y. Wang, Z. Liu, and J. Wang. Learnable sparse customization in heterogeneous edge computing. arXiv preprint arXiv:2412.07216, 2025. F. Sattler, K.-R. Müller, and W. Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 32(8):3710–3722, 2021. M. Duan, D. Liu, X. Ji, R. Liu, L. Liang, X. Chen, and Y. Tan. FedGroup: Efficient federated learning via decomposed similarity-based clustering. In IEEE Intl. Conf. on Parallel & Distributed Processing with Applications (ISPA), 2021. Y. J. Cho, J. Wang, and G. Joshi. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022. O. Marnissi, H. El Hammouti, and E. H. Bergou. Client selection in federated learning based on gradients importance. arXiv preprint arXiv:2111.11204, 2021.

31

Related documents

• When Is Shallow Enough? Adaptive Split Federated Learning with Client-Specific Sufficiency Estimation#459621
• Federated Learning on Non-IID Data: A Survey#263752
• Surviving the Edge: Federated Learning under Networking and Resource Constraints#155176
• Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning#433626
• Joint Optimization for Federated Learning and Transmission over Unreliable Wireless Networks with Heterogeneous Data#1083703