1
AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments
arXiv:2606.08197v1 [cs.CL] 6 Jun 2026
Yan Wang, Ziyi Gao, Rui Wang*
Abstract—Large Language Models (LLMs) have significantly propelled the advancement of edge intelligence and have been widely deployed across various scenarios, including autonomous driving, industrial inspection, and personalized IoT services. However, the collaborative adaptation of LLMs on edge devices continues to face formidable challenges due to strict data privacy constraints, highly heterogeneous computing and communication resources, and the non-independent and identically distributed (non-IID) nature of local data. Federated Fine-Tuning (FFT) enables the collaborative optimization of distributed models without exposing raw data. Yet, traditional synchronous aggregation suffers from a severe ”straggler effect,” resulting in high system latency and low resource utilization. Existing asynchronous federated learning methods are predominantly designed for smallto-medium-scale models and struggle to address the specific challenges inherent in LLM fine-tuning—namely, model drift caused by stale updates, aggravated client drift stemming from data heterogeneity, and aggregation fairness imbalance resulting from the dominance of fast clients. To address these issues, this paper proposes AlignFed, an asynchronous federated finetuning framework for LLMs tailored to heterogeneous edge environments. AlignFed employs a lightweight multi-stage semantic alignment mechanism comprising three core modules: versionaware update grouping, cross-version semantic alignment based on a mini-batch calibration set, and fairness-aware aggregation that integrates both update freshness and client participation frequency. This framework effectively mitigates cross-version model drift and client drift while enhancing aggregation fairness, thereby achieving stable and efficient asynchronous federated optimization in scenarios characterized by high heterogeneity and significant update staleness. Experimental results across multiple benchmark datasets and two mainstream LLM backbones demonstrate that, compared to existing synchronous and asynchronous baseline methods, AlignFed achieves faster and more stable convergence, superior robustness against stale updates, and enhanced generalization performance. Index Terms—Federated fine-tuning, Asynchronous FL, Drift, Large language models, Non-IID data.
I. I NTRODUCTION
T
HE rapid development of edge intelligence is transforming numerous mobile and IoT applications, including autonomous driving, industrial inspection, and personalized IoT assistants [1]–[3]. Recent advances in large language models (LLMs) further enhance the capabilities of these applications by enabling complex reasoning and multimodal interactions. However, deploying and adapting LLMs on edge devices Yan Wang, Ziyi Gao and Rui Wang are with the Department of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China(e-mail: [email protected]; [email protected]; [email protected]).
remains extremely challenging due to three key limitations: strict data privacy (prohibiting the upload of raw data), resource heterogeneity (clients have varying computational and communication capabilities), and skewed data distributions (non-independent and identically distributed local datasets) [4], [5]. Federated learning (FL) offers a promising paradigm by enabling collaborative model training without sharing raw data [6], [7]. Building upon FL, federated fine-tuning (FFT) enables large foundation models to be adapted across distributed edge clients. However, most existing FFT methods rely on synchronous aggregation, where the server must wait for all selected clients to finish local updates before proceeding. In heterogeneous edge environments, this leads to severe straggler effects [8], resulting in high latency and poor resource utilization [9], [10]. In real-world edge environments, the computational and communication capabilities of devices vary significantly. For example, lightweight IoT boards (such as Raspberry Pi and Jetson Nano) operate several orders of magnitude slower than industrial-grade edge servers or GPU-equipped devices. This imbalance leads to the so-called “straggler effect,” [9] where slower devices delay the entire aggregation process. As a result, synchronous training suffers from increased system latency, low resource utilization, and delayed global model updates [11]. This motivates the adoption of asynchronous aggregation to improve timeliness and system efficiency [12]– [14]. However, existing AsyncFL methods [15], [16] are primarily developed for traditional CNN/MLP models with dense full-parameter updates, and are not well-suited for LLM finetuning under low-rank adaptation. This limitation stems from two fundamental structural mismatches: (1) Traditional DNNs rely on full-parameter gradient updates, whereas LLM finetuning typically adopts low-rank adaptations (e.g., LoRA [17]), where updates lie in a constrained low-dimensional subspace. Direct asynchronous aggregation may disrupt this low-rank structure and lead to degraded adaptation quality. (2) The representation space of LLMs is highly structured and semantically sensitive [18], making it vulnerable to version inconsistency. In asynchronous settings, stale updates computed from different model versions can introduce misaligned optimization directions, resulting in unstable convergence and performance degradation. These structural mismatches manifest in several fundamental challenges when extending AsyncFL to LLM fine-tuning
2
in heterogeneous edge environments:
II. R ELATED W ORK AND P RELIMINARIES
First, asynchronous training inevitably generates stale updates from clients trained on different model versions (i.e., staleness accumulation). This issue is further amplified in LLM scenarios: due to the high dimensionality and complex parameter space of LLMs, direct aggregation of these stale updates is more likely to trigger training instability and significant model drift [19].
This section reviews the background and related research relevant to asynchronous federated fine-tuning of LLMs in heterogeneous edge environments. We first discuss asynchronous federated learning, followed by parameter-efficient federated fine-tuning approaches, and finally review studies on semantic consistency and fairness in federated aggregation.
Second, asynchronous training exacerbates heterogeneity amplification. In LLM fine-tuning, edge devices typically hold highly non-independent and identically distributed [5] (nonIID) data; coupled with the more pronounced differences in the magnitude of LLM parameter updates, local updates tend to converge toward different local optima. This not only induces severe client drift but also leads to inconsistent update directions (directional bias and drift) across clients, ultimately degrading model performance [20], [21].
A. Asynchronous Federated Learning
Third, imbalanced client participation can bias the global model. Faster clients contribute updates far more frequently than slower ones, resulting in the global model being dominated by this small subset of fast clients. To address this, the issue of unfair participation under heterogeneous devices needs to be explicitly resolved. To systematically mitigate the above challenges—model drift induced by staleness, client drift caused by heterogeneity amplification, and unfair aggregation due to imbalanced participation—this paper proposes AlignFed, a novel asynchronous federated fine-tuning framework for large language models. AlignFed introduces a multi-stage alignment mechanism consisting of three key components: (i) version-aware update grouping, which explicitly organizes updates according to model versions to manage stale updates; (ii) cross-version semantic alignment, which maps stale updates into a consistent representation space before aggregation using a lightweight calibration set; and (iii) fairness-aware aggregation, which adaptively weights updates based on update freshness, update strength, and participation frequency to balance client contributions. By jointly addressing model drift, client drift, and unfair aggregation, AlignFed enables stable and fair convergence for asynchronous federated LLM fine-tuning in heterogeneous edge environments. Our main contributions are summarized as follows: We propose AlignFed, the first asynchronous federated fine-tuning framework specifically tailored for LLMs on heterogeneous edge devices. • We design a multi-stage alignment mechanism that integrates version-aware update grouping, cross-version semantic alignment, and fairness-aware aggregation, effectively mitigating issues such as model drift, client drift, and unfair aggregation in asynchronous training. • We provide theoretical guarantees showing that AlignFed achieves bounded-drift convergence under asynchronous aggregation.
•
Existing AsyncFL methods aim to mitigate stragglers by relaxing synchronization constraints. Representative work such as FedAsync [13] allows the server to update the global model immediately upon receiving a client update, thereby reducing waiting time caused by slower devices. FedBuff [14] further improves aggregation efficiency by buffering a sufficient number of updates before triggering a global update. They improve training efficiency via immediate or buffered aggregation. Subsequent studies have focused on improving the robustness and convergence behavior of AsyncFL from multiple perspectives. One line of work introduces staleness-aware aggregation strategies, where client updates are reweighted based on their delay to mitigate the negative impact of outdated information [15]. Another line incorporates delay compensation mechanisms [22] or bounded-staleness constraints [23] to stabilize asynchronous optimization dynamics. Beyond aggregation design, recent efforts further investigate the theoretical properties of AsyncFL. For example, convergence guarantees have been established under convex and non-convex settings [16], [24], while other studies explore discrepancy-aware aggregation to address non-IID data distributions [25]. These advances collectively improve the stability and applicability of AsyncFL in heterogeneous environments. These methods fundamentally assume that client updates are compatible within a shared parameter space, and that aggregation operations are based on the full model update. However, in LLM fine-tuning, updates are parametrically efficient (e.g., LoRA [17] for Low-Rank Adaptation), not full model gradients, and operate in a high-dimensional, tightly coupled parameter space with non-IID data. Therefore, outdated updates are not merely outdated—they are semantically inconsistent. This renders traditional weighted methods that account for outdatedness inadequate. Specifically, two fundamental mismatches between existing AsyncFL methods and LLM fine-tuning further highlight their incompatibility: First, traditional AsyncFL relies on dense full-parameter updates for CNN/MLP models, whereas LLM fine-tuning adopts low-rank LoRA updates confined to a low-dimensional subspace. Direct application of asynchronous aggregation (e.g., weighted averaging in FedAsync, buffered aggregation in FedBuff) disrupts the low-rank structure of LoRA updates, leading to semantic degradation and poor adaptation quality. Second, existing AsyncFL methods adopt a simple Euclidean space assumption for aggregation, ignoring the highly structured and semantically sensitive nature of LLM representation spaces [18]. Stale updates from different model versions in asynchronous settings cause misaligned optimization directions, which not only amplifies model drift
3
but also exacerbates client divergence under non-IID data distributions. These observations indicate that directly applying existing AsyncFL mechanisms to LLM fine-tuning may lead to inconsistencies in representation alignment, amplified client drift, and biased aggregation behavior. B. Federated Fine-Tuning with Parameter-Efficient Adaptation FFT reduces communication and computation overhead by transmitting lightweight parameter updates instead of fullmodel gradients [26]. Parameter-efficient fine-tuning (PEFT) methods such as FedPETuning [27] and FedLoRA [28] introduce parameter decomposition techniques to enable personalized federated learning. By fine-tuning only low-rank components and transmitting these lightweight matrices, such approaches significantly reduce communication costs compared to full-parameter federated training. Recent studies further improve communication efficiency. FFA-LoRA [6] freezes a subset of adapter parameters to minimize update size. FLASC [29] introduces sparse adapter communication, while LoRI [30] employs task-specific sparse masking. FedSA-LoRA [31] separates shared and personalized update components. Despite these advances, most existing federated PEFT approaches rely on synchronous aggregation. In heterogeneous edge environments, this synchronization requirement forces faster clients to wait for slower ones, reducing system efficiency and increasing end-to-end training latency. C. Heterogeneity and Fairness in Federated Aggregation Client heterogeneity is a fundamental challenge in federated learning, especially when local data distributions are highly non-IID [20]. Under such conditions, local updates may drift toward different optimization directions, leading to client drift and degraded global model performance [21]. This issue becomes even more pronounced in asynchronous training settings, where stale updates and non-IID data jointly amplify divergence among client updates. Several studies have explored techniques to mitigate representation inconsistency and model divergence. For example, knowledge distillation [32], representation matching [33], and feature alignment [34] have been proposed to maintain consistency among distributed models. However, these methods are primarily designed for synchronous training scenarios and assume homogeneous update timing across clients, making them difficult to directly apply to asynchronous federated finetuning of LLMs. Fairness is another critical concern in federated aggregation, as the global model may become biased toward clients with larger datasets or more frequent participation. Existing studies [35], [36] address this issue by reweighting training objectives or optimizing worst-case client performance. More recent approaches introduce fairness-aware optimization algorithms [37], [38] to mitigate domain skew and participation imbalance.
Nevertheless, most fairness-aware federated learning methods assume synchronous participation and full-parameter training. In asynchronous environments, clients with stronger hardware or more stable network conditions naturally contribute updates more frequently, which further amplifies participation imbalance. III. A LIGN F ED F RAMEWORK AND P ROBLEM F ORMULATION In this section, we introduce the proposed AlignFed framework for asynchronous federated LLM fine-tuning with LoRA. We first formalize the global and local optimization objectives and define core notations, then present the overall pipeline of AlignFed, and finally detail its three key components and theoretical analysis. A. Problem Formulation and Notation We consider an asynchronous federated learning system for LLM fine-tuning (with LoRA as the lightweight finetuning paradigm for implementation), consisting of a central server and a set of participating clients denoted by C = {1, 2, . . . , N }. Each client i ∈ C owns a private dataset |Di | Di = {(xi,j , yi,j )}j=1 for the same downstream task (e.g., code generation, natural language reasoning), where the data in Di is drawn from a non-IID and heterogeneous distribution. All clients perform lightweight fine-tuning via LoRA, which freezes the pre-trained LLM backbone Wbackbone and only optimizes low-rank adapter parameters Ai , Bi (rank r ≪ d, d is the LLM hidden dimension), making the local model Wi = Wbackbone + Ai BiT . 1) Optimization Objectives: The core goal of AlignFed is to learn a global LoRA adapter such that the global model minimizes the aggregated empirical risk over all clients’ private datasets, with the constraint of asynchronous lightweight fine-tuning. a) Global Objective Function: The global optimization objective is defined as: min
A ,W B Wglobal global
A B F(Wglobal , Wglobal )=
N
1 X A B E(x,y)∼Di ℓ LLM(Wbackbone + Wglobal (Wglobal )T ; x), y N i=1 (1) where ℓ(·, ·) is the task-specific loss function (cross-entropy, negative log-likelihood, etc.), and LLM(·; x) denotes the forward pass of the pre-trained LLM with input x. b) Local Objective Function: For each client i, we design a local objective with representation consistency regularization to mitigate client drift from the global model, which is critical for asynchronous federated training [16]: min Fi (Ai , Bi ) = i) 2 E(x,y)∼Di ℓ LLM(Wbackbone + Ai BiT ; x), y + λ∥ϕi − ϕ(v g ∥2 (2) (v ) where ϕi /ϕg i are the local/global feature representations extracted from the LLM’s penultimate layer, and λ > 0 is the regularization coefficient balancing task performance and representation consistency. Ai ,Bi
4
2) Key Notations: The server maintains a versioned global (t) model to track asynchronous updates. Let Wg = Wbackbone + A B (Wglobal )(t) (Wglobal )(t)T denote the global model at global version t (incremented only when aggregation is performed). Client i starts local fine-tuning from a stale global model (v ) snapshot Wg i (vi ≤ t) and uploads the LoRA parameter (v ) (v ) increment ∆i = (Ai − Ag i , Bi − Bg i ) with the version tag vi . Table I summarizes the key notations. TABLE I K EY N OTATIONS U SED IN THE A LIGN F ED F RAMEWORK Symbol
Description
C Di (t) Wg vi ∆i Gk (v ) ϕi , ϕ g i Dc Tk→t αi ω(vi ) ξi τi = t − v i
Set of participating clients {1, 2, . . . , N } Fig. 1. Overall architecture of the proposed AlignFed framework. Private local dataset of client i Global LLM model (backbone + LoRA) at version t Global version index for client i’s local fine-tuning Client i’s lightweight LoRA parameter increment The training pipeline of AlignFed consists of four key Group of updates originating from global version k stages, which are detailed as follows: Local/global LLM feature representations (penultimate layer) 1) Client-side Fine-tuning (① in Fig. 1, Algorithm 1, Small server-side calibration set for semantic alignment Linear transformation for cross-version semantic alignment Lines 3–7) : Normalized aggregation weight for client i’s update Client i downloads the global model with version vi Version-based freshness weight for stale updates and performs lightweight local fine-tuning via LoRA Fairness weight based on client i’s participation frequency Staleness of client i’s update (time lag from global version t) — a classic method for LLM federated fine-tuning due
B. Framework Overview We propose the AlignFed framework, an asynchronous federated LLM fine-tuning method designed to address three core challenges amplified in asynchronous LLM scenarios: accumulation of stale updates, heterogeneity amplification, and unfair participation caused by fast clients. Unlike conventional asynchronous federated learning that directly aggregates unaligned local updates, AlignFed introduces a dedicated multi-stage alignment mechanism to sequentially resolve version inconsistency, directional bias, and contribution imbalance. The overall workflow of AlignFed is illustrated in Fig. 1 and summarized in Algorithm 1. The framework contains three core components that correspond one-to-one to the three key challenges: • Version-aware Grouping & Cross-version Semantic Alignment: Explicitly handles staleness accumulation (which induces model drift) by grouping updates from different model versions and aligning stale updates to the current semantic space, thus suppressing global model deviation caused by outdated local updates. • Intra-group Centering: Mitigates heterogeneity amplification (driven by non-IID data, leading to client drift) by eliminating directional bias within the same version group, ensuring local updates across clients remain consistent in optimization direction. • Fairness-aware Aggregation: Solves unfair participation (resulting from fast clients dominating aggregation, causing unfair aggregation) by adaptively balancing contributions from fast and slow clients, avoiding global model bias toward high-frequency participants.
to its efficiency. Specifically, we freeze the pre-trained LLM backbone Wbackbone and only optimize two lowrank adapter matrices Ai ∈ Rd×r and Bi ∈ Rr×d (where d is the LLM’s hidden dimension, r ≪ d is the rank of LoRA). This design reduces the client’s local parameter volume by over 95% compared to full-model fine-tuning, making it feasible for resource-constrained edge devices [17]. To avoid local feature drift in LLM’s high-dimensional semantic space, we introduce a representation consistency regularization term: i) 2 Llocal = Ltask + λ∥ϕi − ϕ(v g ∥2 ,
(3)
where Ltask is the downstream task loss (e.g., crossentropy for classification, negative log-likelihood for text (v ) generation), ϕi and ϕg i are the feature representations extracted from the penultimate layer of the local/global LLM, and λ ∈ [0.1, 1.0] balances task performance and global alignment. Unlike conventional regularization that constrains parameters directly, this term targets LLM’s semantic representation — a more effective way to mitigate drift in large-scale models. As parameterlevel constraints often fail to capture high-dimensional semantic consistency [39], while semantic-level alignment can better preserve the global consistency of model cognition, especially in federated scenarios with non-IID data [40]. After completing local epochs (adapted to dataset size), the client computes the LoRA parameter increment (v ) (v ) (v ) (v ) ∆i = (Ai − Ag i , Bi − Bg i ) (where Ag i , Bg i are the global LoRA parameters at version vi ) and uploads only ∆i and the version tag vi to the server. This lightweight upload minimizes communication overhead, a key requirement for edge-centric federated systems.
5
2) Version-aware Grouping (② in Fig. 1, Algorithm 1, Lines 8–10) : Upon receiving asynchronous updates {(∆i , vi )}, the server first groups them by their originating version to address staleness accumulation: Gk = {∆i | vi = k}.
(4)
This isolates updates trained on the same global version, avoiding mixed semantic spaces caused by direct aggregation of stale and fresh updates (e.g., updates from version k = t − 3 mixed with version k = t). The server maintains a buffer for pending updates, with a heterogeneity-aware triggering rule for aggregation: trigger aggregation if either (1) the number of pending updates exceeds M (balancing update diversity and timeliness), or (2) the elapsed time since last aggregation exceeds ∆T (preventing fast clients from waiting indefinitely). This rule adapts to variable upload speeds—e.g., if most clients are fast, aggregation triggers via condition (1); if many are slow, condition (2) ensures timely global model updates. 3) Intra-group Centering (③ in Fig. 1, Algorithm 1, Lines 11–12) : This stage sequentially addresses heterogeneity amplification: Intra-group Centering eliminates directional bias caused by non-IID data within each version group (e.g., a group of industrial clients all pushing updates toward equipment-maintenance semantics), which mitigates client drift from heterogeneous data: X ˜ i = ∆i − 1 ∆j . (5) ∆ |Gk | ∆j ∈Gk
˜ i retains client-specific task The centered increment ∆ knowledge while removing group-level shared bias, effectively suppressing heterogeneity amplification in asynchronous environments. 4) Cross-version Semantic Alignment (④ in Fig. 1, Algorithm 1, Lines 13–14) : Building on the versionaware grouping (Stage 2), this step projects stale updates (t) (from Gk , k < t) into the current semantic space of Wg to solve the ”version mismatch” problem induced by staleness accumulation through Cross-version Semantic Alignment. Specifically, this step projects stale updates (from Gk , k < t) into the current semantic space of (t) Wg to solve the ”version mismatch” problem. We use a small server-side calibration set Dc , which is constructed by randomly sampling 10% of the public training dataset. Critically, Dc is completely separated from the test dataset used for final model evaluation — the test dataset is reserved exclusively for performance assessment and does not participate in any alignment or training processes. We learn a lightweight linear transformation Tk→t : Rd×r × Rr×d → Rd×r × Rr×d (applied to LoRA matrices A and B separately) to minimize the semantic discrepancy between versions: ˜ i ), Dc )∥22 , (6) min ∥f (Wg(t) , Dc ) − f (Wg(k) + Tk→t (∆
Tk→t
where f (·, Dc ) extracts the mean feature representation of Dc from the LLM’s penultimate layer. The choice of linear transformation is deliberate for LLM scenarios: (1) Computational efficiency: linear transformations have O(d · r · |Dc |) complexity, which is negligible even for large-scale LLMs; (2) Semantic preservation: LLM’s feature space is approximately linear in the local update range, which is formally proved in Section IV (Theoretical Analysis); (3) No extra client burden: unlike non-linear alignment methods (e.g., neural networks), linear transformations require no additional client communication or computation. For the current version group Gt (k = t), Tt→t is the identity transformation (no alignment needed). 5) Adaptive Fairness-aware Aggregation (⑤ in Fig. 1, Algorithm 1, Lines 15–16): Computational heterogeneity leads to unfair participation: fast clients contribute significantly more updates than slow edge devices (e.g., IoT devices), dominating the global model. AlignFed addresses this via adaptive weights αi that jointly account for three factors (all normalized to avoid scale bias): • Freshness weight ω(vi ): A decaying exponential function of staleness τi = t − vi (time lag between client’s version and current global version), defined as ω(vi ) = e−γ·τi with γ ∈ [0.1, 0.3]. This downweights highly stale updates (e.g., τi > 5) that are semantically inconsistent with the current global model, while retaining valid updates from slightly delayed clients. ˜ i ∥2 + ϵ) • Update strength weight: Defined as 1/(∥∆ −8 (where ϵ = 10 to avoid division by zero), this term down-weights outlier updates with large deviation from the group mean (noisy or drifting clients)and emphasizes stable, consistent updates that align with the group pattern. • Fairness weight ξi : Inversely proportional to client i’s historical participation frequency to prevent overrepresentation. Specifically, num uploadsi is the number of updates that client i has uploaded to the server before the current aggregation round, p and ξi = 1/ num uploadsi + 1. This design ensures that fast clients are not excluded, yet their dominance is curtailed, thereby striking a balance between fairness and update quality. The normalized aggregation weight αi (ensuring sum of weights = 1) is: ω(vi ) · ξi 1 · , ˜ ˜ ω(v ) · ξ /(∥ ∆ ∥ + ϵ) ∥ ∆ ∥ i 2+ϵ j j j 2 j
αi = P
and the global model update rule is: X ˜ i ), Wg(t+1) = Wg(t) + αi Tvi →t (∆
(7)
(8)
i
This weighting ensures that: (1) fast clients contribute fresh, strong updates but are not overrepresented; (2) slow clients’ valid updates are still considered (if fresh enough); (3) small-dataset clients are not marginalized — effectively solving the unfair participation problem.
6
Algorithm 1 AlignFed Framework
B. Theoretical Analysis
(0)
1: Initialize global model Wg 2: for each round t = 1, 2, . . . do 3: for each client i in parallel do
We provide theoretical guarantees for AlignFed, focusing on how it mitigates the three core challenges (model drift, client drift, and unfair aggregation) in asynchronous federated (t) learning. Let Wg denote the global model at version t, client (v ) i fine-tunes from a stale version Wg i (vi ≤ t) with staleness τi = t − vi , and the server aggregates aligned updates via Eq. (8). Assumptions. We adopt standard assumptions for asynchronous FL [16] and introduce additional assumptions tailored to AlignFed:
(v )
Pull global model Wg i Perform local fine-tuning using Eq. (3) 6: Upload (∆i , vi ) 7: end for 8: if |Upending | ≥ M or ∆t > ∆T then 9: Group updates by version: Gk 10: end if 11: for each group Gk do 12: Center updates using Eq. (5) 13: Align to current version using Eq. (6) 14: end for 15: Aggregate aligned updates using Eq. (8) (t+1) 16: Broadcast new model Wg 17: end for 4: 5:
IV. T HEORETICAL A NALYSIS AND J USTIFICATION The following analysis theoretically justifies the three key components of AlignFed: version-aware grouping (staleness control), semantic alignment (drift reduction), and fairnessaware aggregation (bias mitigation). A. Complexity and Discussion Unlike conventional asynchronous federated learning methods (e.g., FedAsync, FedBuff) that transmit full model parameters in each round, AlignFed operates in a lightweight federated fine-tuning paradigm. Each client only uploads lowrank parameter increments, typically accounting for less than 1–5% of the full model size [30], which substantially reduces communication overhead. The additional computational cost mainly arises from the cross-version semantic alignment step on the server, where lightweight linear transformations are estimated using a small calibration set Dc . This process scales linearly with the number of model layers Dlayer and calibration samples |Dc |, i.e., O(Dlayer · |Dc |).
(A1) L-smoothness: For each client i, the local objective function Fi (W ) is L-smooth, i.e., ∥∇Fi (W1 ) − ∇Fi (W2 )∥ ≤ L∥W1 − W2 ∥ for all model parameters W1 , W2 . • (A2) Bounded gradient variance: The stochastic gradient of the local objective satisfies E∥gi (W ) − ∇Fi (W )∥2 ≤ σ 2 , where gi (W ) is the stochastic gradient computed from local data. • (A3) Bounded staleness: The staleness of all client updates is bounded by τmax , i.e., τi = t − vi ≤ τmax , ensured by the server’s heterogeneity-aware triggering rule. • (A4) Contractive alignment: The cross-version linear transformation Tvi →t is ρ-contractive (ρ < 1), i.e., ∥Tvi →t (x) − Tvi →t (y)∥ ≤ ρ∥x − y∥ for any two updates x, y. • (A5) Bounded update magnitude: The LoRA parameter increments of all clients are bounded, i.e., ∥∆i ∥ ≤ ∆max , which holds for fine-tuning with weight decay and gradient clipping. • (A6) Fairness weight boundedness: The fairness weight ξi is bounded in [0, 1], i.e., 0 < ξmin ≤ ξi ≤ 1 for all i, ensuring no client is completely excluded from aggregation. • (A7) Linear feature space: The model’s feature space (extracted from the penultimate layer, denoted f (·, Dc ) for calibration set Dc ) is approximately linear within the local update range—a property theoretically and empirically validated for LLMs under low-rank fine-tuning (e.g., LoRA) [18]. Formally, for any two model parameters (k) ˜ i and W2 = Wg(k) + ∆ ˜ j (within the W1 = W g + ∆ same version group Gk ), the feature mapping satisfies: •
(9)
Since |Dc | is typically small and L is independent of client scale, the overhead remains negligible even for large-scale edge environments. Compared to non-linear alignment methods (e.g., neural network-based mapping), AlignFed significantly reduces server-side computation while maintaining comparable alignment quality, making it more practical for resource-constrained distributed systems. Furthermore, the intra-group centering step adds minimal computational cost (O(|Gk |) per version group Gk ) and requires no additional communication, as it is performed locally on the server using existing client updates. The fairness-aware aggregation step also operates in O(M ) time (where M is the number of pending updates), ensuring the overall framework scales efficiently with the number of clients.
∥f (W1 , Dc ) − f (W2 , Dc ) − Tk→t (W1 − W2 )∥ ≤ ϵlin , (10) where ϵlin is a small constant (negligible for local LoRA updates), and Tk→t is the linear transformation used for cross-version alignment. Lemma IV.1 (Bounded Asynchronous Drift (Mitigates Staleness Accumulation)). Under Assumptions (A1)–(A3) and (A5), the model deviation induced by staleness is bounded by: ∥Wg(t) − Wg(vi ) ∥ ≤ ητmax ∆max .
(11)
where η is the global learning rate. After applying crossversion semantic alignment (Assumptions A4 and A7), the
7
aligned update error is further bounded by: ˜ i ) − ∆∗i ∥ ≤ ρηLτmax ∆max + ϵlin . ∥Tvi →t (∆
(12)
where ∆∗i is the ideal update aligned to the current version t, ˜ i is the centered increment from Eq. (5). and ∆ Proof Sketch: The first bound characterizes the global model deviation caused by asynchronous staleness, which is accumulated by at most τmax bounded update steps. The second bound further illustrates that the stale update can be corrected toward the ideal fresh update via the ρ-contractive semantic alignment, with a small constant error ϵlin introduced by linear feature approximation. This lemma theoretically guarantees that AlignFed effectively suppresses staleness-induced drift and maintains accurate semantic information during crossversion adaptation. Detailed proof is provided in Appendix A. Lemma IV.2 (Bounded Client Drift (Mitigates Heterogeneity Amplification)). Under Assumptions (A1) and (A5), the intragroup centering (Eq. (5)) and representation regularization (Eq. (3)) ensure: ) ( r L − L local task ˜ i ∥ ≤ min ∆max , , (13) ∥∆ λ ˜ i is the centered local update within version group Gk . where ∆ Furthermore, the gradient deviation induced by local update is bounded via L-smoothness as: ˜ i ) − ∇Fi (Wg(vi ) )∥ ≤ L∥∆ ˜ i ∥. ∥∇Fi (Wg(vi ) + ∆
(14)
Proof Sketch: Intra-group centering eliminates the group˜ i | Gk ] = 0. The represenlevel common bias, yielding E[∆ tation regularization term explicitly constrains the norm of ˜ i , which is further bounded by ∆max the centered update ∆ from Assumption (A5). By L-smoothness (A1), the gradient deviation caused by local fine-tuning is linearly bounded by the update norm. This lemma ensures that AlignFed effectively suppresses client drift and mitigates heterogeneity amplification across non-IID clients. Detailed proof is provided in Appendix B. Lemma IV.3 (Fair Contribution Bounding (Mitigates Unfair Participation)). Under Assumptions (A5) and (A6), the fairness-aware aggregation weight satisfies: ˜ i ∥2 + ϵ) ωi ξi /(∥∆ αi = P . ˜ j ωj ξj /(∥∆j ∥2 + ϵ)
(15)
The lower bound is given by ωmin ξmin 1 · , ˜ ∆ max + ϵ j ωj ξj /(∥∆j ∥2 + ϵ)
αi ≥ P
(16)
where ωmin = e−γτmax is the minimum freshness weight, ξmin is the lower bound of fairness weight, and ϵ > 0 is a small constant for numerical stability. Furthermore, the weight ratio between any two clients is bounded by: ˜ j ∥2 + ϵ αi ωmax ξmax ∥∆ ≤ · , ˜ i ∥2 + ϵ αj ωmin ξmin ∥∆ where ωmax = 1 denotes the weight for fresh updates.
(17)
Proof Sketch: The lower bound of αi ensures that slow or infrequent clients always maintain a non-negligible contribution in aggregation. The bounded weight ratio restricts the dominance of fast or frequent clients, preventing weight imbalance across clients. By jointly constraining freshness and fairness scores, AlignFed achieves stable and balanced aggregation under asynchronous settings. Detailed proof is provided in Appendix C. Theorem IV.1 (Ergodic Convergence (Overall Theoretical Guarantee)). Under Assumptions (A1)–(A7) and learning rate η ≤ 1/(2L), the ergodic gradient norm over T iterations satisfies:
T −1 1 1 X 2 E∥∇F (Wg(t) )∥2 ≤ O + O η 2 L2 τmax ρ2 ∆2max T t=0 T 2 σ +O + O(ϵ2lin ), N (18) where N is the total number of clients, σ 2 is the stochastic gradient variance, and ϵlin denotes the residual error of linear semantic alignment. The four terms respectively represent convergence optimality, asynchronous staleness error, client sampling variance, and cross-version alignment error. Proof Sketch: By applying the L-smoothness descent lemma and Young’s inequality, we establish the basic convergence recursion. The bounded staleness from Lemma IV.1, bounded client drift from Lemma IV.2, and fair aggregation from Lemma IV.3 together ensure that all error terms are well constrained. The final bound achieves standard O(1/T ) convergence rate under asynchronous federated settings, which theoretically validates the effectiveness and stability of AlignFed. Detailed proof is provided in Appendix D. The theoretical results highlight three key insights: 1. Staleness mitigation: The contractive alignment (ρ < 1) reduces the staleness penalty by a factor of ρ2 , outperforming conventional methods that only penalize stale updates via weight decay. Additionally, the approximate linearity of the feature space (A7) ensures the linear transformation preserves feature information (i.e., semantic preservation), confirming the design choice in the framework section—linear transformations are sufficient to align stale updates without losing critical taskrelated information. 2. Heterogeneity resilience: Intra-group centering and representation regularization bound client drift, ensuring the global model does not diverge due to non-IID data. 3. Fairness guarantee: The bounded weight ratio prevents fast clients from dominating aggregation, while the lower bound of αi ensures slow clients’ valid updates are retained. Notably, AlignFed achieves the same asymptotic convergence rate as synchronous federated learning and outperforms naive asynchronous methods (e.g., FedAsync) by reducing the staleness penalty via contractive alignment. This confirms that the proposed multi-stage alignment mechanism effectively addresses the three core challenges without sacrificing convergence speed or scalability.
8
V. E XPERIMENTS We evaluate AlignFed across diverse federated edge scenarios to comprehensively validate its effectiveness, stability, and generalization. Specifically, we conduct four sets of experiments: (1) convergence analysis to compare synchronous and asynchronous training; (2) robustness testing under varying staleness tolerance levels; (3) performance comparison with state-of-the-art federated LLM fine-tuning methods; and (4) cross-model validation on a second LLM backbone to verify model-agnostic behavior. A. Experimental Setup All experiments were conducted on a single NVIDIA A100 GPU using two large language models: Llama3-8B and Qwen3-8B. We follow the FederatedScope-LLM framework [41] for system simulation, adopting its heterogeneity settings. All models were trained in FP16 with a maximum sequence length of 650 and batch size of 1, emulating constrained edge devices. Client-to-server communication per round was limited to ≤ 50MB. We evaluate on three datasets: GSM8K [42] (mathematical reasoning), CodeAlpaca [43] (code generation), and Dolly [44] (instruction following). Following prior work, each dataset was partitioned into 50 clients using a Dirichlet distribution with α = 0.5 to simulate non-IID federated environments. For parameter-efficient fine-tuning, we apply LoRA updates on the query and value projection matrices. Each client performs local LoRA fine-tuning and uploads its lightweight update asynchronously to the server.
Fig. 2. Comparison of synchronous vs. asynchronous federated LoRA finetuning on Llama3-8B.
Overall, the results demonstrate that asynchronous federated fine-tuning reduces stale gradient effects and achieves faster, smoother convergence under real-world heterogeneity. C. Robustness to Staleness We further evaluate the robustness of AlignFed by varying the maximum staleness tolerance parameter τmax , which restricts the maximum allowed version gap (τi = t−vi ) between a client’s local model (version vi ) and the current global model (version t). Table II reports pass@1 and pass@10 on CodeAlpaca under τmax ∈ {1, 2, 3, 4, 5}. AlignFed remains stable across all staleness levels. Performance fluctuations are small even at τmax = 5. This validates that the proposed multi-stage alignment mechanism effectively mitigates model drift caused by stale updates.
B. Convergence Analysis on Llama3-8B Due to the heterogeneous computation speeds and non-IID data distributions across edge clients, synchronous FFT often suffers from delayed global updates and amplified client drift. To evaluate the impact of asynchrony under such conditions, we compare synchronous and asynchronous federated LoRA fine-tuning on Llama3-8B across three datasets. Fig. 2 compares synchronous and asynchronous federated LoRA fine-tuning on Llama3-8B across the three datasets. Because asynchronous aggregation updates the global model whenever a subset of clients (about 30 per round) returns updates—rather than waiting for all 50 clients as in synchronous FFT—it achieves significantly faster and more stable convergence. On GSM8K, synchronous training shows strong oscillations (49.58→50.44→48.99) caused by model version drift, while asynchronous updates maintain both higher and more stable performance (50.93–51.51). On Dolly, both methods converge to similar accuracy (≈ 58.5%), consistent with the lower gradient variance of instruction-following tasks. The largest gain appears on CodeAlpaca: asynchronous fine-tuning steadily improves to 59.76, while synchronous optimization remains noisy and suboptimal. This confirms that code-generation tasks, which exhibit stronger client-specific specialization, are particularly vulnerable to gradient inconsistency under synchronous FedAvg [45].
TABLE II ROBUSTNESS OF A LIGN F ED TO STALENESS TOLERANCE τmax ON C ODE A LPACA . Staleness τmax 1 2 3 4 5
pass@1 34.88 34.21 35.06 34.02 36.16
pass@10 58.54 58.54 59.76 57.32 55.49
D. Comparison with State-of-the-Art Federated LLM Methods We further compare AlignFed with representative federated LLM fine-tuning baselines, including: • FedAvg [45]: Standard synchronous federated averaging baseline. • FedAvg+FedBuff: FedAvg combined with FedBuff [14] to handle asynchronous updates. • FFA-LoRA [6]: State-of-the-art federated LoRA finetuning method focusing on client drift mitigation. • FFA-LoRA+FedBuff: FFA-LoRA integrated with asynchronous buffering mechanism. • FedSA-LoRA [31]: Federated semantic alignment method for LLM fine-tuning. • FedSA-LoRA+FedBuff: FedSA-LoRA with asynchronous buffering.
9
To ensure fair comparison with existing federated LLM methods, we strictly follow the client partition setup from FederatedScope-LLM [41]: GSM8K (3 IID clients), CodeAlpaca (9 non-IID clients by programming languages), and Dolly (8 task-specialized clients). In contrast, we use 50 clients in our convergence, robustness, and cross-model experiments to simulate large-scale heterogeneous edge environments with numerous devices, which better reflects real-world asynchronous federated learning scenarios and validates the scalability of AlignFed. TABLE III P ERFORMANCE COMPARISON OF A LIGN F ED WITH STATE - OF - THE - ART FEDERATED LLM METHODS ON L LAMA 3-8B. Method
GSM8K Accuracy
CodeAlpaca pass@1 pass@10
Dolly Score
FedAvg FedAvg+FedBuff FedAvg+AlignFed
50.44 50.42 50.87
37.93 37.50 37.13
59.76 59.76 59.15
58.49 57.89 58.52
FFA-LoRA FFA-LoRA+FedBuff FFA-LoRA+AlignFed
50.72 52.08 52.69
39.57 39.88 40.30
60.37 61.59 62.80
58.21 57.94 58.32
FedSA-LoRA FedSA-LoRA+FedBuff FedSA-LoRA+AlignFed
51.10 51.48 51.97
38.60 39.21 40.73
60.98 59.15 61.59
58.00 57.71 58.13
AlignFed consistently improves performance across all methods and datasets. Among all variants, FFALoRA+AlignFed achieves the best overall performance with 52.69% on GSM8K, 40.30% pass@1 and 62.80% pass@10 on CodeAlpaca, outperforming all other baselines. On CodeAlpaca, FedSA-LoRA+AlignFed significantly improves pass@1 to 40.73% and pass@10 to 61.59%, surpassing both FedSA-LoRA and FedSA-LoRA+FedBuff, which demonstrates that AlignFed effectively mitigates model and client drift caused by asynchronous model updates, thereby improving generation stability. On Dolly, all AlignFed-enhanced methods maintain or slightly improve performance without obvious degradation, showing that multi-stage alignment preserves the generalization ability for instruction-tuning tasks. Additionally, AlignFed consistently outperforms FedBuff-based variants in most metrics, indicating that targeted multi-stage alignment is more effective than simple buffering for mitigating staleness and heterogeneity in asynchronous federated LLM fine-tuning. E. Cross-Model Generalization: Qwen3-8B To verify that AlignFed is model-agnostic, we perform the same experiments on Qwen3-8B. Table IV summarizes the final synchronous vs. asynchronous performance. Across all datasets, the asynchronous variant slightly but consistently matches or outperforms synchronous method. The improvements are most evident on CodeAlpaca, where pass@1 and pass@10 both increase. These results confirm that AlignFed generalizes effectively to different LLM families, including multilingual architectures such as Qwen3-8B. These findings further demonstrate that AlignFed is not tied to a specific backbone and provides stable improvements across heterogeneous LLM architectures.
TABLE IV P ERFORMANCE COMPARISON ON Q WEN 3-8B ( SYNCHRONOUS VS . ASYNCHRONOUS ). Dataset / Metric
Sync
Async
GSM8K (Accuracy)
87.41
88.32
CodeAlpaca (pass@1) CodeAlpaca (pass@10)
53.29 71.34
53.66 72.56
Dolly (MMLU score)
66.65
66.66
F. Summary of Experimental Findings Across all experiments, AlignFed consistently and reliably demonstrated enhanced performance in the asynchronous federated fine-tuning of Large Language Models (LLMs). Convergence analysis confirmed that employing our proposed multi-stage alignment mechanism for asynchronous training facilitates a faster and more stable optimization process compared to synchronous schemes. Robustness tests indicated that AlignFed maintains robust performance—without experiencing significant fluctuations—even when confronted with severe update staleness. Comparisons with state-of-the-art methods further validated that AlignFed integrates effectively with existing advanced federated LLM fine-tuning techniques, while delivering significantly superior performance compared to other asynchronous baseline methods. Cross-model experiments conducted using the Qwen3-8B model confirmed that AlignFed possesses universal generalization capabilities independent of specific model architectures. Collectively, these results demonstrate that AlignFed effectively addresses the critical challenges facing asynchronous federated learning in real-world scenarios: model drift caused by the accumulation of staleness, client drift resulting from data heterogeneity, and unfair aggregation driven by the dominance of fast clients. VI. C ONCLUSION In this work, we introduced AlignFed, an asynchronous federated fine-tuning framework tailored for large language models in heterogeneous edge environments. Motivated by the limitations of synchronous FFT—particularly the straggler effect and inefficient resource utilization—and the inadequacy of existing asynchronous FL methods in handling LLM-specific challenges, AlignFed is designed to address three fundamental issues: staleness-induced model drift, heterogeneity-amplified client drift, and unfair aggregation due to imbalanced client participation. To this end, AlignFed incorporates a multi-stage alignment mechanism that unifies: (i) version-aware update grouping to control staleness accumulation, (ii) cross-version semantic alignment to ensure representation consistency across model versions, and (iii) fairness-aware aggregation to balance client contributions under heterogeneous participation. By explicitly aligning updates in both temporal and semantic dimensions, AlignFed enables stable and consistent model evolution under asynchronous training. Extensive experiments on multiple datasets and LLM backbones demonstrate that AlignFed achieves faster and more stable convergence, improved robustness under non-IID data
10
distributions, and more balanced performance across heterogeneous clients compared with synchronous FFT baselines. Overall, this work highlights the critical role of joint temporal–semantic alignment in asynchronous federated LLM fine-tuning, and provides a practical pathway toward scalable, efficient, and fair model adaptation for real-world edge intelligence systems. R EFERENCES [1] R. Sathyam and Y. Li, “Foundation models for autonomous driving perception: A survey through core capabilities,” IEEE Open Journal of Vehicular Technology, 2025. [2] M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “Edgeshard: Efficient llm inference via collaborative edge computing,” IEEE Internet of Things Journal, 2024. [3] Y. Shen, J. Shao, X. Zhang, Z. Lin, H. Pan, D. Li, J. Zhang, and K. B. Letaief, “Large language models empowered autonomous edge ai for connected intelligence,” IEEE Communications Magazine, vol. 62, no. 10, pp. 140–146, 2024. [4] D. Yu, Y. Yuan, Y. Zou, X. Zhang, Y. Liu, L. Cui, and X. Cheng, “ Pruning-Based Adaptive Federated Learning at the Edge ,” IEEE Transactions on Computers, vol. 74, no. 05, pp. 1538–1548, May 2025. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ TC.2025.3533095 [5] J. Zhang, S. Guo, Z. Qu, D. Zeng, Y. Zhan, Q. Liu, and R. Akerkar, “ Adaptive Federated Learning on Non-IID Data With Resource Constraint ,” IEEE Transactions on Computers, vol. 71, no. 07, pp. 1655–1667, Jul. 2022. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/TC.2021.3099723 [6] Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving lora in privacy-preserving federated learning,” arXiv preprint arXiv:2403.12313, 2024. [7] Q. Li, M. Zhang, N. Yin, Q. Yin, L. Shen, and X. Cao, “ Asymmetrically Decentralized Federated Learning ,” IEEE Transactions on Computers, vol. 74, no. 08, pp. 2745–2756, Aug. 2025. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/TC.2025.3569185 [8] R. Zhang, X. Wu, Y. Zou, Z. Xie, P. Li, X. Cheng, F. Dressler, and D. Yu, “Fed-raa: Resource-adaptive asynchronous federated edge learning with theoretical guarantee,” IEEE Transactions on Mobile Computing, 2025. [9] Z. Ji, L. Chen, N. Zhao, Y. Chen, G. Wei, and F. R. Yu, “Computation offloading for edge-assisted federated learning,” IEEE Transactions on Vehicular Technology, vol. 70, no. 9, pp. 9330–9344, 2021. [10] S. Zheng, Z. Zhang, Y. Deng, G. Min, and L. Cui, “ Communication-Efficient Federated Learning by Exploiting SpatioTemporal Correlations of Gradients ,” IEEE Transactions on Computers, vol. 75, no. 04, pp. 1433–1445, Apr. 2026. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/TC.2026.3654074 [11] S. Samarakoon, M. Bennis, W. Saad, and M. Debbah, “Distributed federated learning for ultra-reliable low-latency vehicular communications,” IEEE Transactions on Communications, vol. 68, no. 2, pp. 1146–1159, 2019. [12] Y. Xu, Z. Ma, H. Xu, S. Chen, J. Liu, and Y. Xue, “Fedlc: Accelerating asynchronous federated learning in edge computing,” IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5327–5343, 2023. [13] C.-S. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” in Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), 2019, pp. 2021–2030. [14] D. C. Nguyen, M. Ding, P. N. Pathirana, and A. Seneviratne, “Federated learning with buffered asynchronous aggregation,” in Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2022, pp. 2146–2154. [15] B. C. Gül, S. Tziampazis, N. Jazdi, and M. Weyrich, “Syncfed: Timeaware federated learning through explicit timestamping and synchronization,” arXiv preprint arXiv:2506.09660, 2025. [16] A. Forootani and R. Iervolino, “Asynchronous federated learning with non-convex client objective functions and heterogeneous dataset,” IEEE Transactions on Artificial Intelligence, 2025. [17] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022. [18] Y. Jiang, G. Rajendran, P. Ravikumar, B. Aragam, and V. Veitch, “On the origins of linear representations in large language models,” arXiv preprint arXiv:2403.03867, 2024.
[19] J. Zhang, S. Li, H. Huang, X. Yu, R. K. Gupta, and J. Shang, “Orthogonal calibration for asynchronous federated learning,” arXiv preprint arXiv:2502.15940, 2025. [20] D. Solans, M. Heikkila, A. Vitaletti, N. Kourtellis, A. Anagnostopoulos, I. Chatzigiannakis et al., “Non-iid data in federated learning: A survey with taxonomy, metrics, methods, frameworks and future directions,” arXiv preprint arXiv:2411.12377, 2024. [21] X. Chang, M. Yao, S. Krishnamurthy, C. R. Shelton, A. Chakraborty, A. Swami, S. Oymak, and A. Roy-Chowdhury, “Mitigating participation imbalance bias in asynchronous federated learning under client heterogeneity,” 2026. [Online]. Available: https://openreview.net/forum? id=JOeW5Jg7ye [22] Y. Wang, S. Wang, S. Lu, and J. Chen, “Fadas: Towards federated adaptive asynchronous optimization,” arXiv preprint arXiv:2407.18365, 2024. [23] J. Ma, A. Tu, Y. Chen, and V. J. Reddi, “Fedstaleweight: Buffered asynchronous federated learning with fair aggregation via staleness reweighting,” arXiv preprint arXiv:2406.02877, 2024. [24] A. Forootani and R. Iervolino, “Asynchronous federated learning: A scalable approach for decentralized machine learning,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2026. [25] W. Liu, J. Chen, B. Wang, G. Zai, W. She, and Z. Tian, “Feddm: A discrepancy-aware federated learning method based on multi-branch feature fusion for non-iid data environments,” IEEE Internet of Things Journal, 2025. [26] J. Liu, Y. Liao, H. Xu, Y. Xu, J. Liu, and C. Qian, “Adaptive parameterefficient federated fine-tuning on heterogeneous devices,” IEEE Transactions on Mobile Computing, 2025. [27] X. Yuan, R. Xu, X. Yu, C. Xu, S. Ji et al., “Fedpetuning: When federated learning meets parameter-efficient tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 11 740–11 749. [28] H. Zhang, W. Liu, C. Zhang, Y. Xu, and F. Li, “Fedlora: Efficient federated fine-tuning of large language models via low-rank adaptation,” IEEE Transactions on Neural Networks and Learning Systems, 2025, early Access. [29] K. Kuo, A. Raje, K. Rajesh, and V. Smith, “Federated lora with sparse communication,” arXiv preprint arXiv:2406.05233, 2024. [30] J. Zhang, J. You, A. Panda, and T. Goldstein, “Lori: Reducing crosstask interference in multi-task low-rank adaptation,” arXiv preprint arXiv:2504.07448, 2025. [31] P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu, “Selective aggregation for low-rank adaptation in federated learning,” in The 13th International Conference on Learning Representations (ICLR)(24/04/202528/04/2025, Singapore), 2025. [32] C. Wu, F. Wu, L. Lyu, Y. Huang, and X. Xie, “Communication-efficient federated learning via knowledge distillation,” Nature communications, vol. 13, no. 1, p. 2032, 2022. [33] H. Mostafa, “Robust federated learning through representation matching and adaptive hyper-parameters,” arXiv preprint arXiv:1912.13075, 2019. [34] F. Yu, W. Zhang, Z. Qin, Z. Xu, D. Wang, C. Liu, Z. Tian, and X. Chen, “Fed2: Feature-aligned federated learning,” in Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2021, pp. 2066–2074. [35] M. Mohri, G. Sivek, and A. T. Suresh, “Agnostic federated learning,” in ICML, 2019. [36] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Fair resource allocation in federated learning,” in ICLR, 2019. [37] Y. Chen, W. Huang, and M. Ye, “Fair federated learning under domain skew with local consistency and domain diversity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 077–12 086. [38] H. Zeng, Z. Yue, Y. Zhang, L. Shang, and D. Wang, “Fair federated learning with biased vision-language models,” in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 10 002– 10 017. [39] B. Na, H. Na, Y. Kim, S. Jo, H. Bae, M. Kang, and I.-C. Moon, “Semantic-aware wasserstein policy regularization for large language model alignment,” arXiv preprint arXiv:2602.01685, 2026. [40] B. Xiong, Y. Xu, X. Yang, Y. Song, Y. Wang, and C. Xu, “A step toward federated pretraining of multimodal large language models,” arXiv preprint arXiv:2603.26786, 2026. [41] W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 5260–5271.
11
[42] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. [43] S. Chaudhary, “Code alpaca: An instruction-following llama model for code generation,” GitHub Repository, 2023, accessed: 2025-06-02. [Online]. Available: https://github.com/sahil280114/codealpaca [44] M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin, “Free dolly: Introducing the world’s first truly open instruction-tuned llm,” Databricks Blog, 2023, accessed: 2025-06-02. [Online]. Available: https://www.databricks.com/blog/2023/ 04/12/dolly-first-open-commercially-viable-instruction-tuned-llm [45] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics. PMLR, 2017, pp. 1273– 1282.
12
A PPENDIX
C. Proof of Lemma IV.3 Proof. Step 1 (Lower bound). By definition:
A. Proof of Lemma IV.1
˜ i ∥2 + ϵ) ωi ξi /(∥∆ . αi = P ˜ j ωj ξj /(∥∆j ∥2 + ϵ)
Proof. Step 1. From the global update rule: Wg(k+1) = Wg(k) + η∆(k) g .
(19)
By (A5) and convex combination: X ˜ i ∥ ≤ ∆max . ∥∆(k) αi ∥∆ g ∥≤
(20)
i
Thus, ∥Wg(t) − Wg(vi ) ∥ ≤
t−1 X
η∆max = ητi ∆max ≤ ητmax ∆max .
k=vi
(31)
˜ i ∥2 ≤ ∆max : Using ωi ≥ ωmin , ξi ≥ ξmin , ∥∆ , X ωmin ξmin ωj ξj αi ≥ . ˜ ∆max + ϵ ∥∆j ∥2 + ϵ j
(32)
Step 2 (Ratio bound). Direct substitution gives: ˜ j ∥2 + ϵ αi ωi ξi ∥∆ = · ˜ i ∥2 + ϵ αj ωj ξj ∥∆ ˜ j ∥2 + ϵ ωmax ξmax ∥∆ ≤ . · ˜ i ∥2 + ϵ ωmin ξmin ∥∆
(33) (34)
(21) Step 2. By L-smoothness (A1): ˜ i − ∆∗i ∥ ≤ L∥Wg(t) − Wg(vi ) ∥ ≤ ηLτmax ∆max . ∥∆
D. Proof of Theorem IV.1 (22)
By triangle inequality: ˜ i ) − ∆∗i ∥ ∥Tvi →t (∆ ˜ i ) − T (∆∗ )∥ + ∥T (∆∗ ) − ∆∗ ∥. ≤ ∥T (∆ i
i
i
(23) (24)
Using contractivity (A4) and linear error (A7): ˜ i − ∆∗i ∥ + ϵlin . ≤ ρ∥∆
(25)
Substituting the bound yields: ≤ ρηLτmax ∆max + ϵlin .
(26)
Proof. The expectation E is taken over client sampling, stochastic gradient noise, and staleness. Step 1 (Smoothness descent). By L-smoothness (A1), we have: Lη 2 2 F (W t+1 ) ≤ F (W t ) − η⟨∇F (W t ), ∆(t) ∥∆(t) g ⟩+ g ∥ . 2 (35) Step 2 (Inner product control). Applying Young’s inequality: 1 1 (36) −⟨a, b⟩ ≤ ∥a∥2 + ∥b∥2 , 2 2 we obtain: 1 1 (t) 2 t 2 ⟨∇F (W t ), ∆(t) (37) g ⟩ ≥ − ∥∇F (W )∥ − ∥∆g ∥ . 2 2 Substituting into Step 1 and rearranging yields: 2(F (W t ) − F (W t+1 )) 2 + (Lη + 1)∥∆(t) g ∥ . η (38) Step 3 (Error decomposition). Decompose the aligned update: ˜ i ) = ∆∗i + estale Tvi →t (∆ + ealign , (39) i i ∥∇F (W t )∥2 ≤
B. Proof of Lemma IV.2 Proof. From representation regularization: ˜ i ∥2 ≤ Llocal − Ltask . λ∥∆
(27)
Rearranging gives: r ˜ i∥ ≤ ∥∆
Llocal − Ltask . λ
(28)
Combining with (A5): ( ˜ i ∥ ≤ min ∆max , ∥∆
r
Llocal − Ltask λ
)
where: (t) (t) ∗ ∗ • ∆i : ideal update at Wg , satisfying E[∆i ] = ∇F (Wg ) (t) and E∥∆∗i − ∇F (Wg )∥2 ≤ σ 2 (A2); stale • ei : bounded staleness error from Lemma IV.1; align • ei : alignment residual with ∥ealign ∥ ≤ ϵlin (A7). i The aggregated update satisfies X X X ∆(t) αi ∆∗i + αi estale + αi ealign , (40) g = i i i
.
(29)
˜ i ) − ∇Fi (W )∥ ≤ L∥∆ ˜ i ∥. ∥∇Fi (W + ∆
(30)
By L-smoothness (A1):
i
i
P
where αi ≥ 0 and i αi = 1 (Lemma IV.3). Using the inequality ∥x+y +z∥2 ≤ 3(∥x∥2 +∥y∥2 +∥z∥2 ), we obtain: 2 2 2 X X X 2 . ∥∆(t) αi ∆∗i + αi estale + αi ealign g ∥ ≤3 i i i
i
i
(41)
13
We bound each term separately: (i) Variance term. Since ∆∗i are unbiased estimators and independent across clients, 2
X
E
αi ∆∗i
≤ ∥∇F (Wg(t) )∥2 +
i
σ2 , N
(42)
P where we use E[αi ] = 1/N and i αi2 ≤ 1/N . (ii) Staleness error. From Lemma IV.1, 2 ∥estale ∥2 ≤ O(η 2 L2 τmax ρ2 ∆2max ). i
Thus,
(43)
2
X
2 ≤ O(η 2 L2 τmax ρ2 ∆2max ).
αi estale i
(44)
i
(iii) Alignment error. 2
X
αi ealign i
≤ ϵ2lin .
(45)
i
Combining all bounds: 2 (t) 2 E∥∆(t) g ∥ ≤ O ∥∇F (Wg )∥ 2 σ +O N 2 2 2 + O(η L τmax ρ2 ∆2max ) + O(ϵ2lin ).
(46)
Step 4 (Plug in and absorb). Substitute into Step 2: 2E[F (W t ) − F (W t+1 )] η + (Lη + 1) · O(∥∇F (W t )∥2 ) 2 σ 2 + O(η 2 L2 τmax ρ2 ∆2max ) + O(ϵ2lin ). +O N (47) 1 Using η ≤ 2L , we have (Lη + 1) · c < 1 for some constant c, thus the gradient term can be absorbed to the left-hand side. Step 5 (Telescoping sum). Summing over t = 0, . . . , T − 1:
E∥∇F (W t )∥2 ≤
T −1 X
E[F (W t ) − F (W t+1 )] ≤ F (W 0 ) − F ∗ ,
(48)
t=0
where F ∗ is the lower bound of F . Dividing by T , we obtain: T −1 1 X 1 2 (t) 2 E∥∇F (Wg )∥ ≤ O + O η 2 L2 τmax ρ2 ∆2max T t=0 T 2 σ +O + O(ϵ2lin ), N (49)