SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

arXiv:2608.10144v1 [cs.LG] 10 Aug 2026

SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

Yue Xia Technical University of Munich Munich, 80333 Germany yue1.xia@tum.de

Tayyebeh Jahani-Nezhad Technische Universität Berlin Berlin, 10587 Germany t.jahani.nezhad@tu-berlin.de

Mayank Bakshi Northern Arizona University Flagstaff, AZ 86011 USA mayank.bakshi@nau.edu

Rawad Bitar Technical University of Munich Munich, 80333 Germany rawad.bitar@tum.de

Abstract We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA, Hu et al. 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate O(1/T ) for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.

1

Introduction

Foundation models such as GPT-4 [1], BERT [2], PaLM 2 [3], Llama 2 [4], Claude 3 [5], and ViT [6] have remarkable learning capabilities and performance across a diverse range of tasks [7–10]. Adapting such foundation models to downstream tasks relies on the availability of vast, potentially heterogeneous datasets and requires tremendous computational resources, especially when fine-tuning all model parameters. Since fine-tuning all model parameters requires substantial computational and memory resources, parameter-efficient fine-tuning (PEFT) methods that adapt a pretrained model by updating only a small, structured subset of parameters, are employed, see [11–22]. Training on large amounts of distributed data can be achieved using Federated learning (FL) [23]. FL is a distributed learning paradigm in which a central server (federator) coordinates the training of a neural network on heterogeneous data generated and owned by participating clients. The clients keep their data, train the neural network locally, and share only their model updates with a central server. As such, FL enhances data privacy by keeping raw data local, reduces communication and storage costs associated with transferring data to a central server, and scales well due to its ease of implementation at the client. As such, FL has attracted significant attention from the scientific community and is being applied across several domains [24–26]. Preprint.

Combining PEFT with FL span adapter-based methods [27, 28], prompt tuning [29, 30], selective parameter optimization [31], and low-rank gradient subspace optimization [32]. We focus on federated PEFT methods using Low-Rank Adaptation (LoRA) [17], which has emerged as one of the most popular PEFT techniques, e.g., [19, 20, 33–37] and used in FL, e.g., [38–44]. A detailed literature review is provided in Section6. Combining LoRA with FL introduces challenges that do not arise in either centralized LoRA or conventional FL alone. The main reason is that, unlike standard FL, LoRA updates cannot be averaged directly. In LoRA, for each layer of the network, each client i computes an update of the form ∆Wi = Bi Ai , where the model update ∆Wi ∈ Rd×s is represented as a multiplication of two factor matrices called adapters Bi ∈ Rd×r , Ai ∈ Rr×s , for a chosen r ≪ min(d, s). This factorization, together with the choice of r, reduces the number of trainable parameters per layer from ds to (d + s)r and restricts the rank of ∆Wi to r 1 . In LoRA-based FL, the server aims to obtain P P P ∆W ≜ i pi ∆Wi = i pi Bi Ai , where 0 ≤ pi ≤ 1 and i pi = 1. Transmitting ∆Wi to the server is generally undesirable, as it incurs the same communication cost as transmitting a full finetuning update. Instead, the clients can transmit Bi and Ai separately, and the server can aggregate them. The server has two options: (i) compute each ∆WP them to obtain i = Bi Ai ’s and average P ∆W; or (ii) average the Ai ’s and Bi ’s first to obtain A = i pi Ai and B = i pi Bi and compute ∆W = BA. Each approach introduces its own challenges, affecting either the computational efficiency or the accuracy of the aggregated update, which in turn influences the convergence of the federated fine-tuning algorithm. We focus on the following challenges. Challenge 1: Rank-heterogeneity In the rank-heterogeneous setting, clients use different adapter sizes ri to match their local compute budgets or the complexity of their data, resulting in matrices with incompatible dimensions in aggregation. Allowing for rank heterogeneity is crucial. In the original LoRA work [17], it is shown that small adapter ranks can be sufficient for certain tasks. However, this may not always work, especially when the downstream task differs significantly from the pretraining task. Potential solutions include computing and averaging the Wi ’s or zero-padding the matrices to ensure dimension compatibility, both of which introduce approximation errors. Challenge 2: Bilinear mismatch Zero-padding and averaging the adapters first has a lower computational cost, but it does not recover the correct average of Pthe updates, leading to the bilinear mismatch problem (also known as aggregation bias): BA ̸= i pi Bi Ai . The bilinear mismatch problem is exacerbated when considering differential privacy guarantees [40] and the non-linearity of the aggregation does not allow for secure aggregation, typically used in privacy-preserving FL [24]. Closely related work We summarize the closest work on rank-heterogeneous LoRA-based federated PEFT here, provide more details in Section 2.3 and give an extensive review in Section 6. FedIT [38] introduced the use of LoRA without accounting for rank-heterogeneity and the bilinear mismatch. Since then, federated LoRA methods aim at addressing these challenges through various trade-offs. In FLoRA [45], the server stacks the adapters, forming larger matrices, and sends them to the clients. This increases the server-to-client communication cost, but allows the clients to compute the full update. A key difference is that the backbone model is not frozen. Instead, the clients update the backbone model at every iteration and keep re-initializing their adapters. To overcome those limitations, in FlexLora [46] the server computes and averages the updates Wi = Bi Ai ’s to obtain ∆W and applies SVD on ∆W and truncates it to a rank-ri update matrix sent to client i. The computational overhead per layer at the server is O(N drs) for computing the Wi ’s and O(d2 s) for the SVD. In [41], the authors remark that operating in the full model space, i.e., as is done in FlexLoRA, loses information on the cross-relation across clients and only retains the knowledge on the cross-relation between the adapters Bi and Ai . This observation is corroborated in numerical experiments showing the underwhelming performance of such methods. Therefore, in HetLoRA [41], the federator zero-pads the clients’ Ai ’s and Bi ’s and aggregates them using a careful weighted average. The computational complexity of this method is O(N drs) for computing the weights and the global update. Nevertheless, this method does not mitigate the bilinear mismatch problem. In FSLoRA [47] the training objective is modified. Sketching matrices are embedded into the forward pass as follows. A global rank r is fixed, and random diagonal r × r matrices with only ri non-zero 1We omit the layer index as the operation is the same for all layers. At the end, the clients concatenate its adapters across

all layers and send them to the server.

2

Table 1: Comparison of rank-heterogeneous federated LoRA aggregation methods. We report the computation cost at the server in the regime d ≫ N + r. The parameters are: sketch dimension by k = O(r), number of clients N , number of rounds T , local LoRA rank ri , and sketching-ratioe and c is a non-negative constant. adjusted smoothness L Method

Bilinear mismatch

Aggregation in full space

Server compute cost

Server-client communication cost

Convergence⋆ rate

HetLoRA [41] FlexLoRA [46] FLoRA [45] FSLoRA [47]

Yes No No Yes

No Yes Yes No

ri (d + s) Pri (d + s) i ri (d + s) r(d + s)

— — O(1/T √ ) e NT ) O(L/

SeFoRA (Ours)

No

No

O(N rds) O(N (d + s) + d2 s) O(1) O(N r(d+s))
O (N +r) r (d+s)

dr + ks

O(1/T + c)

⋆ FLoRA guarantees hold for strongly convex losses, FSLoRA guarantees hold for the modified training, and our guarantees hold for the rank-homogeneous setting.

entries are chosen as sketching matrices called Si . Altering the update to Bi Si Ai ensures that client i trains only ri dimensions. To our knowledge, there is no rank-heterogeneous LoRA-based federated PEFT that mitigates the bilinear mismatch and does not operate in the full model space. Besides, only FSLoRA [47] provides theoretical insights into the convergence of the rank-heterogeneous algorithm for non-convex loss functions. However, this analysis comes at the cost of fixing the global matrix and choosing a random subspace to train per iteration, which reduces the flexibility of allowing the clients to choose their own subspaces to train on. A convergence analysis for FLoRA is given for strongly convex loss functions, which does not match typical neural network optimization constraints. Our contributions We introduce SeFoRA, a rank-agnostic federated LoRA-based PEFT method that accounts for rank heterogeneity, alleviates the bilinear mismatch, does not operate in the full model space, and enables linear aggregation at the federator. Instead of transmitting Ai and Bi , each client sketches its local update ∆Wi into two compact, fixed-size matrices using the sketching method proposed in [48]. Since the sketching operator in [48] is linear, aggregating the sketches at the server is equivalent to computing the sketch of the aggregated update ∆W exactly, thereby overcoming the bilinear mismatch problem with low computational overhead. The main properties and advantages of SeFoRA can be summarized as follows: 1. SeFoRA aggregates fixed-dimensional sketches, irrespective of clients’ local ranks. Importantly, it can be seamlessly combined with adaptive rank-truncation at the clients. 2. We give a rank-homogeneous version called SeFoRA-Ho and provide a convergence guarantee on the original LoRA objective. SeFoRA-Ho converges at rate O(1/T + c), where c is a sketching-dependent error floor. We bound this error using the spectral tail energy of the aggregated client updates. 3. Numerical experiments fine-tuning RoBERTa-Large on GLUE datasets in the rankheterogeneous and rank-homogeneous settings show that SeFoRA and SeFoRA-Ho outperform existing methods. Interestingly, in some settings FlexLoRA and SeFoRA have similar accuracies, despite the difference in computation cost at the server. Similarly, FedIT and SeFoRA-Ho have comparable accuracies in some settings. However, FedIT does not allow for rank heterogeneity. 4. Although this work is not directly concerned with privacy guarantees, mitigating the bilinear mismatch and allowing for linear aggregation open the door for secure aggregation and more efficient differential privacy mechanisms that we will explore in future work.

3

2

Setup and Problem Formulation

In this section, we formally define the setting and present the mathematical formulation of the problem. In the sequel, we provide a detailed review of the literature, summarize our algorithm, and present a formal statement of our main results. 2.1

Notation

For an integer N , let [N ] = 1, · · · , N . Vectors and matrices are denoted by bold lowercase and uppercase letters, respectively. For a matrix A with rank r, we denote its singular values σ1 (A) ≥ pPas r 2 · · · ≥ σr (A) ≥ 0. Let ∥A∥2 = σ1 (A) denote the spectral norm and ∥A∥F = i=1 σi (A) the Frobenius norm. For a matrix A, A:,1:r and A1:r,: denote the submatrices formed by its first r columns and first r rows, respectively, and A1:r,1:r denotes its leading r × rp submatrix. For a P 2 tuple of matrices B = (B1 , · · · , BL ), we let ∥B∥2 ≜ maxl∈[L] ∥Bl ∥2 , ∥B∥F ≜ l ∥Bl ∥F , and we say that B has rank r if rank(Bl ) ≤ r for every l ∈ [L]. The transpose of a tuple B is denoted by B⊤ ≜ ((B1 )⊤ , · · · , (BL )⊤ ). The product and addition of two tuples of matrices A and B are defined as BA = (B1 A1 , · · · , BL AL ) and B + A = (B1 + A1 , · · · , BL + AL ), respectively. The inner PL PL ⊤ product of two tuples A and B is ⟨A, B⟩ ≜ l=1 ⟨Al , Bl ⟩ = l=1 tr((Al ) Bl ). 2.2

Setup

Federated learning setup We consider an FL system consisting of a central server (the federator) and N clients, indexed by i ∈ [N ]. Each client i ∈ [N ] holdsSa local dataset Di ⊆ X × Y, with |Di | being the number of data samples held by client i. Let D = i∈[N ] Di denote the global dataset and let (x, y) ∈ Di denote a single data sample in the dataset Di , where x is the feature vector and y is the label. Let f (W; x) : W × X → Y denote the output of a neural network parameterized by W ∈ W for an input x. Let ℓ : Y × Y → R be the sample-wise loss function that measures the difference between the predicted label f (W; x) and the true label y. The local empirical risk Li (W) for client i is defined as X 1 ℓ(f (W; x), y), Li (W) ≜ |Di | (x,y)∈Di

PN and the global empirical risk is defined as L(W) ≜ i=1 pi Li (W), where the weights pi satisfy PN 0 ≤ pi ≤ 1 and i=1 pi = 1, and are typically chosen proportional to the local dataset sizes, i.e., | pi = PN|Di|D . The goal of FL is to collaboratively learn a model W⋆ that minimizes the global | i=1

i

empirical risk, i.e., find W⋆ ∈ argminW∈W L(W). Fine-tuning and LoRA To fine-tune the pre-trained model for a downstream task, we focus on a subset of the parameters. Let a tuple of matrices W = (W1 , · · · , WL ) denote the L linear layers selected for fine-tuning, where for each layer Wl ∈ Rdl ×sl . Let W0 = (W0,1 , · · · , W0,L ) denote the pre-trained values of W and ∆W = (∆W1 , · · · , ∆WL ) be the change of W during fine-tuning, such that W = W0 + ∆W. In LoRA, W0 is frozen, i.e., it does not receive gradient updates during fine-tuning. For each layer l ∈ [L], the weight change ∆Wl is defined to be a low-rank matrix, represented by a product of two factor matrices of smaller dimensions with a maximum rank rl ≪ min(dl , sl ), i.e., Wl = W0,l + ∆Wl = W0,l + Bl Al , where ∆Wl ∈ Rdl ×sl , Bl ∈ Rdl ×rl , and Al ∈ Rrl ×sl . Following standard practice [17], for each layer l, we initialize Al with random Gaussian entries and Bl as the zero matrix, ensuring ∆Wl = 0 at initialization. We denote the tuple of low-rank factor matrices for all the layers as B = (B1 , · · · , BL ) and A = (A1 , · · · , AL ). Federated LoRA In the federated LoRA setting, each client i performs local LoRA fine-tuning using its local tuples of low-rank matrices Bi = (Bi,1 , · · · , Bi,L ) and Ai = (Ai,1 , · · · , Ai,L ), where the local rank profile ri = (ri,1 , · · · , ri,L ) is specific to each client i and may vary across iterations 4

due to heterogeneity in data, computational and memory resources, and the relative importance of different layers during training. A global rank profile r = (r1 , · · · , rL ) is defined such that rl = max ri,l for all l ∈ [L]. We define the local empirical loss for client i evaluated at the local i∈[N ]

parameters (Bi , Ai ) as: Llora i (Bi , Ai ; Di ) ≜

1 |Di |

X

ℓ(f (W0 + Bi Ai ; x), y).

(x,y)∈Di

Let the global empirical loss for full fine-tuning over the general update tuple ∆W be 1 X Lfull (∆W; D) ≜ ℓ(f (W0 + ∆W; x), y). |D| (x,y)∈D

Under the LoRA factorization, the global empirical loss evaluated at the global parameters (B, A) is equivalent to the weighted sum of the local empirical losses at (B, A): Llora (B, A; D) ≜ Lfull (∆W; D) =

N X

pi Llora i (B, A; Di ),

i=1

To minimize the global objective Llora (B, A), in each communication round t ∈ {0, · · · , T − 1}, the federator broadcasts the current global parameters (B(t) , A(t) ) to the clients. Each client i ∈ [N ] (t,0) (t,0) initializes its local parameters as (Bi , Ai ) = (B(t) , A(t) ), and performs E local steps of mini(t,e) (t,e) batch stochastic gradient descent (SGD). We denote by (Bi , Ai ) the local parameters at step e ∈ {0, · · · , E − 1} during communication round t. (e)

At each local step e, client i samples a mini-batch of data ξi ⊂ Di and computes the stochastic gradient of the local objective. The local update rules are given by: (t,e+1)

Bi

(t,e)

= Bi

(t,e)

− η (t) ∇B Llora i (Bi

(t,e)

, Ai

(e)

; ξi ),

(t,e+1)

Ai

(t,e)

= Ai

(t,e)

− η (t) ∇A Llora i (Bi

(t,e)

, Ai

(e)

; ξi ),

where η (t) > 0 is the learning rate at round t. After E local steps, the resulting updated local (t+1) (t,E) (t+1) (t,E) parameters Bi = Bi and Ai = Ai are sent to the federator for aggregation. 2.3

Problem formulation (t+1)

(t+1)

Challenges of LoRA with FedAvg Upon receiving the clients’ Bi and Ai , the federator (t+1) (t+1) needs to compute a global B and A . In rank-homogeneous settings where all clients have the same rank for each layer, the federator could use vanilla FedAvg as in FedIT [38], (t+1) (t+1) i.e., averaging the Bi and Ai and sending them back to the clients. However, this introduces additional errors in the global model update due to the bilinear mismatch, i.e., since P P (t+1) (t+1)
P (t+1)
(t+1) (t+1) ∆W(t+1) ≜ ̸= ∆W ≜ i∈[N ] pi Bi Ai . In ranki∈[N ] pi Bi i∈[N ] pi Ai heterogeneous settings, the adapters have different dimensions and averaging is not directly possible. Hence, an alternative solution is to operate in the full model space. In [45], the federator horizontally (t+1) (t+1) concatenates the Bi matrices and vertically concatenates the Ai matrices to form two large ma(t+1)

trices sent to the clients. Each client computes ∆W locally and applies it to the backbone model W0 , thereby updating the backbone parameters. The main drawbacks are an additional communication overhead and not freezing the backbone model. Hence, in FlexLoRA [39], the federator computes (t+1) ∆W at the server and, using SVD, truncates it to fit each client’s local rank profile. However, as discussed and corroborated numerically in [41], computing and averaging the ∆Wi = Bi Ai loses information on the cross-relation across clients and only retains the knowledge on the cross-relation (t+1) (t+1) between the adapters Bi and Ai . Thus, in HetLoRA, the federator zero-pads the Bi and Ai to (t+1) ensure they have the same dimension, aggregates them to obtain ∆W by determining the weights (t+1) pi ’s according to the norm of the singular value vector of ∆Wi . Then, similarly to FlexLoRA, the federator uses SVD to truncate ∆W(t+1) to the client’s local rank profile. On top of the computation (t+1) overhead at the federator introduced by computing the ∆Wi , this method does not resolve the 5

Algorithm 1 Sketch-inversion Algorithm Require: Given two random matrices Ψ ∈ Rk×d and Ω ∈ Rs×r drawn from a standard normal distribution, and two matrices Y ∈ Rd×r and Z ∈ Rk×s , where Y = WΩ and Z = ΨW are two sketch matrices produced from the target large matrix W ∈ Rd×s . b ∈ Rd×r and A b ∈ Rr×s of the target matrix W. Ensure: Return the low-rank factors B 1: function U NSKETCHING(Y, Z, Ψ, Ω) b ∼) ← QR(Y) 2: (B, b 3: (U, T) ← QR(ΨB) −1 T b = T (U Z) 4: A b A) b 5: Return (B, 6: end function bilinear mismatch problem. Furthermore, zero-padding the matrices is not always faithful to the rank representation of the clients, which is why a careful computation of the pi ’s is needed and why other works, e.g., [49, 50] replace zero-padding with column repetition. Therefore, the question that remains unanswered is: Research question: Is it possible to construct a rank-heterogeneous LoRA-based FL method that mitigates the bilinear mismatch and does not operate in the full model space? We answer this question in the affirmative as we explain next.

3

Our Algorithm and Contributions

We jointly tackle rank heterogeneity and bilinear mismatch without operating in the full model space. The main ingredient of our algorithm is operating in a sketched-space of the full model using the linear matrix sketching method introduced in [48]. For ease of presentation, we describe the method for a single layer. The same procedure is applied independently to every layer, and the resulting matrices will be concatenated at the end. Therefore, we omit the layer index l. Each client sketches (compresses) its local update ∆Wi ∈ Rd×s into two matrices, Yi ∈ Rd×r and Zi ∈ Rk×s , where r = rl = maxi∈[N ] ri,l is the maximum local rank for each layer and k is a constant satisfying k > r + 1. By aggregating P the Yi and Zi sent by the clients, the federator obtains sketches of the global update ∆W = i∈[N ] pi ∆Wi , from which it computes a low-rank approximation of ∆W, as shown in Algorithm 1. Therefore, this maps local adapters of rank ri to sketches of dimension r and k, respectively, thereby enabling linear aggregation and bypassing the bilinear mismatch while avoiding the aggregation in the full space ∆W. Similar to FSLoRA, this incurs an additional communication overhead proportional to (r − ri ) and (k − ri ). Before presenting our theoretical guarantees and demonstrating that our method outperforms the state of the art, we first explain the sketching method introduced in [48]. Sketching as Low-Rank Approximation [48] Suppose W ∈ Rd×s is an arbitrary matrix. Let r ≪ min{d, s} be the target rank. Given a sketch parameter k, let Ω ∈ Rs×r and Ψ ∈ Rk×d be independent random matrices whose entries are drawn independently from the standard normal distribution, two sketch matrices of W are produced via left and right matrix multiplication as: Y ≜ WΩ

and

Z ≜ ΨW.

c of W with Given the sketch matrices, Y ∈ Rd×r , and Z ∈ Rk×s , the low-rank approximation W b via QR decomposition; rank r is computed using the following steps: (i) factorize Y into Y = BR b and find the QR decomposition (ΨB) b = UT; (iii) compute A b = T−1 (UT Z); and, (ii) compute ΨB c b b (iv) produce the low-rank approximation W = BA. This sketch-inversion procedure is summarized in Algorithm 1. 3.1

SeFoRA

SeFoRA resolves rank-heterogeneity and bilinear mismatch as follows. 6

Algorithm 2 SeFoRA Require: The pretrained model W0 . Total rounds T , local steps E, and local rank updating threshold τ . In total, N clients, each client i holding their local dataset Di . The aggregation PN (0) (0) (0) weights pi = |Di |/ i=1 |Di |. The local rank profiles ri = (ri,1 , · · · , ri,L ). For l ∈ [L], (0)

set rl ≜ maxi∈[N ] ri,l and kl > rl + 1. Set rmax ≜ maxl∈[L] rl , and kmax ≜ maxl∈[L] kl . (0)

(0)

Initialize global matrices Bl ∈ Rd×rl as a full zero matrix and Al ∈ Rrl ×s as a ran(0) (0) (0) (0) dom Gaussian matrix. B(0) = (B1 , · · · , BL ) and A(0) = (A1 , · · · , AL ). A function T RUNCATE(B, A, ri ) that truncates LoRA factors with rank r to rank ri ≤ r (Algorithm 3), a function U PDATE L OCAL R ANK(Bi , Ai , τ ) that updates the client’s local rank ri and truncate the factors to rank ri (Algorithm 4), and a function U NSKETCHING(Y, Z, Ψ, Ω) that unsketch the sketch matrices (Algorithm 1). 1: for round t = 0, 1, · · · , T − 1 do 2: Federator initializes Ψ(t+1) ∈ Rkmax ×d and Ω(t+1) ∈ Rs×rmax from a standard normal distribution, and send the shared seed that generates Ψ(t+1) and Ω(t+1) to all clients. 3: Federator broadcasts B(t) and A(t) to all clients. 4: for all clients i in parallel do 5: for all adapter layer l ∈ [L] do (t+1) (t+1) (t+1) (t+1) 6: Extract Ωl = Ω:,1:rl ∈ Rs×rl and Ψl = Ψ1:kl ,: ∈ Rkl ×d from the shared seed. 7: 8:

(t)

(t)

Obtain Bl and Al . Initialize local LoRA factors as: if Updating local ranks then (t) (0) Set local rank ri,l = ri,l end if (t) if local rank ri,l < rl then (t)

(t)

(t)

(t)

(t)

(Bi,l , Ai,l ) ← T RUNCATE(Bl , Al , ri,l ) else (t) (t) (t) (t) Set Bi,l = Bl and Ai,l = Al end if 9: 10: 11: 12:

13:

end for (t) (t) (t+1) (t+1) Using Bi,l and Ai,l , perform E local steps to obtain Bi,l and Ai,l for all l ∈ [L]. for all adapter layer l ∈ [L] do Rank update: if Updating local ranks then (t+1) (t+1) (t+1) (t+1) (t+1) (Bi,l , Ai,l ), ri,l ← U PDATE L OCAL R ANK(Bi,l , Ai,l , τ ) end if Compute the sketch matrices (t+1)

= ∆Wi,l

(t+1)

= Ψl

Yi,l

Zi,l 14: 15: 16: 17: 18:

(t+1)

(t+1)

(t+1)

= Bi,l

(t+1)

= (Ψl

Ωl

∆Wi,l

(t+1)

(t+1)

(Ai,l

(t+1)

(t+1)

Bi,l

(t+1)

Ωl

) ∈ Rd×rl ,

(t+1)

)Ai,l

∈ Rkl ×s .

end for (t+1) (t+1) (t+1) (t+1) (t+1) (t+1) Send Yi = (Yi,1 , · · · , Yi,L ) and Zi = (Zi,1 , · · · , Zi,L ) to federator. end for for all adapter layer l ∈ [L] do P P (t+1) (t+1) (t+1) (t+1) Federator aggregates Yl = i∈[N ] pi Yi,l and Zl = i∈[N ] pi Zi,l . (t+1)

(t+1)

19: (Bl , Al 20: end for 21: end for 22: Return (B(T ) , A(T ) )

(t+1)

) ← U NSKETCHING(Yl

(t+1)

, Zl

(t+1)

, Ψl

(t+1)

, Ωl

)

Client rank Each client sets its own local rank profile ri = (ri,1 , · · · , ri,L ). SeFoRA is agnostic to how the clients adjust their local ranks. They may use adaptive techniques such as those proposed 7

in [51, 52], adapting the ranks based on their computational and memory resources as well as the contribution of each layer to fine-tuning in terms of effective rank. Since the local rank may change (t) across rounds and may vary across layers, we formally denote it by ri,l . Client rank update Clients can periodically update the rank of their adapters, e.g., every Tu rounds, to match their computational and memory constraints, and data complexity. As a concrete example, in SeFoRA, we choose the rank update method shown in Algorithm 4. This method can be replaced by other rank-update strategies and is chosen since it allows the local ranks to grow as well as shrink. Remark 1. We note that adaptive LoRA rank allocation has been extensively studied in the centralized LoRA setting, e.g., [20, 21, 53, 39]. Algorithm 4 is one option among many to adjust client’s local ranks based on data complexity. Exploring alternative rank-update mechanisms is an interesting direction for future work. 3.1.1

Algorithm procedure

The procedure is summarized in Algorithm 2 and is explained in detail next. For notational simplicity, we present the algorithm under the assumption that all adapted layers share the same input and output dimensions, i.e., dl = d and sl = s. The method extends directly to heterogeneous dimensions by using compatible sizes. Initialization The federator has the pretrained model W0 . Each client sets an initial local rank (0) (0) (0) (0) profile ri = (ri,1 , · · · , ri,L ) based on local compute and memory constraints, i.e., ri,l is the maximum rank that client i can allocate to layer l. This profile contains the maximum rank each client can allocate per layer and will be communicated to the federator. The federator creates a (0) (0) (0) (0) global rank profile r(0) = (r1 , · · · , rL ) where rl = maxi∈[N ] ri,l for all l ∈ [L]. The rank profiles are fixed throughout training, whereas the active local rank of a client may change across rounds. The federator initializes the adapters B(0) and A(0) . A common choice is to initialize (0) (0) Bl ∈ Rd×rl as the zero matrix and draw the entries of Al ∈ Rrl ×s independently from a (0) (0) Gaussian distribution, where rl ≜ rl = maxi∈[N ] ri,l and kl is a parameter chosen to satisfy kl > rl + 1. Define rmax ≜ maxl∈[L] rl , and kmax ≜ maxl∈[L] kl . The federator draws two independent random matrices Ω(1) ∈ Rs×rmax and Ψ(1) ∈ Rkmax ×d from the standard normal distribution, and communicates them to the clients via a shared random seed. For layer l, the clients (1) (1) (1) (1) use the corresponding submatrices Ωl = Ω:,1:rl ∈ Rs×rl and Ψl = Ψ1:kl ,: ∈ Rkl ×d . In the remaining, we focus on one layer l and one round t. (t)

Client per-round computation At round t, client i receives the global adapters Bl ∈ Rd×rl (t) and Al ∈ Rrl ×s for layer l, as well as a shared seed from which it generates the random matrices Ω(t+1) ∈ Rs×rmax and Ψ(t+1) ∈ Rkmax ×d from the federator. It then extracts the layer-specific (t+1) (t+1) (t+1) (t+1) matrices Ωl = Ω:,1:rl ∈ Rs×rl and Ψl = Ψ1:kl ,: ∈ Rkl ×d . Fine-tuning. If at this round no rank update is required, the client truncates the received adapters (t) (t) using Algorithm 3 to obtain adapters of rank ri,l . The client proceeds to fine-tune the rank-ri,l (t+1)

(t)

adapters for E local iterations and sets its local rank to ri,l = ri,l . If local rank update is required, client i ∈ [N ] does the following. First, before fine-tuning, the local rank is set to the maximum (0) rank budget the client can compute for this single layer, i.e., ri,l , according to its local rank profile. (0)

Then, the client truncates the received adapters to rank ri,l via Algorithm 3, and fine-tunes the adapters for E iterations. After local fine-tuning, client i applies Algorithm 4 to its locally updated (t+1)

adapters Bi,l

(0)

(t+1)

∈ Rd×ri,l and Ai,l

(0)

′ ∈ Rri,l ×s , to determine the new local rank ri,l . The client (t+1)

′ then truncates the locally updated adapters to that rank ri,l and sets its local rank to ri,l before computing the sketches.

8

′ = ri,l

Algorithm 3 Truncation to rank ri < r Require: Given two global LoRA factors B ∈ Rd×r and A ∈ Rr×s , and the target local rank ri . Ensure: Return the truncated local factors Bi ∈ Rd×ri and Ai ∈ Rri ×s so that Bi Ai is a rank-ri approximations of BA. 1: function T RUNCATE (B, A, ri ) If B = 0 then return (Bi , Ai ) = (B:,1:ri , A1:ri ,: ) 2: Form full SVD of A = UΣV⊤ and truncate the SVD matrices to rank ri , i.e., the resulting ⊤ U:,1:ri ∈ Rr×ri , Σ1:ri ,1:ri ∈ Rri ×ri , and V1:r ∈ Rri ×s . i ,: 1/2 3: Bi ← BU:,1:ri (Σ1:ri ,1:ri ) ⊤ 4: Ai ← (Σ1:ri ,1:ri )1/2 V1:r i ,: 5: Return (Bi , Ai ) 6: end function Algorithm 4 Update Local Rank and Return Truncated Updates Require: Given the local LoRA factors with rank r, i.e., Bi ∈ Rd×r and Ai ∈ Rr×s . Ensure: Return local factors with updated local rank ri , i.e., Bi ∈ Rd×ri and Ai ∈ Rri ×s . 1: function U PDATE L OCAL R ANK(Bi , Ai , τ ) ⊤ 2: Find the QR decompositions of Bi and A⊤ i , i.e., Bi = QB RB and Ai = QA RA . r×r ⊤ ⊤ . 3: Form M = RB RA ∈ R , and compute its exact SVD: M = UM ΣVM 4: Extract squared singularPvalues from Σ to find the minimal rank ri such that the cumulative ri

σ2

j singular value energy, i.e., Pj=1 r 2 , exceeds threshold τ (e.g., 90%). j=1 σj 5: Compress the local LoRA factors directly using the truncated singular components:

Bi ← QB (UM ):,1:ri (Σ1:ri ,1:ri )1/2 ∈ Rd×ri Ai ← (Σ1:ri ,1:ri )1/2 (VM )T:,1:ri QTA ∈ Rri ×s 6: Return (Bi , Ai ), ri 7: end function

(t+1)

Adapter sketching. After fine-tuning the adapters, client i computes Ai,l

(t+1)

Ωl

(t+1)

and then Yi,l

=

(t+1) (t+1) (t+1) (t+1) (t+1) (t+1) (t+1) Bi,l (Ai,l Ωl ). Similarly, the client computes Zi,l = (Ψl Bi,l )Ai,l also by first (t+1) computing the multiplication in parentheses to reduce the computation cost. The client sends Yi,l (t+1) and Zi,l to the federator.

Federator aggregation and communication Due to the linearity of the sketching operation, (t+1) (t+1) (t+1) aggregating Yi,l and Zi,l is implicitly aggregating ∆Wi,l but in a smaller subspace. Hence, P P (t+1) (t+1) (t+1) (t+1) the federator computes Yl = i∈[N ] pi Yi,l and Zl = i∈[N ] pi Zi,l and uses them (t+1)

as input for Algorithm 1 to obtain Bl

(t+1)

and Al

. (t+1)

To initiate the next round, the federator transmits the global adapters Bl shared seed for the new random matrices. 3.2

(t+1)

and Al

and a fresh

SeFoRA-Ho: An algorithm for the rank-homogeneous case

To enable comparison with the state-of-the-art rank-homogeneous LoRA-based federated PEFT algorithms and provide theoretical convergence analysis, we provide a rank-homogeneous version of SeFoRA. The main additional benefit of SeFoRA-Ho compared with existing rank-homogeneous algorithms is linearity in aggregation in a small subspace without the bilinear mismatch problem. This property is essential for enabling secure aggregation [24] and mitigates the noise factors resulting from multiplying noisy LoRA adapters when employing differential privacy mechanisms, see e.g., [40]. We will analyze the privacy benefits of SeFoRA-Ho and SeFoRA in future work. 9

(t)

(t)

SeFoRA-Ho is obtained by modifying Step 7 in Algorithm 2 by setting Bi,l = Bl (t) and Ai,l = Al (t) , and removing Step 9. For completeness, SeFoRA-Ho is summarized in Section A. We show in Theorem 1 that this algorithm converges under standard assumptions stated in Section 4.

4

Theoretical Analysis and Insights

This section analyzes the proposed method theoretically. Specifically, we analyze SeFoRA-Ho, the rank-homogeneous version of SeFoRA. We use the tuple notations introduced in Section 2.1 throughout. We first state the smoothness and stochastic gradient assumptions used below. Assumption 1 (Per-sample Lipschitz smoothness). Let ℓ(f (W0 + ∆W; x), y) denote the loss evaluated on a single data sample (x, y). There exists a real value µ > 0 such that for any two model updates ∆W and ∆W′ : ∥∇∆W ℓ(f (W0 + ∆W; x), y) − ∇∆W ℓ(f (W0 + ∆W′ ; x), y)∥F ≤ µ∥∆W − ∆W′ ∥F . Lemma 1. Under Assumption 1, the local empirical loss Lfull i (∆W), the global empirical loss Lfull (∆W), and the stochastic gradient Lfull (∆W; ξ ) evaluated on mini-batch ξi are µ-smooth with i i respect to ∆W, i.e., full ′ ′ ∥∇∆W Lfull i (∆W) − ∇∆W Li (∆W )∥F ≤ µ∥∆W − ∆W ∥F ,

∥∇∆W Lfull (∆W) − ∇∆W Lfull (∆W′ )∥F ≤ µ∥∆W − ∆W′ ∥F , and i h full ′ Eξi ∥∇∆W Lfull (∆W; ξ ) − ∇ L (∆W ; ξ )∥ ≤ µ∥∆W − ∆W′ ∥F . i F i ∆W i i Proof. Each loss in the statement is an average of sample losses. Averaging the inequality in Assumption 1 and using the triangle inequality concludes the proof. Assumption 2 (Unbiased stochastic gradients with bounded moments). For each client i and all ∆W ∈ Rd×s , the stochastic gradient ∇∆W Lfull i (∆W; ξi ) is an unbiased estimator of the local gradient full ∇∆W Li (∆W), i.e., h i Eξi ∇∆W Lfull (∆W; ξ ) = ∇∆W Lfull i i i (∆W). Moreover, there are constants χ, κ > 0 such that h i 2 2 Eξi ∥∇∆W Lfull (∆W; ξ )∥ and i F ≤χ , i

h i 4 4 Eξi ∥∇∆W Lfull (∆W; ξ )∥ i F ≤κ . i

To facilitate the convergence analysis, we adopt the following bounded-factor assumption, which is also used in prior theoretical analyses of federated LoRA [43, 54]. Assumption 3 (Bounded LoRA factors). There exist constants MA , MB > 0 such that, for every layer l, round t, client i, and local step e, the spectral norms of both the local LoRA factors and the (t,e) (t,e) (t) global LoRA factors are bounded, i.e., ∥Ai,l ∥2 ≤ MA , ∥Bi,l ∥2 ≤ MB and ∥Al ∥2 ≤ MA . (t)

Notice that, we have ∥Bl ∥2 ≤ 1 as it is the Q-factor from the QR step at the federator. Assumption 3 implies that √ √ (t,e) (t,e) ∥Ai ∥F ≤ LrMA and ∥Bi ∥F ≤ LrMB , and ∥A(t) ∥F ≤

√ LrMA

and

∥B(t) ∥F ≤

√ Lr.

To quantify the sketching error incurred by reconstructing the aggregated update from its sketches, we recall the following guarantee. Lemma 2 (Low-Rank Approximation (Sketching) Error [48]). Let r ≥ 2, ϱ ∈ {0, · · · , r − 2}, and the sketch parameter satisfy k > r + 1. Draw random matrices Ω ∈ Rs×r and Ψ ∈ Rk×d 10

c obtained from independently from the standard normal distribution. The rank-r approximation W Algorithm 1 satisfies r ϱ 2 c − W∥2 ≤ (1 + E∥W ) · min (1 + ) · τϱ+1 (W), F ϱ<r−1 k−r−1 r−ϱ−1 where the j-th tail energy τj2 is defined as: τj2 (W) ≜

min′

rank(W )<j

∥W − W′ ∥2F =

X

σi2 (W).

i≥j

PL For a tuple W = (W1 , · · · , WL ), use the corresponding tuple tail energy τj2 (W) ≜ l=1 τj2 (Wl ). This definition makes the sketching error and the tail-energy condition below well defined for all adapted layers. This ensures that the same inequality in Lemma 2 holds for the tuple after applying the sketch independently to each layer, using the tuple tail energy defined above. Let Ft be the algorithmic history at the beginning of round t, and write Et [·] = E[· | Ft ]. The next theorem shows that SeFoRA-Ho converges under Assumptions 1–3. Theorem 1. Suppose Assumption 1, Assumption 2, and Assumption 3 hold. Let the global minimum of (t+1) the unregularized loss be Lfull,⋆ ≜ min∆W Lfull (∆W). Take constant learning rate η (t) = η. Let Esketch P (t+1) (t+1) (t+1) 2 denote the reconstruction error, Esketch = B(t+1) A(t+1) − i∈[N ] pi Bi Ai . Let C1 = χ2 MA + 2 4 4 2 2 2 2 4 4 2 2 2 4 2µ (MB + MA ), C2 = χ MBs + 2µ MA (MB + MA ), C3 = MA MB κ , C4 = µχ (MB + MA ),   i h 2 PT −1 h (t+1) i (t+1) (t+1) (t+1) +2µEt ∥Esketch ∥2F , S1 = T1 t=0 E S1 the sketching floor S1 = χ· Et Esketch , F   and ∆0 = E Lfull (∆W(0) ) − Lfull,⋆ . SeFoRA-Ho satisfies:    T −1   2 2 1 X + E ∇B Llora (B(t) , A(t) ) E ∇A Llora (B(t) , A(t) ) T t=0 F F ≤

2∆0 2 2 S1 + ηE · 2 (χC3 + 2C4 ) + η 2 E 2 · χ2 (C1 + C2 ) + 4η 3 E 3 · µC32 . + ηET ηE 3

The proof of Theorem 1 can be found in Section B.1. When we further assume that the spectral tail is bounded as in Assumption 4, we present Corollary 1. Assumption 4. There exist a fixed number ϱ ∈ {0, · · · , r − 2} and a constant τ ≥ 0 such that, for every T ≥ 1,   T −1 X 1 X  2 (t+1) (t+1)  E τϱ+1 ( pi Bi Ai ) ≤ τ 2. T t=0 i∈[N ]

Corollary 1. Suppose Assumption 1, Assumption 2, Assumption 3, and Assumption 4 hold, and define r ϱ ϵ2 ≜ (1 + ) · (1 + ) · τ 2. k−r−1 r−ϱ−1 Choose

s γT ≜

∆0 2 T + χϵ + 2µϵ

χC3 + 2C4

,

and

η=

γT , E

SeFoRA-Ho satisfies:    T −1   2 2 1 X lora (t) (t) lora (t) (t) E ∇A L (B , A ) + E ∇B L (B , A ) T t=0 F F ≤ 4γT (χC3 + 2C4 ) + γT2

2χ2 (C1 + C2 ) + 4γT3 µC32 . 3

The proof of Corollary 1 can be found in Section B.2. 11

5

Experiments

We conduct experiments to evaluate the performance of SeFoRA-Ho in the rank-homogeneous setting and SeFoRA in the rank-heterogeneous setting. We compare the proposed methods against their respective federated LoRA baselines across four natural language understanding datasets from the GLUE benchmark. Models and datasets We employ RoBERTa-Large [55] as the pretrained backbone, and evaluate it on four datasets from the General Language Understanding Evaluation (GLUE) [56] benchmark, i.e., SST-2, QNLI, QQP, and MNLI. SST-2 is a sentiment-classification task, QNLI identifies whether a sentence contains the answer to a question, QQP evaluates whether two sentences are semantically equivalent, and MNLI is a three-class classification task, which evaluates the entailment between two sentences. The LoRA adapters are inserted into the query, key, and value projection matrices of every selfattention layer. The pretrained backbone parameters are frozen during federated fine-tuning, while the adapters and the classification head are optimized. Baselines For the rank-homogeneous case, we compare SeFoRA-Ho with two baselines, namely, FedIT [38] and FFA-LoRA [44]. FedIT aggregates the two LoRA factors uploaded by the clients separately. Although straightforward and communication-efficient, FedIT suffers from the bilinear mismatch. FFA-LoRA eliminates the bilinear mismatch by freezing one of the LoRA factor matrices. For the rank-heterogeneous case, we compare SeFoRA with the three most relevant approaches, i.e., FlexLoRA [46], FLoRA [45] and FSLoRA [47]. FlexLoRA reconstructs each client’s full update from the LoRA adapters, thereby sacrificing server-side computation to the bilinear mismatch. FLoRA concatenates client factors so that the aggregation can be represented exactly, but the effective adapter rank increases with the number of participating clients, and clients re-initialize the adapters in every round. FSLoRA communicates selected components of fixed-rank local adapters, supporting heterogeneous ranks across clients but not rank adaptation during training. In our FSLoRA experiments, each client is assigned a fixed rank at initialization, which remains unchanged throughout all communication rounds. Experimental setup We use 40 clients, all of whom participate in every round. To simulate data heterogeneity, the data is allocated among clients by sampling from a Dirichlet distribution that determines the portion of samples each client receives for each label. The parameter α controls the level of heterogeneity. A smaller α corresponds to a higher level of heterogeneity. We set α = 0.5 to model data heterogeneity. Each client performs local steps E ∈ {1, 5} per communication round. In each local step, clients randomly sample a mini-batch of 100 samples from their local training dataset. We use Adam with a learning rate of η = 9e − 4 across all experiments. The adapter contribution is scaled by 0.1. Although SeFoRA allows clients to have different maximum rank budgets due to heterogeneous computational and memory resources, in these experiments, all clients are assigned the same maximum rank. The random matrices used in the simulation for sketching are independently generated in each round. In the rank updating setting, the rank is updated every 100 rounds, and τ = 90% energy is preserved. Although the algorithm allows layer-specific ranks, in all experiments we use a uniform rank configuration across adapted layers. Specifically, in the SeFoRA-Ho experiments, we set ri,l = ri = r for every l and i. In the SeFoRA experiments, we set ri = maxl ri,l and r = maxi ri . Consequently, each client has a single scalar local rank shared by all adapted layers. Unless stated otherwise, we set the maximum adapter rank to r = 16 and the sketch dimension k = 18. For FlexLoRA and FLoRA, we employ the same rank adaptation mechanism as in SeFoRA. For FSLoRA, heterogeneous client ranks are sampled from the set {2, 4, · · · , 14} and remain fixed throughout training. We set the total number of rounds to be T = 500. We repeat each experiment using three random seeds and report the mean accuracy and standard deviation. Rank-homogeneous results In Table 2, we compare the proposed method SeFoRA-Ho with rankhomogeneous baselines, using r = 16 and k = 18. With one local step, SeFoRA-Ho obtains results 12

Local Steps

Method SST-2 QNLI QQP MNLI FedIT 95.1 ± 0.4 89.5 ± 0.8 83.8 ± 2.2 85.1 ± 2.2 E=1 FFA-LoRA 92.0 ± 0.4 75.3 ± 2.8 76.6 ± 1.9 57.5 ± 1.0 SeFoRA-Ho 95.0 ± 0.2 88.8 ± 0.9 83.1 ± 2.2 84.6 ± 0.5 FedIT 95.4 ± 0.2 93.5 ± 0.1 88.1 ± 0.4 89.2 ± 0.4 FFA-LoRA 94.4 ± 0.3 88.9 ± 0.7 83.5 ± 1.0 84.9 ± 0.4 E=5 SeFoRA-Ho 95.7 ± 0.5 93.7 ± 0.2 87.9 ± 0.4 89.6 ± 0.3 Table 2: We report the accuracy (%) in the rank-homogeneous setting. SeFoRA-Ho uses r = 16 and k = 18.

k SST-2 QNLI QQP MNLI 32 95.5 ± 0.1 93.3 ± 0.3 87.6 ± 0.4 89.5 ± 0.2 r = 16 18 95.5 ± 0.1 93.2 ± 0.2 87.7 ± 0.4 89.4 ± 0.3 16 95.4 ± 0.2 93.1 ± 0.2 87.4 ± 0.4 89.2 ± 0.3 r=8 10 95.5 ± 0.2 93.2 ± 0.1 87.4 ± 0.4 89.1 ± 0.3 8 95.4 ± 0.4 93.0 ± 0.1 87.0 ± 0.5 88.9 ± 0.3 r=4 6 95.5 ± 0.1 93.0 ± 0.4 87.0 ± 0.5 88.9 ± 0.1 r=2 4 95.3 ± 0.6 93.0 ± 0.3 86.5 ± 0.6 88.3 ± 0.1 Table 3: We report the accuracy (%) to illustrate the sensitivity of SeFoRA to the rank r and sketch dimension k, conducting 5 local steps. We vary r ∈ {2, 4, 8, 16} and k ∈ {r + 2, 2r}. rank (r)

close to FedIT, while substantially outperforming FFA-LoRA. Increasing the number of local steps to 5 improves all three methods. FFA-LoRA performs consistently worse than SeFoRA-Ho and FedIT, suggesting that freezing one LoRA factor restricts the model’s expressivity, particularly on more challenging tasks. Rank-heterogeneous results Fig. 1 presents the convergence behavior of SeFoRA, FlexLoRA, FLoRA, and FSLoRA on SST-2 and QNLI. FLoRA converges considerably more slowly, particularly on QNLI and when the local step is 1. This behavior can be explained by the fact that FLoRA integrates the updated global model with the backbone model and reinitializes the local adapters in each round. Using 5 local steps substantially accelerates convergence and improves final performance for all methods. SeFoRA achieves performance comparable to or slightly better than FlexLoRA, without the need to reconstruct all clients’ updates, which incurs a O(d2 s) computational cost at the server. Table 3 studies the influence of the adapter rank r and the sketch dimension k under E = 5. We consider r ∈ {2, 4, 8, 16} and compare k = r + 2 with k = 2r. For a fixed rank r, increasing the sketch dimension from k = r + 2 to 2r produces little or no improvement. These results indicate that k = r + 2 is sufficient to preserve the update information while requiring less communication than k = 2r. The table also shows that small ranks are sufficient for SST-2 and QNLI, whereas the more challenging tasks, such as QQP and MNLI, benefit more from larger adapter ranks. Fig. 2 shows the adaptive rank behavior. Rank adaptation is performed every 100 communication rounds. For the larger initial ranks, the average local rank decreases substantially across the communication rounds. This shows that the adapter updates demand a reduced rank as federated fine-tuning progresses. Comparing rank evolution across datasets, the results suggest that rank adaptation also reflects differences in task complexity. As task complexity increases, the adapter requires a higher rank to express the information contained.

6

Related Work

Parameter-efficient fine-tuning (PEFT). PEFT methods adapt a pretrained model W0 by updating only a small, structured subset of parameters. They fall into three families: additive (inserting trainable modules [11–14]), selective (updating a subset of existing weights [15, 16]), and reparametrized (imposing low-dimensional structure on the update [17–21]); see [22] for a survey. 13

Figure 1: We report the accuracy in the rank-heterogeneous setting. We use r = 16 and k = 18 for SeFoRA. FSLoRA clients use fixed ranks sampled from {2, 4, · · · , 14}.

Figure 2: We report the evolution of the clients’ average adaptive rank for SeFoRA using E = 5 local steps and k = r + 2. Local ranks are updated every 100 communication rounds. Among reparametrized methods, LoRA [17] is perhaps the most popular method due to its simplicity and negligible inference overhead. The weight updates are parametrized as ∆W = BA with factor matrices B ∈ Rd×r , A ∈ Rr×s , r ≪ min(d, s), thus reducing the number of trainable parameters from ds to (d + s)r. Beyond the original LoRA formulation [17], numerous variants have been proposed to improve adaptation efficiency, convergence, and parameter utilization. LoRA+ [33] accelerates optimization by assigning different learning rates to the two low-rank matrices. DoRA [34] improves fine-tuning performance by decomposing pretrained weights into magnitude and directional components, allowing LoRA to optimize only the directional updates. Other methods focus on improving parameter efficiency through adaptive resource allocation. LoRA-drop [35] prunes redundant LoRA parameters based on their contribution to the model output, whereas AdaLoRA [20] dynamically reallocates the adaptation rank across layers according to their relative importance under a fixed parameter budget. HydraLoRA [19] modifies the factorization structure. Furthermore, LoRA has been adopted in application-specific settings, such as automatic prompt engineering for logical reasoning [36]. On the theory side, [37] shows that ℓ2 -regularized LoRA is equivalent to nuclear-norm-regularized full fine-tuning subject to a rank constraint. PEFT and federated learning Federated learning (FL) has become the de facto paradigm for distributed machine learning, enabling collaborative model training while keeping data decentralized. 14

By allowing clients to train models locally without sharing raw data, FL offers better privacy guarantees and has consequently found widespread adoption in privacy-sensitive applications, including finance [57, 58] and healthcare [59–62]. Motivated by the success of large language models (LLMs), recent research has extended FL to support their efficient fine-tuning and deployment [63, 64, 38]. Several frameworks have subsequently been proposed to accelerate research in this emerging area. FederatedScope-LLM [65] provides a comprehensive platform for federated fine-tuning and instruction tuning of LLMs, while OpenFedLLM [66] offers a unified framework for training LLMs over decentralized private data and systematically studies federated instruction tuning and value alignment. Complementing these systems, FedLLM-Bench [67] introduces realistic benchmarks and extensive empirical evaluations for federated fine-tuning, enabling standardized assessment of optimization algorithms and communication-efficient methods. Together, these works have established the foundational infrastructure, benchmarks, and evaluation methodologies that drive the rapidly growing field of federated fine-tuning of LLMs. Combining PEFT with FL enables for adapting large foundation models on vast amount of data while substantially reducing communication overhead, computational complexity, and privacy risks. Existing approaches span adapter-based methods [27, 28], prompt tuning [29, 30], selective parameter optimization [31], and low-rank gradient subspace optimization [32]. More recently, Low-Rank Adaptation (LoRA) [17] has become a popular PEFT technique on its own and in FL. Federated LoRA. Early works such as FedIT [38] integrate LoRA into the standard FedAvg framework for federated instruction tuning, while subsequent methods address the limitations of naive LoRA aggregation under heterogeneous data and system conditions. However, combining LoRA with FL raises several challenges faces challenges that do not exist in centralized LoRA or in federated averaging alone. Namely: the error introduced by the bilinear P mismatch P resulting from first aggregating the Ai ’s and Bi ’s and then computing ∆W = ( i pi Bi )( i pi Ai ) as done P in FedIT [38]; or the rank-r approximation error resulting from first computing ∆W = i pi Bi Ai which is not necessarily rank-r and then computing a rank-r approximation, e.g., using SVD decomposition as in FlexLoRA [46]. In addition, the proposed aggregation rule should account for the rank heterogeneity problem: when clients use different ranks ri , the factor matrices have mismatched dimensions, rendering direct factor averaging infeasible. To overcome those hurdles, several approaches have been proposed, as we detail in the sequel. Beyond model parameterization, several recent works have focused on improving the optimization process itself. FedBCGD [68] introduces an accelerated block coordinate gradient descent algorithm, FedSWA [69] employs stochastic weight averaging to improve generalization under highly non-IID data, FedAdamW [70] proposes a communication-efficient AdamW-inspired optimizer for federated large models, FedNSAM [71] analyzes the relationship between local and global sharpness to improve optimization consistency, FedMuon [72] accelerates convergence through matrix orthogonalization, and DPFedPGN [73] encourages globally flatter minima in differentially private FL by penalizing gradient norms. Client data heterogeneity is tackled in [74]. In LA-LoRA [40], during local iterations, the clients alternately update B and A and send both updated adapters to the federator for aggregation. This alternation is shown to provide stability during training and to allow for differential privacy with lower noise. FedASK [75] uses a double-sketching approach primarily to provide a differential privacy guarantee, but does not account for rank-heterogeneity. Robustness to label noise in federated LoRA is studied in [76]. Personalized federated adaptation is also considered, e.g., [77, 78]. Tackling the bilinear mismatch The bilinear mismatch introduces an error of O(η 2 ) per round for a single local step and grows as O(η 2 E 2 ) for multiple local steps [45, 79] and is more prominent when considering differential privacy guarantees. In [44], FFA-LoRA freezes the initialized adapter A and trains only the adapter B. As such, the federator avoids the bilinear mismatch problem. However, freezing the adapter A reduces the expressiveness of the adaptation and affects the performance. In FedSVD [80], the clients only train B locally, while the global adapter A(t) is computed based on an SVD of the matrix B(t) A(t−1) ; thus enabling the update of the factor A while clients still train and aggregate only B. In a similar spirit, FedSA-LoRA [43] trains A globally while each client trains B locally. A formal convergence analysis; however, since B is different for each client, the algorithm is suitable for personalized FL. The concurrent works introducing LoRA-A2 [81] and RoLoRA [54] alternate between freezing A training B, and training A freezing B. In LoRA-A2 , the federator selects an adaptive rank for each client to handle data heterogeneity. The client updates only a subset of all possible ranks, using a mask to perform the update without modifying the dimensionality of 15

the adapters; however, the paper lacks a theoretical foundation. In RoLoRA, all clients maintain the same rank, and a convergence guarantee is given. In [82], the authors propose FedRot-LoRA, which takes a different approach: aligning local subspaces via orthogonal rotations at the clients’ side before aggregating. FedEx-LoRA [79] keeps the LoRA factors trainable across rounds and absorbs the mismatch into the frozen pretrained weights as a residual term; however, it increases both computation and communication overheads and does not provide a convergence guarantee. This reduces the bilinear mismatch error but does not eliminate it. LoRA-FAIR [83]P introduces P a correction term in B to prevent the BA from deviating too much from the actual update i Bi i Ai . Despite their advantages, those methods are not applicable in rank-heterogeneous settings. Rank-heterogeneous federated LoRA In [84], the authors investigate methods for finding a good initialization of the adapters via full fine-tuning and then running federated LoRA, without altering the federated LoRA process. Adapting the federation of LoRA to account for rank heterogeneity is proving to be more successful. The authors of [45] propose FLoRA, in which the federator stacks the received Ai ’s and Bi ’s from the clients in a way that the multiplication of the stacked matrices emulates the computation and aggregation of the local ∆Wi ’s. As such, the globally updated A and B have larger dimensions, and the resulting ∆W has a rank that could be larger than r; thus increasing the communication and computation cost at the clients. Furthermore, in each round, the updated global model is integrated into the base model and the adapters are reinitialized, thereby potentially losing information about the adapters obtained from the previous rounds. Bai et al. [46] took another approach in their FlexLoRA. The federator computes and aggregates the clients’ ∆Wi ’s, and, for each client, the federator uses SVD to decompose and send a rank-ri approximation of the global ∆W. This method incurs an O(d2 s) computational cost at the server, and the paper analyzes the number of samples the clients need to have to obtain a good generalization bound, but it does not give a proof of convergence for the algorithm. Similarly, in FedMomentum [85], the federator computes and aggregates the local ∆Wi ’s to obtain ∆W and then performs a few mathematical transformations on ∆W to efficiently compute an approximation of its SVD decomposition. A key e 1/2 and A = Σ1/2 V, where U, e difference in this work is computing the global adapters as B = UΣ Σ and V are the matrices obtained from the approximation of the SVD of ∆W. The computation complexity at the federator is O(dN 2 r2 ). Also, no convergence guarantee is given. Similarly, in FedSRD [51] the clients apply local sparsification and pruning techniques to reduce the ranks of their local adapters, and the federator reconstructs the clients’ full update Bi Ai and aggregates in the full-rank space, which increases the computation at the server. The server then computes the SVD of ∆W to compute a low-rank update and uses sparsification techniques to further reduce the communication cost; this work also lacks a convergence analysis. In HetLoRA [41], the clients prune the adapters to smaller ranks by adding a regularization factor to the optimization problem. Then, the federator zero-pads the clients’ Ai ’s and Bi ’s to ensure they have the same dimension. However, when aggregating the zero-padded adapters, a careful weighted average is performed. For each adapter, a weight pi proportional to the norm of the singular value vector of ∆Wi is computed. The computational complexity of this method is O(N drs) for computing the global update. Similarly, no convergence guarantee is given. FSLoRA [47] modifies the update to BSA to embed the sketching matrices §i at the clients and account for rank heterogeneity. The matrices Si are r × r diagonal matrices with only ri non-zero entries √ chosen at random. The algorithm can e be theoretically proven to converge with a rate of O(L/ N T ) when run for T iterations. However, the choice of which ri entries are non-zero is random and not based on the properties of client’s data. The authors of [49] replace the zero-padding by replicating columns of A and B to make the matrices have the same ranks. In RB-LoRA [50], a unifying framework for completing the local adapters (via replication) and stacking them for computation at the federator is proposed. A client’s local adapters are weighted based on the amount of data the client holds, proportional to the total number of data (for A) and to the total number of data held by clients with ranks less than or equal to the client’s (for B). The method’s performance is evaluated via numerical experiments. In [86], the proposed AFLoRA adds a diagonal matrix Λ of dimension r × r to the clients’ local adapters to have ∆Wi = Bi Λi Ai . The diagonal matrix is meant to compute the contribution of each dimension of Bi to the local ∆W. Then, the clients accordingly adjust their local ranks. The adapter A is only trained at the federator using a public dataset, and the Bi and Λi are trained at the clients. To aggregate the local adapters, zero padding is used, and aggregation weights proportional to the clients’ data volume and local LoRA ranks are computed. FedARA [52] adaptively allocates local ranks to each layer at clients, and layers with small ranks are not updated. Through a majority vote at the federator, global layers will also be pruned. However, aggregating client updates still requires 16

zero-padding and matrices of the same dimension. Recent work has further examined the biases caused by heterogeneous ranks [87, 88]. In [87], rank collapse is identified, i.e., the energy of the aggregated update progressively concentrates on the smallest rank shared by the participating clients. They propose raFLoRA, which splits each local update into rank partitions and aggregates them separately. This prevents higher-rank components from being diluted by clients who do not update them. Nevertheless, the server reconstructs the partitioned updates in the full model space and applies an SVD to recover global factors, incurring computational overhead at the server. FedHL [89] focuses on the truncation-induced bias arising in federated LoRA with heterogeneous ranks. The method maintains a full-rank global model as a calibrated aggregation basis, and the truncation bias can be eliminated by using a compensation mechanism. In [90], the authors introduce a subspace regularization term in the objective, making sure the subspaces spanned by the clients’ low-rank updates stay aligned. However, in each round, the clients send ∆Wi = Bi Ai to the federator, increasing the communication overhead between the clients and the federator.

References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019. [3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv preprint arXiv:2305.10403, 2023. [4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [5] Antrhopic. The claude 3 model family: Opus, sonnet, haiku. online. URL https://www-cdn.anthropic. com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. accessed on 03.Aug.2026. [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [7] Desirée Bill and Theodor Eriksson. Fine-tuning a llm using reinforcement learning from human feedback for a therapy chatbot application, 2023. [8] Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next-generation intelligent assistants leveraging llm techniques. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5792–5793, 2023. [9] Dominique Kelly, Yimin Chen, Sarah E Cornwell, Nicole S Delellis, Alex Mayhew, Sodiq Onaolapo, and Victoria L Rubin. Bing chat: The future of search engines? Proceedings of the Association for Information Science and Technology, 60(1):1007–1009, 2023. [10] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. [11] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790–2799. PMLR, May 2019. [12] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a Unified View of Parameter-Efficient Transfer Learning. In International Conference on Learning Representations, October 2021.

17

[13] Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.353. [14] Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045– 3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.243. [15] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, September 2022. [16] Baohao Liao, Yan Meng, and Christof Monz. Parameter-Efficient Fine-Tuning without Introducing New Latency. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4242– 4260, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-long.233. [17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. [18] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-Decomposed Low-Rank Adaptation. In Proceedings of the 41st International Conference on Machine Learning, pages 32100–32121. PMLR, July 2024. [19] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. HydraLoRA: An Asymmetric LoRA architecture for efficient fine-tuning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, volume 37 of NIPS ’24, pages 9565–9584, Red Hook, NY, USA, June 2025. Curran Associates Inc. [20] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023. [21] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3274–3287, 2023. [22] Zeyu Han, Chao Gao, and Jinyang Liu. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey. Transactions on Machine Learning Research, October 2024. [23] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. Pmlr, 2017. [24] Peter Kairouz and H Brendan McMahan. Advances and open problems in federated learning. Foundations and trends in machine learning, 14(1-2):1–210, 2021. [25] Wenhao Jiang, Yuchuan Luo, Guilin Deng, Silong Chen, Xu Yang, Shihong Wu, Xinwen Gao, Lin Liu, and Shaojing Fu. Federated large language models: Feasibility, robustness, security and future directions. arXiv preprint arXiv:2505.08830, 2025. [26] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 35(4):3347–3366, 2021. [27] Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Efficient federated learning for modern NLP. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking (MobiCom), pages 1–16, 2023. [28] Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, and Ramtin Pedarsani. Communication-efficient and tensorized federated fine-tuning of large language models. arXiv preprint arXiv:2410.13097, 2024.

18

[29] Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. In ICASSP, pages 1–5, 2023. [30] Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. Text-driven prompt generation for vision-language models in federated learning. arXiv preprint arXiv:2310.06123, 2023. [31] Sixing Yu, J. Pablo Muñoz, and Ali Jannesari. Bridging the gap between foundation models and heterogeneous federated learning. arXiv preprint arXiv:2310.00247, 2023. [32] Hongyi Peng, Han Yu, Xiaoxiao Li, and Qiang Yang. Rethinking LoRA for data heterogeneous federated learning: Subspace and state alignment. arXiv preprint arXiv:2602.01746, 2026. [33] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICML), 2024. [34] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024. [35] Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. LoRA-drop: Efficient LoRA parameter pruning based on output evaluation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2025. [36] Songlin Chen, Weicheng Wang, Xiaoliang Chen, Peng Lu, Zaiyan Yang, and Yajun Du. LLaMA-LoRA neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains. Data Intelligence, 2024. [37] Junsu Kim, Jaeyeon Kim, and Ernest K Ryu. LoRA training provably converges to a low-rank global minimum or it fails loudly (but it probably won’t fail). In International Conference on Machine Learning, volume 2025, 2025. [38] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federated GPT: Federated instruction tuning. In IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6915–6919. IEEE, 2024. [39] Muqing Liu, Chongjie Si, and Yuheng Jia. Flexlora: Entropy-guided flexible low-rank adaptation. arXiv preprint arXiv:2601.22905, 2026. [40] Jin Liu, Yinbin Miao, Ning Xi, and Junkang Liu. Rethinking loRA for privacy-preserving federated learning in large models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=BPzSV4uw0x. [41] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 12903–12913, 2024. [42] Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust federated finetuning of foundation models via alternating minimization of lora. arXiv preprint arXiv:2409.02346, 2024. [43] Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. In 13th International Conference on Learning Representations Iclr 2025, 2025. [44] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=NLPzL6HWNl. [45] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513–22533, 2024. [46] Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. Advances in Neural Information Processing Systems, 37:14457–14483, 2024. [47] Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G Brinton. Federated sketching LoRA: On-device collaborative fine-tuning of large language models. In International Conference on Machine Learning (ICML), 2026.

19

[48] Joel A Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher. Practical sketching algorithms for low-rank matrix approximation. SIAM Journal on Matrix Analysis and Applications, 38(4):1454–1485, 2017. [49] Yuji Byun and Jaeho Lee. Towards federated low-rank adaptation of language models with rank heterogeneity. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 356–362, 2025. [50] Sihyeon Ha, Yongjeong Oh, and Yo-Seb Jeon. RB-LoRA: Rank-balanced aggregation for low-rank adaptation with federated fine-tuning. In Findings of the Association for Computational Linguistics: EACL 2026, pages 1737–1746, 2026. [51] Guochen Yan, Luyuan Xie, Qingni Shen, Yuejian Fang, and Zhonghai Wu. Fedsrd: Sparsify-reconstructdecompose for communication-efficient federated large language models fine-tuning. In Proceedings of the ACM Web Conference 2026, pages 5087–5098, 2026. [52] Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang. Adaptive rank allocation for federated parameter-efficient fine-tuning of language models. IEEE Transactions on Computers, 2026. [53] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 4133–4145, 2023. [54] Shuangyi Chen, Yuanxin Guo, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust federated finetuning of LLMs via alternating optimization of LoRA. Advances in Neural Information Processing Systems, 38:120038–120090, 2026. [55] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [56] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353–355, 2018. [57] Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open banking. In Federated Learning: Privacy and Incentive, pages 240–254. Springer, 2020. [58] Pushpita Chatterjee, Debashis Das, and Danda B. Rawat. Federated learning empowered recommendation model for financial consumer services. IEEE Transactions on Consumer Electronics, 2023. [59] Chun-Mei Feng, Yunlu Yan, Shanshan Wang, Yong Xu, Ling Shao, and Huazhu Fu. Specificity-preserving federated learning for mr image reconstruction. IEEE Transactions on Medical Imaging, 42(7):2010–2021, 2022. [60] Meirui Jiang, Holger R. Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu. Fair federated medical image segmentation via client contribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16302–16311, 2023. [61] Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8064–8073, 2023. [62] Yunlu Yan, Hong Wang, Yawen Huang, Nanjun He, Lei Zhu, Yong Xu, Yuexiang Li, and Yefeng Zheng. Cross-modal vertical federated learning for mri reconstruction. IEEE Journal of Biomedical and Health Informatics, 2024. [63] Youyang Qu. Federated learning driven large language models for swarm intelligence: A survey. arXiv preprint arXiv:2406.09831, 2024. [64] Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. FATE-LLM: An industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049, 2023.

20

[65] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. FederatedScope-LLM: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5260–5271, 2024. [66] Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. OpenFedLLM: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6137–6147, 2024. [67] Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. FedLLMBench: Realistic benchmarks for federated learning of large language models. Advances in Neural Information Processing Systems, 37:111106–111130, 2024. [68] Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YunXiang Gong. FedBCGD: Communication-efficient accelerated block coordinate gradient descent for federated learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2955–2963, 2024. [69] Junkang Liu, Yuanyuan Liu, Fanhua Shang, Hongying Liu, Jin Liu, and Wei Feng. Improving generalization in federated learning with highly heterogeneous data via momentum-based stochastic controlled weight averaging. In International Conference on Machine Learning (ICML), 2024. [70] Junkang Liu, Fanhua Shang, Kewen Zhu, Hongying Liu, Yuanyuan Liu, and Jin Liu. FedAdamW: A communication-efficient optimizer with convergence and generalization guarantees for federated large models. arXiv preprint arXiv:2510.27486, 2025. [71] Junkang Liu, Fanhua Shang, Yuxuan Tian, Hongying Liu, and Yuanyuan Liu. Consistency of local and global flatness for federated learning. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 3875–3883, 2025. [72] Junkang Liu, Fanhua Shang, Junchao Zhou, Hongying Liu, Yuanyuan Liu, and Jin Liu. FedMuon: Accelerating federated learning with matrix orthogonalization. arXiv preprint arXiv:2510.27403, 2025. [73] Junkang Liu, Yuxuan Tian, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Junchao Zhou, and Daorui Ding. DP-FedPGN: Finding global flat minima for differentially private federated learning via penalizing gradient norm. arXiv preprint arXiv:2510.27504, 2025. [74] Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu, and Lei Zhu. Federated residual low-rank adaptation of large language models. In The Thirteenth International Conference on Learning Representations, 2025. [75] Ming Wen, Jiaqi Zhu, Yuedong Xu, Yipeng Zhou, and Dingding Han. Differentially private federated low rank adaptation beyond fixed-matrix. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=TecJ926Vgn. [76] Xiuwen Fang and Mang Ye. Towards robust parameter-efficient fine-tuning for federated learning. Advances in Neural Information Processing Systems, 38:141777–141800, 2026. [77] Yicheng Zhang, Zhen Qin, Zhaomin Wu, Jian Hou, and Shuiguang Deng. Personalized federated finetuning for LLMs via data-driven heterogeneous model architectures. In Proceedings of the ACM Web Conference 2026, pages 5099–5110, 2026. [78] Yao Lu, Jiaxing QI, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. FDLoRA: Personalized federated learning of large language model via dual lora tuning. arXiv preprint arXiv:2406.07925, 2026. URL https://arxiv.org/abs/2406.07925. [79] Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of foundation models. arXiv preprint arXiv:2410.09432, 2024. [80] Seanie Lee, Sangwoo Park, Dong Bok Lee, Dominik Wagner, Haebin Seong, Tobias Bocklet, Juho Lee, and Sung Ju Hwang. Fedsvd: Adaptive orthogonalization for private federated learning with lora. Advances in Neural Information Processing Systems, 38:119733–119757, 2026. [81] Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 416–429, 2025. [82] Haoran Zhang, Dongjun Kim, Seohyeon Cha, and Haris Vikalo. FedRot-LoRA: Mitigating rotational misalignment in federated lora. arXiv preprint arXiv:2602.23638, 2026.

21

[83] Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. LoRA-FAIR: Federated LoRA fine-tuning with aggregation and initialization refinement. arXiv preprint arXiv:2411.14961, 2024. [84] Sara Babakniya, Ahmed Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee-Bong Song, MOSTAFA EL-Khamy, and Salman Avestimehr. SLoRA: Federated parameter efficient fine-tuning of language models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023. URL https://openreview.net/forum?id=06quMTmtRV. [85] Peishen Yan, Yang Hua, Hao Wang, Jiaru Zhang, Xiaoyu Wu, Tao Song, and Haibing Guan. FedMomentum: Preserving lora training momentum in federated fine-tuning. arXiv preprint arXiv:2603.08014, 2026. [86] Yajie Zhou, Xiaoyi Pang, and Zhibo Wang. AFLoRA: Adaptive federated fine-tuning of large language models with resource-aware low-rank adaption. arXiv preprint arXiv:2505.24773, 2025. [87] Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang. Preventing rank collapse in federated low-rank adaptation with client heterogeneity. arXiv preprint arXiv:2602.13486, 2026. [88] Zihao Peng, Jiandian Zeng, Boyuan Li, Guo Li, Shengbo Chen, and Tian Wang. Fedhl: Federated learning for heterogeneous low-rank adaptation via unbiased aggregation. arXiv preprint arXiv:2505.18494, 2025. [89] Chuiyang Meng, Ming Tang, and Vincent Wong. Florg: Federated fine-tuning with low-rank gram matrices and procrustes alignment. In International Conference on Learning Representations, volume 2026, pages 82804–82824, 2026. [90] Neranjan Senarath, Rohit Muralitharan, and Sadia Asif. Subspace-constrained federated learning with low-rank adaptation. arXiv preprint arXiv:2606.22724, 2026.

A

Algorithmic representation of SeFoRA-Ho

SeFoRA-Ho is summarized in Algorithm 5.

B

Proof of Theorem 1 and Corollary 1

In the proofs below, we adopt the following notations: Let Ft represent all algorithmic history up to the beginning of round t. We define Et [·] ≜ E[· | Ft ] as the conditional expectation taking into account all sources of randomness occurring during round t. Let Eξi [·] denote the expectation taken with respect to the local mini-batch sampling at round t, conditioned on the model state at the moment of sampling. We define (t)

gB = ∇B Llora (B(t) , A(t) )

(t)

(t)

(t) (t) gB,i = ∇B Llora i (B , A )

gA = ∇A Llora (B(t) , A(t) ),

(t)

(t) (t) gA,i = ∇A Llora i (B , A ), (t,e)

= ∇A Llora i (Bi

(t,e)

= ∇A Llora i (Bi

ĝA,i

gA,i

(t,e)

, Ai

(t,e)

; ξi ),

(t,e)

, Ai

(e)

ĝB,i

(t,e)

),

gB,i

E−1

(t,e)

= ∇B Llora i (Bi

(t,e)

= ∇B Llora i (Bi

(t,e)

; ξi )

(t,e)

, Ai

(e)

(t,e)

)

1 X (t,e) ĝ E e=0 B,i h i X (t) (t) (t) pi Et g B,i − gB DB = (t)

g B,i =

i∈[N ]

i∈[N ]

=

, Ai

E−1

1 X (t,e) ĝ , E e=0 A,i h i X (t) (t) (t) DA = pi Et g A,i − gA , (t)

g A,i =

X

(t,e)

h i (t) (t) pi Et g A,i − gA,i ,

=

i∈[N ]

X

h i (t) (t) pi Et g B,i − gB,i

(Drift Terms)

i∈[N ]

Lemma 3. Under Assumption 2 and Assumption 3, the local drift of the LoRA factors B and A, and the update ∆W at the local step e ∈ {0, · · · , E − 1}, and for any round t ∈ {0, · · · , T − 1} and client i ∈ [N ] in SeFoRA-Ho are bounded as: (t,e)

E Bi

− B(t)

2 F

2 ≤ (η (t) )2 e2 χ2 MA ,

22

Algorithm 5 SeFoRA-Ho Require: The pretrained model W0 . Total rounds T and local steps E. In total, N clients, each PN holding their local dataset Di . The aggregation weights pi = |Di |/ i=1 |Di |. The global rank profile r = (r1 , · · · , rL ) and kl > rl + 1 for every l ∈ [L]. Set rmax ≜ maxl∈[L] rl , and kmax ≜ (0) (0) ∈ Rd×rl as a full zero matrix and Al ∈ Rrl ×s as (0) (0) (0) (0) a random Gaussian matrix. B(0) = (B1 , · · · , BL ) and A(0) = (A1 , · · · , AL ). A function

maxl∈[L] kl . Initialize global matrices Bl

U NSKETCHING(Y, Z, Ψ, Ω) that unsketch the sketch matrices (Algorithm 1). 1: for round t = 0, 1, · · · , T − 1 do 2: Federator initializes Ψ(t+1) ∈ Rkmax ×d and Ω(t+1) ∈ Rs×rmax from a standard normal

distribution, and send the shared seed that generates Ψ(t+1) and Ω(t+1) to all clients. 3: Federator broadcasts B(t) and A(t) to all clients. 4: for all clients i in parallel do 5: for all adapted layers l ∈ [L] do (t+1) (t+1) (t+1) (t+1) 6: Extract Ωl = Ω:,1:rl ∈ Rs×rl and Ψl = Ψ1:kl ,: ∈ Rkl ×d from the shared seed. (t)

(t)

(t+1)

= ∆Wi,l

(t+1)

= Ψl

Obtain Bl and Al . (t) (t) (t) (t) Initialize local LoRA factors Bi,l = Bl and Ai,l = Al . end for (t) (t) (t+1) (t+1) Using Bi,l and Ai,l , perform E local steps to obtain Bi,l and Ai,l for all l ∈ [L]. for all adapter layer l ∈ [L] do Compute the sketch matrices

7: 8: 9: 10: 11: 12:

Yi,l

Zi,l 13: 14: 15: 16: 17:

(t+1)

(t+1)

(t+1)

= Bi,l

(t+1)

= (Ψl

Ωl

(t+1)

(t+1)

∆Wi,l

(t+1)

(Ai,l

(t+1)

Bi,l

(t+1)

Ωl

) ∈ Rd×rl ,

(t+1)

)Ai,l

∈ Rkl ×s

end for (t+1) (t+1) (t+1) (t+1) (t+1) (t+1) Send Yi = (Yi,1 , · · · , Yi,L ) and Zi = (Zi,1 , · · · , Zi,L ) to federator. end for for all adapter layer l ∈ [L] do P P (t+1) (t+1) (t+1) (t+1) Federator aggregates Yl = i∈[N ] pi Yi,l and Zl = i∈[N ] pi Zi,l . (t+1)

(t+1)

18: (Bl , Al 19: end for 20: end for 21: Return (B(T ) , A(T ) )

(t+1)

) ← U NSKETCHING(Yl

(t,e)

E Ai

− A(t)

(t,e)

− ∆W(t)

E ∆Wi

2

(t,0)

(t,e)

(t,e)

∇B Lfull i (Bi

E

(t,e)

, Ai

(e)

F

(t,e)

− η (t) ĝB,i ,

(t,e)

− η (t) ĝA,i ,

= Ai

(t,e)

(t,e)

(t,e)

(e)

(t,e) ⊤

; ξi ) = (Bi

(t,e)

2

) = (B(t) , A(t) ) and updates

(t,e+1)

; ξi ) = ∇∆W Lfull i (∆Wi

(t,e) Bi − B(t)

(t,0)

= Bi

, Ai

)

and

, Ai

(t,e+1)

(t,e)

where we define ĝA,i = ∇A Lfull i (Bi

(t+1)

, Ωl

4 4 ≤ 2(η (t) )2 e2 χ2 (MB + MA ).

Proof. The local fine-tuning starts from (Bi

Ai

(t+1)

, Ψl

2 ≤ (η (t) )2 e2 χ2 MB ,

F 2 F

Bi

(t+1)

, Zl

(e)

(t,e) ⊤

; ξi ) · (Ai

e−1 X (t,j) = E −η (t) ĝB,i j=0

2

F

23

(t,e)

) · ∇∆W Lfull i (∆Wi

(e)

(t,e)

; ξi ) and ĝB,i =

) . Thus, for e ∈ {0, · · · , E −1},



e−1 X  (t,j) ĝB,i = E (η (t) )2 · j=0 (a)

≤ (η (t) )2 e ·

e−1 X

≤ (η (t) )2 e ·

e−1 X

  

F (t,j)

E ĝB,i

j=0 (b)

2

2 F

(t,j)

E ∇∆W Lfull i (∆Wi

(j)

; ξi )

j=0

2 F

(t,j)

· Ai

2 2

(c)

2 ≤ (η (t) )2 e2 χ2 MA

where (a) holds due to the Jensen’s inequality, (b) employs ∥XY ∥2F ≤ ∥X∥22 · ∥Y ∥2F , and (c) utilizes (t,e)

Assumption 2 and Assumption 3. Similarly, E Ai (t,e)

E ∆Wi

− ∆W(t)

2 F

(t,e) (t,e) Ai − B(t) A(t)

= E Bi =E

2

− A(t)

F

2 ≤ (η (t) )2 e2 χ2 MB . Then,

2 F

(t,e) (t,e) (t,e) Bi (Ai − A(t) ) + (Bi − B(t) )A(t) (t,e) Bi

≤ 2E

2 ≤ 2MB E

2 F

(t,e) Ai − A(t)

(t,e) Ai − A(t)

2 F

2

+ 2E F

2 + 2MA E

2 F

(t,e) Bi − B(t) (t,e) Bi − B(t)

2 F

A(t)

2 F

2 F

4 4 ≤ 2(η (t) )2 e2 χ2 (MB + MA )

Lemma 4. For round t ∈ {0, · · · , T − 1}, under Assumption 1, Assumption 2 and Assumption 3, the following bounds hold in SeFoRA-Ho:
(E − 1)(2E − 1) 2 (t) 2 2 2 (t) 4 4 χ (η ) χ MA + 2µ2 (MB ∥DA ∥2F ≤ + MA ) , 3 and
(E − 1)(2E − 1) 2 (t) 2 2 2 (t) 2 4 4 χ (η ) χ MB + 2µ2 MA (MB + MA ) . ∥DB ∥2F ≤ 3 Proof. We prove the bound on A; the B bound follows by the symmetric argument. The Frobenius (t) norm of the drifts ∥DA ∥2F is bounded as: 2 (t)

DA

2

= F

X

≤

X

(t)

pi Et g A,i − gA

F

E−1 h i X 1 1 X (t) (t,e) pi Et ĝA,i − pi g E e=0 E e=0 A,i i∈[N ]

pi

i∈[N ] (a)

(t)

i

E−1 X

i∈[N ]

=

h

i∈[N ]

X

=

X

pi

i∈[N ]

E−1 i 1 X h (t,e) (t) Et gA,i − gA,i E e=0 E−1 X

X i∈[N ]

pi

F

2

F

2

1 (t,e) (t) Et gA,i − gA,i E e=0 F E−1

=

2

2 1 X (t,e) (t,e) (t) Et (Bi )⊤ · ∇∆W Lfull ) − (B(t) )⊤ ∇∆W Lfull i (∆Wi i (∆W ) E e=0 F

24

X

=

pi

i∈[N ]

 E−1    2 1 X (t,e) (t,e) (t,e) (t) ⊤ full full (t) (∆W ) + (B ) · ∇ L (∆W ) − ∇ L (∆W ) Et (Bi )⊤ − (B(t) )⊤ · ∇∆W Lfull ∆W ∆W i i i i i E e=0 F E−1

(b)

X

≤2

pi

i∈[N ]

2 2 1 X (t,e) (t,e) ∇∆W Lfull ) − B(t) Et Bi i (∆Wi E e=0 F F E−1

+2

X

pi

i∈[N ] (c)

X

≤2

2 2 1 X (t,e) (t) ∇∆W Lfull ) − ∇∆W Lfull Et B(t) i (∆Wi i (∆W ) E e=0 F 2

pi χ2

i∈[N ] (d)

X

≤ 2

E−1 E−1 X 2 2 1 X 1 X (t,e) (t,e) pi µ2 − B(t) + 2 · − ∆W(t) Et Bi Et ∆Wi E e=0 E e=0 F F i∈[N ]

pi χ2

i∈[N ]

E−1 X

E−1 X 1 1 X 2 4 4 (η (t) )2 e2 χ2 MA +2· pi µ2 2(η (t) )2 e2 χ2 (MB + MA ) E e=0 E e=0 i∈[N ]

X
1 E−1 2 4 4 = 2χ2 (η (t) )2 χ2 MA + 2µ2 (MB + MA ) · e2 E e=0 =

(E − 1)(2E − 1) 2 (t) 2 2 2 4 4 χ (η ) χ MA + 2µ2 (MB + MA ) 3

where ∥XY ∥2F h

(a)

holds

by

≤

∥X∥22

· ∥Y ∥2F

applying ≤

i 2 (t,e) ∇∆W Lfull ; ξi ) i (∆Wi

Jensen’s

∥X∥2F

inequality

· ∥Y ∥2F ,

three

times,

(c) holds since (t,e)

(b)

holds

(t,e) ∇∆W Lfull ) i (∆Wi

2

since 2

= F

≤ χ2 (Assumption 2), AsPE−1 sumption 1 and Assumption 3, and (d) holds due to Lemma 3 and e=1 e2 = (E−1)E(2E−1) . 6 Similarly, Eξi

F

(t)

∥DB ∥2F ≤ 2

X

pi χ2

i∈[N ]

≤ Eξi ∇∆W Lfull i (∆Wi

; ξi )

F

E−1 E−1 X 2 2 1 X 1 X (t,e) (t,e) 2 Et Ai − A(t) + 2MA Et ∆Wi − ∆W(t) · pi µ2 E e=0 E e=0 F F i∈[N ]

(E − 1)(2E − 1) 2 (t) 2 2 2 2 4 4 χ (η ) χ MB + 2µ2 MA (MB + MA ) . ≤ 3 Lemma 5. For each client i ∈ [N ] and round t ∈ {0, · · · , T − 1}, under Assumption 2 and (t)

(t)

Assumption 3, the following bounds hold in SeFoRA-Ho: Et g B,i g A,i

2 F

2 2 4 ≤ MA MB κ ;

Proof.  2 E−1 X (t,e) E−1 X (t,e) 1  = Et  2 ĝ · ĝA,i E e=0 B,i F e=0 F   X (a) 1 2 (t,e ) (t,e ) ≤ 4 E2 · Et ĝB,i 1 · ĝA,i 2 E F e1 ,e2   2 1 X (t,e1 ) (e1 ) (t,e1 ) ⊤ (t,e2 ) ⊤ (t,e2 ) (e2 ) full = 2· Et ∇∆W Lfull (∆W ; ξ ) · (A ) · (B ) · ∇ L (∆W ; ξ ) ∆W i i i i i i i i E e ,e F 1 2 " L # X 2 1 X (t,e1 ) (e1 ) (t,e1 ) ⊤ (t,e2 ) ⊤ (t,e2 ) (e2 ) full full Et ∇∆W Li (∆Wi,l ; ξi )(Ai,l ) · (Bi,l ) ∇∆W Li (∆Wi,l ; ξi ) = 2 E e ,e F l=1 1 2 " L # X (b) 1 X 2 2 2 2 (t,e1 ) (e1 ) (t,e1 ) (t,e2 ) (t,e2 ) (e2 ) full full · Bi,l · ∇∆W Li (∆Wi,l ; ξi ) ≤ 2 Et ∇∆W Li (∆Wi,l ; ξi ) · Ai,l E e ,e F 2 2 F 

(t) (t) Et ḡB,i ·ḡA,i

2

1

2

l=1

25

(c) M 2 M 2 ≤ A2 B E (d) M 2 M 2 ≤ A2 B E (e) M 2 M 2 ≤ A2 B E (f ) M 2 M 2 A B ≤ E2

X

" L X

Et

e1 ,e2

X

(t,e1 ) (e1 ) ∇∆W Lfull ; ξi ) i (∆Wi,l

l=1



(t,e1 )

Et

∇∆W Lfull i (∆Wi

s



(e )

; ξi 1 )

e1 ,e2

X

(t,e1 )

∇∆W Lfull i (∆Wi

Et

2

e1 ,e2

X√

(t,e2 )

4

(e )

· F

(t,e2 ) (e2 ) ∇∆W Lfull ; ξi ) i (∆Wi,l

∇∆W Lfull i (∆Wi

F

; ξi 1 )

2



F

 Et

(e )

; ξi 2 ) (t,e2 )

∇∆W Lfull i (∆Wi

2

2

#

F



F

(e )

; ξi 2 )

4



F

κ4 · κ4

e1 ,e2

2 2 4 ≤ MA MB κ , 2 where in (a) holds due to Jensen’s inequality, in (b) we employ ∥2F ≤ P ∥X∥22 · ∥YP∥2F ≤ ∥X∥ P 2 ∥XY P P F 2· 2 2 2 2 2 ∥Y ∥F , (c) holds due to Assumption 3, (d) holds because xl yl ≤ xl yl ≤ ( xl ) ( yl ) for xl ≥ 0, (e) holds due to Cauchy-Schwarz inequality, and (f ) uses Assumption 2.

Lemma 6. For each client i ∈ [N ] and round t ∈ {0, · · · , T −1} in SeFoRA-Ho, under Assumption 2 2

(t)

and Assumption 3, Et g A,i

F

2 2 ≤ MB χ

(t)

Et g B,i

and

2

F

2 2 ≤ MA χ .

(t)

(t)

2 2 χ in the rank-homogeneous case. The bound on g B,i is symmetProof. We prove Et ∥g A,i ∥2F ≤ MB ric.

Et

(t) ḡA,i

E−1 X (t,e) 1 = 2 Et ĝA,i E F e=0

2

2

(a)

F

E−1 X

≤

2 1 (t,e) Et ĝA,i E 2 E F e=0

=

2 1 X (t,e) (t,e) (e) Et (Bi )⊤ · ∇∆W Lfull ; ξi ) i (∆Wi E e=0 F

E−1

(b)

≤

E−1

2 1 X (t,e) 2 (t,e) (e) ∥Bi ∥2 Et ∇∆W Lfull ; ξi ) i (∆Wi E e=0 F

2 2 ≤ MB χ , (t)

where (a) is Jensen’s inequality and (b) uses ∥XY ∥2F ≤ ∥X∥22 ∥Y ∥2F . The bound on g B,i is identical with the roles of A and B swapped. B.1

Proof of Theorem 1

Proof. Since the full objective is µ-smooth with respect to ∆W, we have h i (t+1) Et Lfull (∆WSeFoRA-Ho ) (1) h i hD Ei µ h i (t+1) (t+1) ≤Et Lfull (∆W(t) ) + Et ∇∆W Lfull (∆W(t) ), ∆WSeFoRA-Ho − ∆W(t) + Et ∥∆WSeFoRA-Ho − ∆W(t) ∥2F , | {z } |2 {z } Term 1

Term 2

(t)

(t)

Suppose the global LoRA factors at round t are A and B , the global model obtained by SeFoRA-Ho P (t+1) (t+1) at round t + 1 is a low-rank approximation of the true aggregation result i∈[N ] pi Bi Ai . The (t,0)

local fine-tuning starts from (Bi

(t,0)

, Ai

) = (B(t) , A(t) ) and updates

(t,e+1)

= Bi

(t,e+1)

= Ai

Bi Ai

(t,e)

− η (t) ĝB,i ,

(t,e)

− η (t) ĝA,i .

26

(t,e)

(t,e)

The global model at round t + 1 can be expressed as (t+1)

(t+1)

(t+1)

∆WSeFoRA-Ho = BSeFoRA-Ho ASeFoRA-Ho X (t+1) (t+1) (t+1) = pi Bi Ai + Esketch i∈[N ]

=

X

(t,E) (t,E) (t+1) Ai + Esketch

pi Bi

i∈[N ]

=

X

   (t) (t) (t+1) pi B(t) − η (t) Eg B,i A(t) − η (t) Eg A,i + Esketch

i∈[N ]

= B(t) A(t) − η (t) E

X

(t)

pi B(t) g A,i − η (t) E

X

(t)

pi g B,i A(t) + (η (t) )2 E 2

i∈[N ]

i∈[N ]

X

(t) (t)

(t+1)

pi g B,i g A,i + Esketch ,

i∈[N ]

h i P (t+1) (t+1) (t+1) ϱ r 2 where Et ∥Esketch ∥2F ≤ (1 + k−r−1 ) · minϱ<r−1 (1 + r−ϱ−1 ) · Et τϱ+1 ( i∈[N ] pi Bi Ai ) (Lemma 2). Thus, the model update X X X (t+1) (t) (t) (t) (t) (t+1) ∆WSeFoRA-Ho − ∆W(t) = −η (t) E pi g B,i A(t) + (η (t) )2 E 2 pi g B,i g A,i + Esketch . pi B(t) g A,i − η (t) E i∈[N ]

i∈[N ]

i∈[N ]

(2) For Term 1: From Eq. (2), we can write h i (t+1) Et ∆WSeFoRA-Ho − ∆W(t)       h i X X X (t) (t) (t) (t) (t+1) pi g B,i A(t)  + Et (η (t) )2 E 2 pi g B,i g A,i  + Et Esketch = − Et η (t) E pi B(t) g A,i  − Et η (t) E i∈[N ]

i∈[N ]

= − η (t) E · B(t) ·

X

i∈[N ]

  h i h i h i X X (t) (t) (t) (t) (t+1) pi Et g A,i − η (t) E · pi Et g B,i · A(t) + (η (t) )2 E 2 Et  pi g B,i g A,i  + Et Esketch

i∈[N ]

i∈[N ]

i∈[N ]



 (t)

(t)

(t)

(t)

= − η (t) E · B(t) · (DA + gA ) − η (t) E · (DB + gB ) · A(t) + (η (t) )2 E 2 Et 

X

h i (t) (t) (t+1) pi g B,i g A,i  + Et Esketch

i∈[N ]



 (t)

(t)

(t)

(t)

= − η (t) EB(t) gA − η (t) EgB A(t) − η (t) EB(t) DA − η (t) EDB A(t) + (η (t) )2 E 2 Et 

X

h i (t) (t) (t+1) pi g B,i g A,i  + Et Esketch .

i∈[N ]

Thus, hD Ei (t+1) Et ∇∆W Lfull (∆W(t) ), ∆WSeFoRA-Ho − ∆W(t) D h iE (t+1) = ∇∆W Lfull (∆W(t) ), Et ∆WSeFoRA-Ho − ∆W(t) E D E D (t) (t) = − η (t) E · ∇∆W Lfull (∆W(t) ), B(t) gA − η (t) E · ∇∆W Lfull (∆W(t) ), gB A(t) D E D E (t) (t) − η (t) E · ∇∆W Lfull (∆W(t) ), B(t) DA − η (t) E · ∇∆W Lfull (∆W(t) ), DB A(t)  + * D h iE X (t) (t) (t+1) + (η (t) )2 E 2 ∇∆W Lfull (∆W(t) ), Et  pi ḡB,i ḡA,i  + ∇∆W Lfull (∆W(t) ), Et Esketch i∈[N ]

  D E D E (a) (t) (t) (t) (t) (t) (t) = − η (t) E · ∥gA ∥2F + ∥gB ∥2F + η (t) E · gA , −DA + η (t) E · gB , −DB  + * D h iE X (t+1) (t) (t) + (η (t) )2 E 2 ∇∆W Lfull (∆W(t) ), Et  pi ḡB,i ḡA,i  + ∇∆W Lfull (∆W(t) ), Et Esketch i∈[N ]

27

(b)

  1 (t) 1 (t) 1 (t) 1 (t) (t) (t) ≤ − η (t) E · ∥gA ∥2F + ∥gB ∥2F + η (t) E · ( ∥gA ∥2F + ∥DA ∥2F ) + η (t) E · ( ∥gB ∥2F + ∥DB ∥2F ) 2 2 2 2   h i X (t+1) (t) (t) pi ḡB,i ḡA,i  + ∇∆W Lfull (∆W(t) ) · Et E + (η (t) )2 E 2 · ∇∆W Lfull (∆W(t) ) · Et  F

sketch

F

i∈[N ]

F

F

  η (t) E 1 η (t) E (t) (t) (t) (t) ≤ − η (t) E · ∥gA ∥2F + ∥gB ∥2F + ∥DA ∥2F + ∥DB ∥2F 2 2 2 v   u 2 u u  X  (t) (t) pi ḡB,i ḡA,i  + ∇∆W Lfull (∆W(t) ) + (η (t) )2 E 2 · ∇∆W Lfull (∆W(t) ) · u tEt 

(c)

F

i∈[N ]

s ·



Et

F

(t+1) Esketch

2

 ,

F

F

where (a) holds since the definition of the tuple inner product and the cyclic property of the trace, i.e.,   X   L L D E X (t) ⊤ (t) ⊤ (t) (t) ⊤ (t) (t) full (t) (t) (t) full ∇∆W L (∆W ), B gA = tr (gAl ) (Bl ) ∇∆W L (∆Wl ) = tr (gAl ) gAl = ∥gA ∥2F , i=1

i=1

  X   L L D E X (t) (t) ⊤ (t) (t) ⊤ (t) ⊤ (t) (t) tr (gBl ) gBl = ∥gB ∥2F , ∇∆W Lfull (∆W(t) ), gB A(t) = tr (gBl ) ∇∆W Lfull (∆Wl )(Al ) = i=1

i=1

D E D E D E (t) (t) (t) (t) = gA , DA , ∇∆W Lfull (∆W(t) ), DB A(t) = and similarly, ∇∆W Lfull (∆W(t) ), B(t) DA D E (t) (t) gB , DB , (b) holds since Young’s inequality and ⟨X, Y ⟩ ≤ |⟨X, Y ⟩| ≤ ∥X∥F ∥Y ∥F , and (c) (t)

(t)

holds since Jensen’s inequality. Since ∥DA ∥F and ∥DB ∥F are bounded by Lemma 4, and according to Lemma 5, we know   2   X X 2   (t) (t) (t) (t) 2 2 4 Et  pi ḡB,i ḡA,i  ≤ pi Et ḡB,i · ḡA,i ≤ MA MB κ , i∈[N ]

F

i∈[N ]

F

we only need to derive the bounds for ∇∆W Lfull (∆W(t) ) F . Due to Jensen’s inequality and the inequality that ∥E[X]∥2 ≤ E[∥X∥2 ], we have 2 2

∇∆W Lfull (∆W(t) )

=

X

≤

X

F

(t) pi ∇∆W Lfull i (∆W )

i∈[N ]

pi

(t) ∇∆W Lfull i (∆W )

i∈[N ]

=

X

F 2 F

i 2 (t) pi Et ∇∆W Lfull (∆W ; ξ ) i i h

F

i∈[N ]

≤

X

 pi Et

(t) ∇∆W Lfull i (∆W ; ξi )

i∈[N ]

2



F

≤ χ2 .

(3)

Hence, by combining the previous bounds and utilizing Lemma 5, Term 1 is bounded as hD Ei (t+1) Et ∇∆W Lfull (∆W(t) ), ∆WSeFoRA-Ho − ∆W(t)   η (t) E η (t) E 1 (t) (t) (t) (t) ∥DA ∥2F + ∥DB ∥2F ≤ − η (t) E · ∥gA ∥2F + ∥gB ∥2F + 2 2 2 v   u 2 u u  X  (t) (t) + (η (t) )2 E 2 · ∇∆W Lfull (∆W(t) ) · u pi ḡB,i ḡA,i  + ∇∆W Lfull (∆W(t) ) tEt  F

i∈[N ]

28

F

s · F

 Et

(t+1) Esketch

2 F



 
1 (E − 1)E(2E − 1) 2 (t) (t) 2 2 2 4 4 ≤ − η (t) E · ∥gA ∥2F + ∥gB ∥2F + (η (t) )3 χ · χ2 (MA + MB ) + 2µ2 (1 + MA )(MB + MA ) 2 6 s   2 (t+1) . + (η (t) )2 E 2 χ · MA MB κ2 + χ · Et Esketch F

For Term 2: Using Eq. (2), we have i h (t+1) Et ∥∆WSeFoRA-Ho − ∆W(t) ∥2F   2 X X X   (t) (t) (t) (t) (t+1) pi B(t) g A,i − η (t) E =Et  −η (t) E pi g B,i A(t) + (η (t) )2 E 2 pi g B,i g A,i + Esketch  i∈[N ]

i∈[N ]



2

(a)

(b)

(t)

pi Et ḡA,i

i∈[N ]

+ 4(η (t) )4 E 4

X

 pi Et

i∈[N ]

(t)

F 2

i∈[N ]

F

X

≤4(η (t) )2 E 2 ∥B(t) ∥22 ·



(t)

ḡB,i · ḡA,i

2

+ 4(η (t) )2 E 2

F 2 F

X

h

(t+1)

F (t)

pi Et ḡB,i

i∈[N ]



+ 4Et ∥Esketch ∥2F



2 F



2

  X (t) (t) (t) 4 4 pi ḡB,i ḡA,i  + 4(η ) E Et 

  X (t) (t) 2 2 pi ḡB,i A(t)  + 4(η ) E Et 

 X (t) pi B(t) ḡA,i ≤ 4(η (t) )2 E 2 Et  i∈[N ]

i∈[N ]



i∈[N ]

   + 4Et

F

· ∥A(t) ∥22

i

(c)

h i (t+1) 2 4 2 2 4 ≤ 4(η (t) )2 E 2 χ2 (MB + MA ) + 4(η (t) )4 E 4 MA MB κ + 4Et ∥Esketch ∥2F ,

where (a) holds by grouping the first two terms and using ∥W + X + Y + Z∥2F ≤ 4(∥W ∥2F + ∥X∥2F + ∥Y ∥2F + ∥Z∥2F ), (b) holds due to ∥XY ∥2F ≤ ∥X∥22 · ∥Y ∥2F and Jensen’s inequality, and (c) holds from Assumption 2, Assumption 3, Lemma 5, and Lemma 6. By combining the bounds on Term 1 and Term 2, we have h i h i (t+1) Et Lfull (∆WSeFoRA-Ho ) − Et Lfull (∆W(t) ) hD Ei µ h i (t+1) (t+1) ≤ Et ∇∆W Lfull (∆W(t) ), ∆WSeFoRA-Ho − ∆W(t) + Et ∥∆WSeFoRA-Ho − ∆W(t) ∥2F 2  
1 (t) (t) 2 (t) 2 (t) 2 2 2 4 ≤ − η E · ∥gA ∥F + ∥gB ∥F + (η ) E · χMA MB κ2 + 2µχ2 (MB + MA ) 2
(E − 1)E(2E − 1) 2 2 2 2 2 4 4 + (η (t) )3 · χ χ (MA + MB ) + 2µ2 (1 + MA )(MB + MA ) 6 s   h i 2 (t+1) (t+1) (t) 4 4 2 2 4 + (η ) E · 2µMA MB κ + χ · Et Esketch + 2µEt ∥Esketch ∥2F . F

Therefore, ∇A Llora (B(t) , A(t) )

2

+ ∇B Llora (B(t) , A(t) )

2

≤

  h i  (t+1) 2 Et Lfull (∆W(t) ) − Et Lfull (∆WSeFoRA-Ho )

η (t) E
(E − 1)(2E − 1) 2 2 2 4 2 2 2 4 4 +2η (t) E · χMA MB κ2 + 2µχ2 (MB + MA ) + 2(η (t) )2 ·χ χ (MA +MB )+2µ2 (1+MA )(MB + MA 6 s  !  h i 2 2 (t+1) (t+1) 2 2 4 + 4(η (t) )3 E 3 · µMA MB κ + (t) χ · Et Esketch + 2µEt ∥Esketch ∥2F . F η E F

F

By taking constant learning rate η (t) = η, taking total expectation E[·] on both sides, and summing this inequality from t = 0 to T − 1 and computing the average, we have h i  E Lfull (∆W(0) ) − E Lfull (∆W(T )   T −1   ) X 2 2 SeFoRA-Ho 1 ≤ E ∇A Llora (B(t) , A(t) ) + E ∇B Llora (B(t) , A(t) ) 1 T t=0 F F T ηE 2 29

2
2 4 2 2 2 4 4 + 2ηE · χMA MB κ2 + 2µχ2 (MB + MA ) + η 2 E 2 ·χ2 χ2 (MA +MB )+2µ2 (1 + MA )(MB +MA ) 3 " s  #  T −1 h i 2 1 X 2 (t+1) (t+1) 2 3 3 2 2 4 + 2µEt ∥Esketch ∥F . + 4η E · µMA MB κ + E χ · Et Esketch T t=0 ηE F 2 4 4 Next, we rearrange the terms in the above bound. Define C1 = χ2 MA + 2µ2 (MB + MA ), C2 = 2 2 2 2 4 4 2 2 2 4 χ MB + 2µ M (M + M ), C = M M κ , C = µχ (M + M ), the sketching floor 3 A B 4 A B A sA B  h i h i 2 PT −1 (t+1) (t+1) (t+1) (t+1) + 2µEt ∥Esketch ∥2F , S1 = T1 t=0 E S1 S1 = χ · Et Esketch , and ∆0 = F   E Lfull (∆W(0) ) − Lfull,⋆ . We have h i  E Lfull (∆W(0) ) − E Lfull (∆W(T )   T −1   ) X 2 2 SeFoRA-Ho 1 ≤ + E ∇B Llora (B(t) , A(t) ) E ∇A Llora (B(t) , A(t) ) 1 T t=0 F F 2 T ηE T −1 2 2 1 X h (t+1) i + ηE · 2 (χC3 + 2C4 ) + η 2 E 2 · χ2 (C1 + C2 ) + 4η 3 E 3 · µC32 + · E S1 3 ηE T t=0

2 2 2∆0 + S1 + ηE · 2 (χC3 + 2C4 ) + η 2 E 2 · χ2 (C1 + C2 ) + 4η 3 E 3 · µC32 ηET ηE 3 This concludes the proof of Theorem 1. =

B.2

Proof of Corollary 1

P (t+1) (t+1) Proof. According to Lemma 2, the error between the aggregated client update i∈[N ] pi Bi Ai and its low-rank approximation can be bounded as:   ′ X r ϱ (t+1) (t+1)  (t+1) ) · min (1 + ) · E τϱ2′ +1 ( pi Bi Ai ) Et ∥Esketch ∥2F ≤ (1 + k − r − 1 ϱ′ <r−1 r − ϱ′ − 1 i∈[N ]   X r ϱ (t+1) (t+1)  2 ≤ (1 + ) · (1 + ) · E τϱ+1 ( pi Bi Ai ) , k−r−1 r−ϱ−1 i∈[N ]

where the second inequality holds by evaluating at the fixed ϱ from Assumption 4. ϱ r Define ϵ2 ≜ (1 + k−r−1 ) · (1 + r−ϱ−1 ) · τ 2 . Averaging over the T communication rounds and invoking Assumption 4, we obtain T −1 1 X h (t+1) 2 i r ϱ E ∥Esketch ∥F ≤ (1 + ) · (1 + ) · τ 2 = ϵ2 . T t=0 k−r−1 r−ϱ−1

Thus, the sketching floor is s   T −1 T −1 T −1 2 1 X 1 X h (t+1) 2 i 1 X h (t+1) i (t+1) E S1 =χ· E Esketch + 2µ · E ∥Esketch ∥F T t=0 T t=0 T t=0 F v u T −1   T −1 u1 X 2 1 X h (t+1) 2 i (t+1) ≤χ·t E Esketch E ∥Esketch ∥F + 2µ · T t=0 T t=0 F = χϵ + 2µϵ2 , where the inequality holds due to Jensen’s inequality. By substituting this bound into the result of Theorem 1, we obtain    T −1   2 2 1 X lora (t) (t) lora (t) (t) E ∇A L (B , A ) + E ∇B L (B , A ) T t=0 F F 30

≤

2 2 1 2∆0 · + (χϵ + 2µϵ2 ) + ηE · 2 (χC3 + 2C4 ) + η 2 E 2 · χ2 (C1 + C2 ) + 4η 3 E 3 · µC32 ηE T ηE 3

1 0 Define X = 2∆ 2µϵ2 ) and Y = 2 (χC3 + 2C4 ), the function f (ηE) = ηE · X + ηE · Y T + 2(χϵ + q √ X ⋆ ⋆ is minimized at (ηE) = Y , and f ((ηE) ) = 2 XY . Thus, by choosing s r ∆0 2 1 X 1 T + χϵ + 2µϵ η= = , E Y E χC3 + 2C4

we have    T −1   2 2 1 X lora (t) (t) lora (t) (t) + E ∇B L (B , A ) E ∇A L (B , A ) T t=0 F F 3  2 √ X 2 2 X ≤ 2 XY + · µC32 · χ (C1 + C2 ) + 4 Y 3 Y s  ∆0 +χϵ + 2µϵ2 2χ2 (C1 +C2 ) ∆0 2 =4 +χϵ+2µϵ (χC3 +2C4 ) + T +4 · T χC3 + 2C4 3

31

∆0 2 T +χϵ+2µϵ

χC3 + 2C4

! 32 · µC32 .

Related documents

• FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection#1156891
• CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning#1132187
• CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning#1122260
• FedCSGA: Evolutionary client selection with joint statistical and system heterogeneity in federated learning#268418
• LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights#609874