Conceptio › Archive › arXiv CS

arXiv CSopen access

Low-Rank Adaptation Redux for Large Models

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

Low-Rank Adaptation Redux for Large Models

arXiv:2604.21905v1 [cs.LG] 23 Apr 2026

Bingcong Li, Member, IEEE, Yilang Zhang, Georgios B. Giannakis, Fellow, IEEE

altered distributions. However, even on moderate-size datasets, full fine-tuning of LLMs with billions of parameters is still formidable, given the considerable resources required. In practice, fine-tuning a large model by updating all model parameters entails exaFLOPs (×1018 ) of computation and hundreds of GPU hours, incurring non-negligible financial and energy costs. This makes full fine-tuning infeasible for individual users or small institutes. Memory is an equally critical bottleneck. A widely used rule of thumb is about 16 GB of GPU RAM per billion parameters in half precision [123]. Using this estimate, fine-tuning an 8B-parameter model demands ∼128 GB of memory, far exceeding the capacity of standard commodity hardware. Scaling to models with over 100B parameters pushes the requirement to the terabyte scale, necessitating distributed training across large GPU clusters. Parameter-efficient fine-tuning (PEFT) [68], [109], [71], [128] offers a compute- and memory-frugal alternative to full fine-tuning. Instead of updating all pre-trained weights, PEFT freezes the backbone and optimizes only a small set of additional parameters. Among these methods, low-rank adaptation (LoRA) [71] has gained widespread prominence due to its simplicity and strong empirical performance. Concretely, LoRA augments the parameters over layer with an additive low-rank update XY⊤ , where X and Y are small trainable (so-termed “adapter”) matrices. This can reduce the number of trainable parameters by over 100× relative to full fine-tuning, Index Terms—low rank, LLMs, fine-tuning, optimization substantially lowering both compute and memory costs. Adapter matrices are lightweight, often accounting for ∼ 1% − 2% of I. I NTRODUCTION the entire model, and can be stored, shared, and swapped across Recent advances of large language models (LLMs) in tasks and users with minimal overhead. This modularity has artificial intelligence (AI) have unlocked unprecedented ca- further accelerated adoption in open-source communities; large pabilities across a broad spectrum of fields. State-of-the-art models such as LLaMA-7B can now be fine-tuned by individual models such as GPT [149], [150], [15], LLaMA [188], [189], users on a single consumer GPU of 24 GB memory [243]. [39], Gemma [51], [183], [182], and QWen [8], [218], [184] Consequently, LoRA has become the standard approach for involve billions of parameters. There are even larger ones fine-tuning large models [128], spawning dozens of variants such as Qwen3-Max [184] that have surpassed the trillion- that further impel efficiency or alleviate limitations. parameter milestone, enabling complicated tasks including code LoRA and its variants have been adopted across a wide range generation, reasoning, and autonomous agents. Pre-training to of foundation-model applications, from LLM-based coding learn model parameters at this scale typically requires sustained and mathematical reasoning [71] to diffusion-based image large-cluster computation and vast Internet-scale corpora, which generation [54], as well as multi-modal systems that jointly remain accessible only to a small number of organizations. handle language, images, video, and audio [2]. This ubiquity A more accessible way to leverage LLMs is to start from has in turn catalyzed LoRA’s expansion beyond standard an open-weight pre-trained model (e.g., via Hugging Face) downstream fine-tuning. In particular, recent advances have and fine-tune it on user- or domain-specific data [128]. This designed efficient LoRA variants across the full lifecycle of approach preserves the broad capabilities of LLMs learned foundation models, that includes pre-training, post-training, during pre-training, while adapting the model to user data with and deployment from the providers to serve the users [113], Research in this paper was supported in part by the NSF grants 2212318, [163], [33], [173]. 2220292, and 2312547; B. Li is now with the Dept. of CS, ETH Zürich, However, the rapid proliferation of the literature has made Switzerland; G. B. Giannakis and Y. Zhang are with the Dept. of ECE, U. of it difficult to discern what first-principles guidelines should Minnesota, USA; Y. Zhang is now with Morgan Stanley, USA. Emails: binginform the design and application of fast, efficient, and practical [email protected], [email protected], [email protected] April 24, 2026 LoRA algorithms across diverse real-world settings, especially

Abstract—Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networks with minimal computational and memory overhead. Despite its empirical success and rapid proliferation of variants, it remains elusive which architectural choices, optimization techniques, and deployment constraints should guide practical method selection. This overview revisits LoRA through the lens of signal processing (SP), bridging modern adapter designs with classical low-rank modeling tools and inverse problems, as well as highlighting how SP principles can inform principled advances of fine-tuning approaches. Rather than providing a comprehensive enumeration and empirical comparisons of LoRA variants, emphasis is placed on the technical mechanisms underpinning these approaches to justify their effectiveness. These advances are categorized into three complementary axes: architectural design, efficient optimization, and pertinent applications. The first axis builds on singular value decomposition (SVD)-based factorization, rank-augmentation constructions, and cross-layer tensorization, while the second axis deals with initialization, alternating solvers, gauge-invariant optimization, and parameterization-aware methods. Beyond fine-tuning, emerging applications of LoRA are accounted across the entire lifecycle of large models, ranging from pre- and post-training to serving/deployment. Finally, open research directions are outlined at the confluence of SP and deep learning to catalyze a bidirectional frontier: classical SP tools provide a principled vocabulary for designing principled PEFT methods, while the unique challenges facing modern deep learning, especially the overwhelming scale and prohibitive overhead, also offer new research lines benefiting the SP community in return.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

for researchers new to this field. One natural bridge is to revisit these questions through the lens where low-rank methodologies have been systematically studied and applied for decades. Classical tools in signal processing (SP) such as principal component analysis (PCA) [145], [133], subspace tracking [168], and matrix/tensor decompositions [169] demonstrate the power of leveraging low-dimensional structure hidden within highdimensional data. Not only offering computational savings and statistical robustness, these tools also underpin applications ranging from communications, imaging sciences to graph learning; see e.g., [131], [169], [175]. Cross-fertilizing the SP community’s expertise in low-rank modeling to develop more principled LoRA methods offers an impactful direction. This survey recaps recent LoRA advances through an SP lens. Similar to SP, where progress is often driven by coupling structured parameterizations with algorithmic insights, we group LoRA variants in two broad categories depending on: architecture (model) designs, and tailored optimization algorithms. We will start with a brief recapitulation of lowrank modeling and relevant SP tools, which provide a principled vocabulary for interpreting LoRA variants and suggesting novel designs. Next, we review representative LoRA methods, emphasizing methodological insights over numerical results, since results are often difficult to compare fairly across heterogeneous models, datasets, and evaluation protocols. Our goal is to highlight underlying ideas transferred across settings without being overly anchored to specific benchmarks. Finally, we will outline emerging applications of LoRA beyond standard fine-tuning, that span serving, pre- and post-training. Our survey will be concluded with open directions in cross-fertilizing SP and LoRA research. Beyond the unique SP angle, our survey also differs from existing reviews in several aspects. While [58], [197] overview PEFT, we focus on LoRA to enable detailed treatment of its design space. Compared to [60], which bridges AI and SP through LLM-facilitated education, we connect the two communities from a low-rank perspective. Unlike [129] which catalogs LoRA variants, we emphasize efficient mechanisms, especially from optimization and geometry perspectives, so that our discussion leans more toward “why” rather than “what.” The remainder of this survey is organized as follows. Section II recaps low-rank approaches in SP. Section III provides an overview for LoRA, and the link between standard LoRA and low-rank matrix sensing. Subsequently, Section IV discusses efficient architectural model parameterizations beyond the Burer-Monteiro factorization used in vanilla LoRA, followed by efficient optimization approaches in Section V. The joint designs of architecture and optimization schemes will lead to high-performance fine-tuning approaches. Section VI deals with applications of LoRA, ranging from fine-tuning, pre-training, and serving, to broader use cases in diffusion and multi-modal models. The survey is concluded with future directions that not only bring classical SP tools to LoRA, but also bounce LoRA ideas back to the SP community. Notation. Bold lowercase, uppercase, and calligraphic letters (e.g., a, A, A) denote column vectors, matrices, and tensors, respectively; (·)⊤ and ∥·∥F refer to transposition and Frobenius norm of a matrix; ∥ · ∥ is the ℓ2 (spectrum) norm of a vector

(matrix); rank() denotes the rank of a matrix, while ⊙ and ⊗ stand for Hadamard and Kronecker products. II. L OW- RANK IN SP Low-rank learning has been a cornerstone of the SP research for decades, long predating the rise of modern deep learning [145], [98]. Given the ubiquity of low-rank structure in real-world problems, SP has documented success leveraging the low-rank feature across a broad spectrum of applications, underpinned by interpretable modeling assumptions, mature statistical theory, and efficient implementations with clear computational benefits. This success has also produced a rich algorithmic toolbox. For convex objective functions, nuclear-norm minimization and its variants provide principled formulations with rigorous recovery guarantees [153]. For nonconvex objectives, factor models based on Burer–Monteiro (BM) parameterizations [16], combined with alternating schemes [78], (projected) gradient descent [25], [59], [124], and Riemannian optimization on low-rank manifolds [137], [89], enable scalability in largescale settings. Due to space limitations, we do not attempt a comprehensive review of this extensive literature, and instead provide a brief account for context of the LoRA methods. A. Learning low-rank subspaces Subspace methods explain high-dimensional observations using a small number of latent components. For example, principal component analysis (PCA) [145], [19], [133] formalizes the idea that a data ensemble concentrates near a low-dimensional linear subspace, and is closely related to the Karhunen–Loève (KL) expansion viewpoint for stochastic processes. This perspective has also been extended via kernel methods to incorporate nonlinearity [161]. Canonical correlation analysis (CCA) [67] extracts maximally correlated low-dimensional representations from two or more data sets, whereas independent component analysis (ICA) [30], [75] identifies statistically independent latent sources. Dictionary learning [143] generalizes fixed-basis subspace by adaptively learning an overcomplete set of atoms from data, with which each observation can be expressed as a sparse linear combination of these atoms. Going beyond a single global subspace, subspace clustering addresses the problem of grouping data that lie in a union of multiple low-dimensional subspaces. Multidimensional scaling (MDS) [225] embeds data points into a low-dimensional Euclidean space while preserving pairwise distance or dissimilarity. In contrast, local linear embedding (LLE) [185] performs nonlinear dimensionality reduction by preserving local geometry, which represents each data point as a weighted combination of its neighbors, and optimizes the subspace to maintain these reconstruction weights. These subspace-based techniques have impacted a wide range of applications, including face recognition [192], data visualization [10], multi-view [24] and multi-modal learning [5], and blind source separation [221]. Recovering a ground-truth signal from its limited and noisy linear measurements is one of the central themes in SP. When the unknown signal is a low-rank matrix A ∈ Rm×n , these inverse problems are known as matrix sensing under general

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

linear measurements [22], or low-rank matrix completion when the measurements are partially observed entries [153]. When the two factor matrices are non-negative, the problem is termed non-negative matrix factorization (NFM) [99]. Such lowrank structures are intrinsic to many real-world applications, and have been extensively exploited across areas such as phase retrieval [21], quantum state tomography [53], image inpainting [85], recommender systems [94], sensor network localization [36], and wireless communications [87]. While low rank can be accounted via convex relaxations such as nuclear-norm [153]), a scalable approach is to rely on the BM factorization [16], which optimizes over two smaller factor matrices X ∈ Rm×r and Y ∈ Rn×r such that XY⊤ ≈ A. The associated factorization task has been studied from multiple perspectives, including the loss function landscape [49], [48], efficient algorithms with convergence guarantees [153], [27], and statistical properties such as identifiability and finite-sample error rates [153], [139]. It will be argued later that LoRA is structurally isomorphic to BM factorization. Thus, the rich literature on matrix sensing and completion offers a powerful analytical basis for understanding the effectiveness of LoRA. B. Learning low-rank tensors

as CP and Tucker decompositions. The resultant nonconvex optimization solvers leverage the loss function geometry to establish statistical guarantees [37], [50]. Besides low rank, SP modeling approaches also account for sparsity features; see for example [20], [133], [130]. Given that LoRA also imposes low-rank structure in the parameter space of LLMs for efficient fine-tuning, it naturally shares a common foundation with these SP methods. From an SP viewpoint, however, a central distinction is that classical SP problems are often concrete and well-defined, whereas LLMs remain considerably more ambiguous given our still limited theoretical and empirical understanding. Prompted by the link as well as the gap, the present survey re-examines the PEFT variants to distill a self-contained account that bridges these two fields. Our goal is twofold: (i) bring SP insights to design principled and well-justified fine-tuning methods for LLMs; and (ii) cross-fertilize ideas developed in modern deep learning to inspire innovative approaches to low-rank SP problems. III. L OW- RANK IN FINE - TUNING LLM S This section reviews LoRA, compares it with other PEFT methods, and highlights open-source codebases that implement LoRA and its variants. Our goal is a streamlined guide for uninitiated readers. We also demonstrate that classical matrix sensing can be viewed as LoRA applied to a single linear layer, which offers a simple testbed to bridge the two communities.

Another SP perspective, where low-rank structure plays a central role, is tensor modeling. While tensor algebra generalizes many familiar matrix concepts, it also introduces important differences. For instance, determining tensor rank is A. LoRA recap NP-hard in general, and unlike the matrix case, a best low-rank The majority of LLM parameterizations comprise weight approximation of a higher-rank tensor may not exist. These matrices of their linear layers. Taking GPT-3 as an example, the characteristics are closely tied to the fact that there are multiple model involves multiple Transformer blocks, each containing 6 definitions of tensor rank, together with richer identifiability of linear mappings with associated weight matrices: the query Q, factor models and more intricate optimization approaches. For key K, value V, and output O pertaining to the (self-)attention comprehensive background, the reader is referred to relevant module, and two feed-forward network (FFN) weights F1 and SP tutorials such as [175], [91]; nevertheless, the present survey F (also known as the up- and down-projection) [15]; see 2 assumes no extensive prior knowledge of tensors. also Figure 1. Full fine-tuning updates all six matrices, whose Two of the most widely adopted low-rank tensor representadimensions are huge, typically in the order of millions each. tions are canonical polyadic (CP), and Tucker decompositions. Given the sheer scale of modern LLMs, this approach incurs The former was introduced in [65], [66], and later rediscovered prohibitive memory and computational costs in resource-limited under abbreviated names such as PARAFAC, CANDECOMP, or settings, which scale at least linearly with model size1 . CP form [91]. It expresses a tensor as a sum of rank-one tensors, Motivated by the empirical observations that adapting within extending the familiar rank-one expansion of matrices to higha low-rank subspace can retain most of the performance order data arrays (slabs); the formal expression will be provided of full fine-tuning [71], LoRA freezes the pre-trained linear later when discussing tensorized adapters. Alternatively, Tucker layers, and postulates that the task-specific weight updates are decompositions [191], [32] can be viewed as a high-order approximately low-rank. For notational simplicity, we do not generalization of the matrix singular value decomposition distinguish individual layer types, but instead represent a pre(SVD), where a tensor is represented with mode-wise factor trained weight matrix by Wl ∈ Rml ×nl , where l indexes the matrices together with a smaller core tensor. Low-rank tensor models have been widely adopted in both linear layers across Transformer blocks. With ∆Wl denoting SP and machine learning, with applications to coding in the additive update to the given Wl , fine-tuning under this telecommunications [176], community detection in social net- constraint of low-rank r can be expressed as ft works [29], and topic modeling [4]. Classical matrix recovery min f ({Wl + ∆Wl }L l=1 ; D ) L {∆W } l l=1 tasks such as sensing and completion have also been generalized (1) s.t. rank(∆Wl ) ≤ r, l = 1, . . . , L to tensor settings, addressing a range of higher-order inverse problems; see, e.g., [88], [242], [45]. Methodologically, tensor where f is the chosen loss function over the fine-tuning data low-rank structures can be promoted via convex surrogates in a Dft of downstream tasks. Loss f and the low-rank constraint manner analogous to the matrix case [45], but scalable methods 1 Exact dependence also relies on hyper-parameters such as sequence length. typically optimize over explicit factorized representations such

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

methods, and what accounts for its popularity in practice? LLMs rely on an autoregressive process to generate responses conditioned on a given prompt. Building on this, prompt and prefix tuning adapt models by learning a small set of taskspecific “soft prompts” [101], [109]. Prompt tuning prepends learnable prompt vectors to the input, while prefix tuning injects learnable prefix key/value vectors into attention calculation, steering the model’s behavior without modifying the pre-trained weights. However, because these methods are equivalent to leveraging additional tokens, they can introduce additional inference overhead, e.g., longer context and extra KV-cache. In contrast, LoRA introduces no extra inference overhead when serving a single task, since the low-rank updates can be merged into the model weights prior to deployment. Moreover, LoRA Fig. 1. LoRA fine-tuning of a GPT-3 model. Grey and orange boxes are respectively frozen (snowflake icon) weights of linear layers, and trainable does not rely on the autoregressive prompting interface and (fire icon) LoRA weights. is therefore applicable to non-autoregressive settings such as language-diffusion models [141]. Finally, prompt- and prefixin (1) are nonconvex. Following low-rank approaches in SP, based methods typically do not directly adapt FFN layers, LoRA achieves parameter efficiency by adopting the BM which are often conjectured to store factual knowledge, and factorization that sets ∆Wl = Xl Yl⊤ , where Xl ∈ Rml ×r are commonly targeted in knowledge editing [41]. Adapter-based fine-tuning [68] and zeroth-order optimization and Yl ∈ Rnl ×r [16]. For brevity, the range of l and the methods (often referred to as MeZO) [127], [237], [235] are ft dataset D will be omitted in the following. This simple two other popular approaches to memory-efficient adaptation reparameterization represents an ml nl -entry matrix using of LLMs. Adapter methods insert small trainable modules only (ml + nl )r trainable parameters. Under the low-rank into an otherwise frozen backbone [68]. At the extreme end r ≪ min{ml , nl }, the parameter count is drastically reduced. of parameter efficiency, BitFit [229] freezes all weights and An additional practical benefit is that the rank constraint fine-tunes only the bias terms, which minimizes the number in (1) is implicitly enforced by construction, and yields the of trainable parameters but can also limit expressiveness. unconstrained optimization problem Alternatively, MeZO targets extreme memory efficiency by min f ({Wl + Xl Yl⊤ }l ) (2) estimating updates without storing backpropagation activations, Xl ,Yl requiring negligible additional memory beyond the backbone which can be directly optimized using standard deep learning model. Compared with these approaches, LoRA offers a algorithms; see also Figure 1 for a graphical illustration. It is desirable accuracy–efficiency tradeoff, and flexibility with also worth noting that the original LoRA in [71] adapts merely respect to available computing and memory. As resources the query and value matrices in the attention module, whereas increase, one can simply increase the LoRA rank r to allocate subsequent variants often treat the choice of target layers as a more expressive capacity to the adapters. A full comparison flexible, user-specified design option. can be found in Table I. LoRA can markedly lower the required resources for finetuning, while only incurring a small performance drop, or C. LoRA codebase even outperforming full fine-tuning when the latter overfits the Thanks to the open-source community, LoRA and its dataset. For example, [71] reports that fine-tuning RoBERTavariants have been integrated into multiple software packages. Large (335M) requires only about 0.8M LoRA parameters, Early libraries such as AdapterHub (now adapters)2 and fine-tuning GPT-3 175B requires 37.7M LoRA parameters, and HuggingFace PEFT [128] played a key role in making yet achieving performance comparable to (or slightly better LoRA accessible, and are also being updated with recent than) full fine-tuning on GLUE and on WikiSQL/MNLI, LoRA variants. Today, LoRA is deeply integrated into the respectively. For GPT-3 175B, LoRA reduces the reported GPU HuggingFace ecosystem, with strong support for both LLMs via memory requirement from 1.2 TB to 350 GB. More recent transformers and diffusion models through diffusers. advances further push the limits of resource efficiency. For Tools such as Unsloth3 and bitsandbytes4 further proinstance, [33] demonstrates that even a 65B-parameter model vide GPU-friendly optimizers that make LoRA-style fine-tuning can be fine-tuned on a single 48GB GPU, substantially lowering faster and more memory-efficient. Meanwhile, vLLM [96] and the hardware barrier for individuals and small labs to adapt SGLang5 offer efficient inference and deployment support reasonably large-scale models. Beyond resource efficiency, for LoRA-adapted models. Recently, Thinking Machines Lab’s LoRA has also enabled a wide range of applications across Tinker6 has introduced LoRA-based fine-tuning as a managed diverse domains, as elaborated later in Section VI. 2 https://github.com/adapter-hub/adapters

B. Comparison with other PEFT methods

3 https://github.com/unslothai/unsloth

Beyond LoRA, several PEFT alternatives have emerged. The natural questions are: how does LoRA compare to these

5 https://github.com/sgl-project/sglang

4 https://github.com/bitsandbytes-foundation/bitsandbytes 6 https://thinkingmachines.ai/tinker/

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

Fig. 2. Example code snippet of using LoRA through PEFT [128] codebase. TABLE I C OMPARISON OF PARAMETER - EFFICIENT FINE - TUNING APPROACHES . Approach

Memory

Performance on simple tasks (e.g., GLUE)

Performance on challenging tasks (e.g., math, coding. . . )

Extendability

Full parameter FT Adapter BitFit Prefix tuning LoRA

High Moderate Very low Low Low

Strong Strong Moderate Strong Strong

Strong Moderate Weak or unknown Moderate Moderate – Strong

Low Moderate Moderate Moderate High

API for open-weight models, and has also showcased LoRA’s Given fine-tuning data Dft , and viewing Wpre as a frozen linear effectiveness in reinforcement learning workflows for LLMs. layer, applying LoRA to this layer leads to To illustrate how simple it is to use LoRA, we modify an 1 ft min ∥y − m(Wpre + XY⊤ )∥2 (4) example from the PEFT library [128] in Figure 2. In particX∈Rm×r ,Y∈Rn×r 2 ular, enabling LoRA requires only two additional commands (lines 7–13), highlighting how the open-source ecosystem has where yft = m(Woft ). Upon defining y := yft − m(Wpre ) = m(∆Wo ), the linearity of m simplifies (4) to substantially lowered the barrier to entry for beginners. D. LoRA on a single linear layer

min

X∈Rm×r ,Y∈Rn×r 2

∥y − m(XY⊤ )∥2

(5)

Applying the BM factorization to eliminate the explicit rank which is exactly a low-rank matrix sensing loss written in its constraint in (1) hints to a link of LoRA with low-rank SP BM-factorized form. Although this single-layer reduction is techniques. Next, we unveil this link by showing that the low- far simpler than fine-tuning a deep LLM, it already provides a rank matrix sensing problem can be viewed as the simplest clean analytical testbed along with several useful insights. First, it offers a straightforward sanity check on the exinstance of LoRA fine-tuning. pressiveness of a LoRA variant: among the many proposed Recall that matrix sensing recovers an unknown sigm×n designs, it is reassuring if a method can solve the basic onenal matrix Wo ∈ R from a collection of N data layer problem (5) efficiently. That said, this should not be (or measurements) D := {(Mi , yi )}N , where M ∈ i i=1 Rm×n is a sensing matrix that yields observation yi = interpreted as a strict requirement, but rather an informative ⊤ N indicator of potential tradeoffs. Some variants are designed with Tr(M⊤ i Wo ). With y := [y1 , . . . , yN ] ∈ R , and m(W) := ⊤ ⊤ different priorities in mind, such as faster training, or escalated [Tr(M⊤ W), . . . , Tr(M W)] , signal sensing is obtained as 1 N hardware efficiency, thus may intentionally trade expressiveness Ŵo = arg min ∥y − m(W)∥2 . (3) for advantages in specific regimes. Second, this problem also W∈Rm×n provides an analytically tractable setup for understanding how Now revisit sensing through the lens of pre-training and fine- optimization algorithms cope with the nonconvex, globally tuning. Suppose that the ground-truth matrix to learn during nonsmooth loss function landscape and rich gauge invariance pre-training is Wopre , and that the model has been fit to global (see Section V-D) induced by LoRA’s bilinear structure. Since optimality with Wpre = Wopre . In the subsequent fine-tuning tracking the full dynamics of LoRA fine-tuning in LLMs stage, let Woft = Wopre +∆W denote the perturbed new ground remains largely open, this simplified problem offers a useful truth, where the perturbation ∆A is constrained to be low-rank. proxy that retains these core optimization challenges while

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

trainable parameters. Using a full Σl ∈ Rr×r introduces r2 additional parameters compared to standard LoRA (or only r additional parameters if diagonal). In practice, this overhead is negligible as commonly used ranks are sufficiently small, e.g., r = 8 or 32. A graphical illustration of these parameterizations can be found in Figure 3. Dynamic rank allocation. Standard LoRA allocates the same rank budget r to every layer of an LLM, yet not all layers contribute equally to downstream tasks. It thus motivates dynamic, data-dependent rank allocation across layers. For a diagonal rl × rl matrix Σl , SVD reduces to rl X Ul Σl Vl⊤ = σls usl (vls )⊤ s=1 s where σl is the s-th diagonal entry of Σl , and usl and vls are

Fig. 3. SVD factorization in AdaLoRA [236] (top) and PoLAR [114] (bottom), where Stiefel manifold St(m, r) := {U ∈ Rm×r | U⊤ U = Ir }.

remaining amenable to analysis. Indeed, several methods originally developed for solving (5) have proven effective in LoRA fine-tuning; cf. Sections V-B and V-C. IV. M ODEL A RCHITECTURES OF L O RA Having outlined low-rank tools from SP, attention now turns to fine-tuning methods that are transferable from SP and those developed more recently. The focus is first finetuning itself, with broader applications deferred to later sections. Akin to conventional SP problems such as (5), efficiency of a fine-tuning method often relies on the interaction between i) the adapter model architecture, and ii) a paired optimization algorithm that leverages the architecture. For this reason, the ensuing section begins with architectural (model) designs beyond the BM factorization that is the cornerstone of LoRA. A. SVD-based parameterization Another parameter-efficient alternative for parameterizing ∆Wl ∈ Rm×n is the factorization provided by SVD ∆Wl = Ul Σl Vl⊤

m×r

n×r

(6)

where Ul ∈ R and Vl ∈ R have orthonormal ⊤ columns; that is, U⊤ U = I and V V l r l = Ir , or satisfy l l relaxed versions thereof. Relaxing the orthogonality constraints will turn out to simplify modeling and enable GPU-friendly optimization. The middle factor Σl ∈ Rr×r can be chosen to be diagonal or left unconstrained depending on the target settings; these choices will become explicit shortly. The motivation behind an SVD-type model, for the same expressiveness as BM factorization, is two-fold. First, SVD facilitates explicit growth or pruning of singular directions, and thus naturally supports task-adaptive, layer-wise rank allocation [236]. Second, orthogonality in Ul and Vl promotes effective utilization of allocated parameters, especially when the number of optimization steps is limited [114]. A minor limitation of this parameterization is the slightly increased

the s-th columns of Ul and Vl , respectively. This admits a natural interpretation: σls serves as an explicit importance score for the s-th rank-one summand usl (vls )⊤ . Consequently, this allows for pruning the less important components σls usl (vls )⊤ when σls falls below a certain threshold. Moreover, the removal of individual subspaces becomes fully decoupled, making it possible to prune rank in a flexible manner. AdaLoRA [236] leverages this idea with two modifications. First, instead of enforcing exact orthogonality, it encourages approximately orthogonal U and V by adding soft penalty terms ∥U⊤ U−I∥2F and ∥V⊤ V − I∥2F to the loss. The resulting unconstrained formulation is often easier to optimize in standard deeplearning pipelines. Second, the importance score is enriched by incorporating its sensitivity and uncertainty of subspaces, thereby making the allocation mechanism more robust to the stochasticity of training. Concretely, for a layer l with effective rank rl⋆ , AdaLoRA starts from an over-parameterized rank r > rl⋆ , and progressively prunes less important subspaces every few fine-tuning steps, thereby producing a task-dependent rank allocation across layers. As opposed to progressive pruning, IncreLoRA [233] instead grows the singular space incrementally during fine-tuning. Further, it is worth stressing that dynamicrank methods are not limited to SVD-type. SoRA [35] removes the orthogonality constraints on U and V. DyLoRA [193] proposes an analogous layer-wise rank allocation Prl strategy s s ⊤ under the BM form by having Xl Yl⊤ = s=1 xl (yl ) . However, the absence of orthogonality allows different rankone components to interact through correlated directions. This entangles the contributions of each component, making the notion of “importance” ambiguous and the associated pruning criterion delicate to design. Improving parameter efficiency. For the SVD-type parameterization (6), PoLAR [114] and SteLLA [112] impose respectively approximate and strict orthogonality on Ul and Vl , both with an unconstrained Σl . This unconstrained Σl can be viewed as a generalized form of weight normalization [157], [205] induced by the BM factorization. Specifically, applying the polar decomposition to vanilla LoRA renders Xl = Ul Σ1l , 1 r×r where U⊤ l Ul = Ir encodes the “direction,” and Σl ∈ R is a positive semi-definite (PSD) matrix representing the “magnitude,” which is analogous to weight normalization. Similarly, Yl = Vl Σ2l with Vl⊤ Vl = Ir and PSD Σ2l . This SVD-type model with unconstrained Σl can be then obtained by

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

HeSw

PoLAR LoRA DoRA

14 12

k∆Wk2F /k∆Wk22

PoLAR LoRA

Stable Rank

10 8

layer type down projection up projection query projection key projection value projection adapter PoLAR LoRA

6 4

2 r

A Q

W iG

CAR

C-

A Q

BQ O

eS H

ol Q

Rank

Task

(a) Stable rank across different datasets

(b) Stable versus algebraic rank

100 Step

125

150

175

200

Fig. 4. (a) PoLAR parameterization facilitates better utilization of rank. (b)(c) Fine-tuning a LLaMA2-7B model on the HellaSwag dataset [230]. ⊤ rewriting the BM factorization as Xl Yl⊤ = Ul (Σ1l Σ2⊤ l )Vl , and merging the two magnitude matrices into a single one Σl := (Σ1l Σ2⊤ l ). This clarifies why Σl is unconstrained: product of two PSD matrices need not be PSD. To handle the orthogonality constraints U⊤ l Ul = Ir and Vl⊤ Vl = Ir , SteLLA uses retraction-based Riemannian optimization (discussed in Section V). PoLAR instead adopts the Landing algorithm [1], which adds a penalty 2 term ∥U⊤ l Ul − Ir ∥F (and similar for Vl ) to avoid the SVD operation required by retraction, thus improving the compute efficiency. When combined with these optimization methods, the SVD-type parameterization enables effective utilization of the allocated subspace. Figure 4(a) reports the stable rank7 of ∆Wl across layers of a LLaMA2-7B model fine-tuned on commonsense reasoning tasks [116]. Despite using r = 32, the stable rank is close to 1, which demonstrates that LoRA often exploits a rank-one subspace, and its expressiveness is not fully explored. In contrast, PoLAR achieves substantially higher stable ranks. Figure 4(b) plots the stable rank of PoLAR and LoRA against the algebraic rank r, where the stable rank of the former increases with r. Figure 4(c) further illustrates the finetuning dynamics. It is seen that PoLAR consistently maintains a larger stable rank than LoRA. These results indicate that for the same expressiveness, PoLAR more effectively utilizes the available rank budget than LoRA in practice.

B. Rank-augmented parameterization While BM and SVD factorizations are efficient, their expressiveness is limited to a rank-r matrix. This naturally raises the question of whether one can represent higher-rank ∆Wl without increasing the number of trainable parameters. Hadamard based parameterization. A means of increasing the rank is to leverage the Hadamard product whose rank is submultiplicative [179]. Theorem 1 ([135]). The Hadamard product of A, B ∈ R is   a11 b11 · · · a1n b1n   .. .. .. A ⊙ B :=   . . . am1 bm1

···

m×n

amn bmn

and its rank satisfies rank(A ⊙ B) ≤ rank(A)rank(B). 7 Defined as ∥∆W ∥2 /∥∆W ∥2 , the stable rank is a smooth surrogate of l l F the algebraic rank that downweighs small singular values.

Fig. 5. Rank-augmented parameterization. From up to down are FedPara [74], HiRA [73], KronA [40], and RoSA [142].

In particular, the Hadamard product of two rank-r matrices can have rank as large as r2 . Motivated by this observation, FedPara [74] (or LoHA [223]) has additive weight update ∆Wl = (X1l Yl1⊤ ) ⊙ (X2l Yl2⊤ )

(7)

where X1l , X2l ∈ Rm×r and Yl1 , Yl2 ∈ Rn×r are learnable LoRA adapters. The next proposition characterizes the expressiveness of Hadamard-based parameterization. Proposition 1 (Expressiveness of FedPara [74]). The parameterization ∆W = (X1 Y1⊤ ) ⊙ (X2 Y2⊤ ) can express (i) all matrices with rank up to r; and, (ii) a subset of matrices having rank r < rank(∆W) ≤ r2 . A naive implementation of (7) would first learn the two dense m × n matrices X1l (Yl1 )⊤ and X2l (Yl2 )⊤ (in both forward and backward passes), and then take their Hadamard product. However, this incurs an O(mn) memory footprint for the intermediates. A memory-efficient implementation was pointed out by [178], which avoids forming any m×n intermediate via

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

a row-wise Khatri–Rao product [90]. Specifically, for A, B ∈ Rm×n written in rows  ⊤  ⊤ a1 b1 ⊤  a⊤   2   b2  A= · · · B =  · · ·  , a⊤ b⊤ m m the row-wise Khatri–Rao product ⋆ is defined as  ⊤  a1 ⊗ b ⊤ 1 ⊤  a⊤ m×n2 2 ⊗ b2  A ⋆ B :=   ···  ∈ R ⊤ a⊤ m ⊗ bm where ⊗ denotes the Kronecker product  a11 B a12 B · · ·  a21 B a22 B · · ·  A⊗B= . .. ..  .. . . am1 B

am2 B

···

 a1n B a2n B   ..  . . 

r 2 ×n

amn B

∆Wl =

K X k=1

}

As a consequence, the forward/backward calculation can be implemented using two “thin” factors of sizes m × r2 and n × r2 , thus avoiding the sizable m × n matrix. While FedPara is more expressive, it also poses additional optimization challenges as the adapter becomes “deeper,” involving four learnable matrices as opposed to two in standard LoRA. In general, deeper parameterizations are harder to optimize, and can exhibit slower training dynamics [166]. HiRA8 [73] aims to endow ∆Wl with a higher rank by tying one branch of the Hadamard product to the frozen pre-trained weight, which may itself have high rank: ∆Wl = Wl ⊙ (Xl Yl⊤ ).

where Xl ∈ Rd1 ×d2 and Yl ∈ Rd3 ×d4 . KronA requires the pre-trained weight Wl ∈ Rm×n satisfying the so-called shape constraints m = d1 d4 and n = d2 d3 . This mild requirement is typically compatible with common pre-trained model weight sizes. The rank of ∆Wl can be as large as min{d1 , d2 } × min{d3 , d4 }. But as asserted next, not every matrix of desirable size can be represented as a single Kronecker product.

A known remedy is to rely on an ensemble (weighted sum) of Kronecker factors. This idea is leveraged in [156] with

∆Wl = (X1l Yl1⊤ ) ⊙ (X2l Yl2⊤ ) = (X1l ⋆ X2l ) (Yl1 ⋆ Yl2 )⊤ . m×r 2

∆Wl = Xl ⊗ Yl

Proposition 3. For matrices X, Y of rank at most r, parameterization ∆W = X ⊗ Y can express a subset of matrices of rank up to r2 .

With this notation,the update (7) can be rewritten as

This property is exploited in KronA (also known as LoKr) [40], [223]. The adapter matrix is parameterized via

(8)

The expressiveness of this parameterization is presented in the next proposition. Proposition 2 (Expressiveness of HiRA). Given W, the parameterization ∆W = W ⊙ (XY⊤ ) can express 1) a subset of matrices of rank up to r, and 2) a subset of matrices whose rank is r < rank(∆W) ≤ min{rank(W) · r, m, n}. A caveat however, is that expressiveness depends critically on Wl . For m = n and Wl = Im , for example, HiRA can represent solely diagonal matrices. Kronecker based parameterization. While the rank of a Hadamard product is submultiplicative, the rank of a Kronecker product is multiplicative, as asserted next. Theorem 2 ([194]). For matrices A ∈ Rd1 ×d2 and B ∈ Rd3 ×d4 , the d1 d3 × d2 d4 Kronecker product matrix has rank satisfying rank(A ⊗ B) = rank(A)rank(B). 8 A similar construction can be found in [207] yet for different purposes; cf. Section VI-C.

αlk (Xkl ⊗ Ylk )

to improve KronA’s expressiveness. The coefficients αlk are learned from data alongside the factors Xkl , Ylk during finetuning, though they may equivalently be absorbed into Xkl . Low-rank plus sparsity parameterization. Leveraging lowrank and sparsity has well-documented impact across SP areas, including robust PCA [133], [19]. The upshot of such properties is expressiveness: superposition of low-rank plus sparse matrix components can represent a broad family of matrices, including full-rank ones. A simple example for this rank augmentation is a properly scaled identity matrix as the sparse component; adding this to a low-rank component yields a full-rank matrix. Building on this idea, RoSA [142] augments LoRA with a sparse matrix Sl ∈ Rm×n via ∆Wl = Xl Yl⊤ + Sl .

While robust PCA optimizes the sparse term Sl via an ℓ1 regularizer of the vectorized matrix min

Xl ∈Rm×r ,Yl ∈Rn×r ,Sl ∈Rm×n

f (Xl Yl⊤ + Sl ) + λ∥vec(Sl )∥1

this formulation does not account for parameter efficiency. During training, learning the sparsity pattern end-to-end still requires viewing Sl as a dense m × n matrix, which is prohibitively expensive to store at the scale of modern LLMs. Many practical approaches sidestep this issue by fixing the sparsity pattern a priori and optimizing only the nonzero values [142], [12]. This is a justifiable design choice: a random sparse matrix can be full-rank, provided its density is above certain threshold, as summarized in the following result. Theorem 3 ([115]). An n×n matrix A with i.i.d. Bernoulli-(p) entries has full rank with high probability p ≳ logn n . Consequently, this allows for representing full-rank matrices. Proposition 4. Parameterization ∆W = XY⊤ +S can express 1) all matrices of rank up to r, and 2) a subset of matrices with rank r < rank(W) ≤ min{m, n}.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

(a) Representation similarity

(b) Weight similarity

Fig. 6. Visualization of correlated layers in Llama 3.1-8B-Instruct using linear centered kernel alignment (CKA) [95]. (a) Representation similarity before the feedforward network. (b) Weight similarity of the MLP (up projection). This figure is taken from [136].

Recent works have also pursued parameterizations relying purely on sparsity; see e.g., SHiRA [12] that adopts ∆W = W0 ⊙ S where S ∈ Rm×n is a fixed sparse mask. This sparsity pattern is a design choice and can be sampled randomly or based on importance heuristics. To collect sparsity benefits on hardware, one can design specialized compute kernels9 that can effectively leverage the sparse engines available in recent GPUs. C. Tensorized parameterization

Fig. 7. LoRA with [69], Tucker [11], and TT [220] parameterizations.

generalizes the notion of matrix rank to higher-order tensors. It represents an N -th order tensor as the superposition of rankone summands, given by the outer product (denoted by ◦) of N vectors A Q := s1 ◦ . . . ◦ sN , with (i1 , . . . , iN )-th entry N A[i1 , . . . , iN ] = n=1 sn,in . For a three-way additive fine-tuning update ∆W ∈ Rm×n×L , CP based methods parameterize it as a rank-r tensor

While LoRA and the aforementioned variants improve efficiency via layer-wise low-rank updates, they typically treat adapters in different layers as unrelated matrices, and do not account for potential correlations across layers. However, recent studies suggest that layers in deep networks can be related, and r often exhibit substantial redundancy or shared structure [136], X ∆W = JS , S , S K := s1,i ◦ s2,i ◦ s3,i [97], [31]; see also Figure 6 for an exmaple in Llama-31 2 3 i=1 Instruct. Overlooking these inter-layer dependencies may leave parameter efficiency on the table. where S1 ∈ Rm×r , S2 ∈ Rn×r , and S3 ∈ RL×r are factor Parameters of tensor models offer a prudent approach to matrices. Here, sk,i denotes the i-th column of Sk , k = 1, 2, 3. capturing such topological connections. Consider a collection of This parameterization involves r(L + m + n) parameters, weight matrices with identical dimensions {Wl ∈ Rm×n }L offering remarkable savings compared to Lmn parameters l=1 , e.g., the query projection matrices in attention blocks. During present in the full update ∆W. fine-tuning, their additive updates {∆Wl ∈ Rm×n }L In practice, the effectiveness of these approaches also l=1 can be stacked to form a three-way tensor ∆W ∈ Rm×n×L . Notably, depends on how weights are tensorized before applying CP these weights can be reshaped into higher-order tensor models factorization. For example, CaRA [195] for Vision Transformin various ways. This subsection will focus on third-order ers (ViTs) groups the multi-head query, key, and value (QKV) tensors for clarity, while the analysis can be readily extended projection matrices across layers to form a fourth-order tensor, to higher-order tensors. If ∆W exhibits low tensor rank while aggregating the output projection and the two FFN layers under an appropriate definition10 , it is natural to adopt tensor into another tensor, with CP applied to each tensor separately. factorization schemes to parameterize the updates compactly, Likewise, in LLM settings, LoRTA [69] structures the updates thereby capturing the relation across layers while reducing the of QKV and output projection matrices as a fifth-order tensor number of trainable parameters. The total parameter count can parameterized using the CP decomposition. drop further from the LoRA-style scaling of O((m + n)rL) Beyond parameter efficiency, a major upshot of CP in SP to O((m + n + L)r), where the dependence on the L layers is its identifiability, which asserts under mild conditions that becomes additive rather than multiplicative. An overview of the CP decomposition of a specific tensor is unique up to the tensor-based parameterization is depicted in Figure 7. permutation and scaling of components; see, e.g., [83], [175]. Canonical-Polyadic (CP)-based parameterization. CP Intriguing questions for fine-tuning is whether identifiability decomposition is a popular tensor factorization model that could yield more interpretable updates, or, be leveraged to improve the optimization landscape of tensorized adapters. 9 Here “kernels” refers to GPU compute kernels (i.e., codes), not e.g., Tucker decomposition-based parameterization. Another Gaussian/RBF kernels. 10 Computing tensor rank is generally NP-hard. popular SP tool for tensor factorization is the Tucker decom-

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

position [191], [32], which can be viewed as a higher-order counterpart of the matrix SVD:

stacked additive weights form a three-way tensor ∆W ∈ Rm×n×L and rely on a three-core TT factorization to obtain

A = G ×1 U1 ×2 . . . ×N UN

∆W[i, j, k] = G1 [i, :]G 2 [:, j, :]G3 [:, k]

where G1 ∈ Rm×r1 , G 2 ∈ Rr1 ×n×r2 , and G3 ∈ Rr2 ×L (recall where G is a smaller core tensor, and {Un }N n=1 are mode-wise factor matrices (generalized “singular vectors”). Unlike the that r0 = r3 = 1). TT involves r1 m + r1 r2 n + r2 L parameters, matrix SVD where the core (singular value matrix) is strictly but its parameter count is sensitive to the permutation of modes; diagonal, the Tucker core G is typically dense. Conceptually, e.g., swapping the second and third modes changes intermediate this can be viewed as a generalization of the PoLAR magni- rank requirements, and thus the total parameter count. In practice, implementation strategies vary by how the tude–direction decomposition, where the factor matrices capture subspace directions per mode, while the core tensor depicts weights are tensorized. Fact-TT [84] concatenates the weights the multi-way “magnitudes” and couplings across modes. per layer of a ViT to form a three-way tensor, and applies TT deSP algorithms often impose orthogonality constraints on the composition for parameterization. In contrast, LoRETTA [220] factor matrices, whereas modern deep learning implementations reshapes the weight matrices of a single layer to form highertypically relax this requirement to simplify optimization. Our order tensors, and uses TT cores to parameterize the associated updates. MetaTT [120] further extends this approach by discussion hereafter follows the latter. In fine-tuning, the stacked additive weights ∆W ∈ Rm×n×L incorporating both the layer dimension and the intra-layer dimensions into a single high-order TT parameterization. can be parameterized via Tucker decomposition as Comparison. Next, a brief comparison of these tensor-based ∆W = G ×1 U1 ×2 U2 ×3 U3 (9) methods is provided. In terms of parameter count, CP and r1 X r2 X r3 TT are often more compact than Tucker. Specifically, for X := gijk u1,i ◦ u2,j ◦ u3,k an N -way tensor, their trainable parameters typically scale i=1 j=1 k=1 roughly linearly with order N , whereas Tucker includes a where U1 ∈ Rm×r1 , U2 ∈ Rn×r2 , and U3 ∈ RL×r3 are factor core tensor whose size grows multiplicatively with N . While matrices; G ∈ Rr1 ×r2 ×r3 is the core tensor; and ×k denotes efficient, TT is often less interpretable than CP or Tucker. the mode-k tensor–matrix product. Tucker represents an mnL- Furthermore, because TT parameterization involves a long dimensional tensor using r1 r2 r3 +Lr1 +mr2 +nr3 parameters. chain of matrix products, it can be viewed as a generalization of This full three-mode factorization is also known as Tucker- deep linear networks. Consequently, it may introduce additional 3. In practice, tensorization can markedly affect parameter optimization challenges as N grows [6]. For instance, the efficiency, as the core size r1 r2 r3 grows multiplicatively with iteration complexity of gradient descent can grow exponentially the chosen multilinear ranks. Fact-TK [84] concatenates weight with depth (and thus with tensor order) in certain regimes [166]. matrices across ViT layers (with FFN linear layers sliced to For fine-tuning, all three parameterizations appear promising compatible sizes) to form a three-way tensor, and applies empirically, and in many reported settings they can match Tucker-3 factorization to parameterize the update. Alternatively, matrix-based adapters while using fewer trainable parameters. LoTR [11] stacks weights from LLM layers to form a third- However, empirical evidence is still fragmented across tasks, order tensor, but adopts a Tucker-2 variant, which leaves one models, and tensorization choices. Developing principled guidelines for, e.g., which particular decomposition is preferable mode uncompressed by setting U3 = I and r3 = n; that is, under specific hardware or data regimes, remains an important uncharted territory. ∆W = G ×1 U1 ×2 U2 ⇔ ∆Wl = U1 Gl U⊤ . (10) 2 This formulation naturally admits a layer-wise interpretation: D. Other SP-related parameterizations all layers share the same U1 and U2 , while each layer learns Beyond structured low-rank matrices and tensors, several its own core slice Gl . Note that if we merge Ul := U1 Gl variants also connect naturally to SP through the lenses in (10), then each layer update becomes ∆Wl = Ul U⊤ . This 2 recovers a weight-sharing variant of LoRA [154], where the of parameter sharing and rotation-based parameterizations. Parameter sharing has long played a central role in SP, as right projection matrix is tied for (shared by) all layers. Tensor train (TT)-based parameterization. TT decomposi- exemplified by dictionary learning and sparse coding [144], tion represents a high-order tensor as a sequence or “train,” of whereas rotations and orthogonal transforms are classical low-order cores. This enables a parsimonious parameterization tools for representing signals in more efficient and structured whose complexity grows linearly (rather than exponentially coordinate systems [198]. Parameter sharing. Sharing parameters across layers has as in Tucker) with the tensor order. Formally, given an N -th I1 ×···×IN been leveraged in deep learning to reduce the number of order tensor A ∈ R , TT expresses each entry as a trainable parameters, thereby improving memory efficiency; see product of matrices e.g., [97]. In the fine-tuning context, we have already discussed A[i1 , . . . , iN ] = G 1 [:, i1 , :]G 2 [:, i2 , :] · · · G N [:, iN , :] LoTR [11] through the lens of Tucker-2 parameterization. Additional examples include VeRA [93] that introduces a vectorwhere each core G k ∈ Rrk−1 ×Ik ×rk , and the matrix slice G k [: based variant of LoRA using the per-layer update , ik , :] ∈ Rrk−1 ×rk . Integers r0 , r1 , . . . , rN are the TT ranks, with boundary conditions r0 = rN = 1. For fine-tuning, the ∆Wl = diag(al )A diag(bl )B⊤

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

TABLE II A SUMMARY OF EFFICIENT ARCHITECTURE VARIANTS OF L O RA. Method

reference

Architecture

Notes

SP relation

LoRA

[71]

∆Wl = Xl Yl⊤

N/A

BM factorization

(m + n)rL

AdaLoRA PoLAR/SteLLA

[236] [114], [112]

∆Wl = Ul diag(σl )Vl⊤ Ul Σl Vl⊤

⊤ U⊤ l Ul ≈ I, Vl Vl ≈ I ⊤ r×r U⊤ l Ul ≈ I, Vl Vl ≈ I, Σl ∈ R

SVD SVD

≤ (m + n + 1)rL (m + n + r)rL

RoSA FedPara HiRA KronA

[142] [74] [73] [73]

Sl ∈ Rm×n yet sparse N/A N/A m = d1 d2 , n = d3 d4

Robust PCA Hadamard product Hadamard product Kronecker product

[(m + n)r + s]L 2(m + n)rL (m + n)rL (d1 d2 + d3 d4 )L

CaRA/LoRTA LoTR FacT-TK FacT-TT/MetaTT

[195], [69] [11] [84] [84], [120]

∆Wl = Xl Yl⊤ + Sl ∆Wl = (X1l Yl1⊤ ) ⊙ (X2l Yl2⊤ ) ∆Wl = Wl ⊙ (Xl Yl⊤ ) ∆Wl = Xl ⊗ Yl , Xl ∈ Rd1 ×d2 , Yl ∈ Rd3 ×d4 P ∆W = ri=1 s1,i ◦ s2,i ◦ s3,i ∆W = G ×1 U1 ×2 U2 ∆W = G ×1 U1 ×2 U2 ×3 U3 ∆W[i, j, k] = G1 [i, :]G 2 [:, j, :]G3 [:, k]

N/A weight sharing across layers N/A G1 ∈ Rm×r , G 2 ∈ Rr×n×r , and G3 ∈ Rr×L

CP Tucker-2 Tucker-3 TT

mr + nr + Lr mr + nr + r2 L mr + nr + Lr + r3 mr + nr2 + Lr

DoRA

[116]

N/A

Rotation

(m + n)rL + nL

Vera NoLA

[93] [92]

Wl +Xl Yl⊤ ∥Wl +Xl Yl⊤ ∥c ∆Wl = diag(al )A diag(bl )B⊤ PK PK k k k k ∆Wl = k=1 αl A k=1 βl B

A ∈ Rm×r and B ∈ Rm×r shared k n×r }K Dictionaries: {Ak ∈ Rm×r }K k=1 and {B ∈ R k=1

Parameter sharing Parameter sharing

(m + r)L K(m + n)r + 2KL

Wl + ∆Wl = ml

where A ∈ Rm×r and B ∈ Rn×r are fixed Gaussian random matrices shared across all layers. The only trainable parameters are the per-layer scaling vectors al ∈ Rm and bl ∈ Rr . In comparison, NoLA [92] employs a fixed dictionary of random k n×r K matrices {Ak ∈ Rm×r }K }k=1 shared k=1 and {B ∈ R across layers, and learns the layer-specific mixing weights αl ∈ RK and βl ∈ RK , to arrive at the layer-wise update ∆Wl =

K X k=1

αlk Xk

K X

βlk Yk .

(11)

# parameters

naturally viewed within the rotation framework. Concretely, DoRA parameterizes the update as Wl + ∆Wl = ml

Wl + Xl Yl⊤ ∥Wl + Xl Yl⊤ ∥c

where ∥ · ∥c denotes the column-wise ℓ2 norms of a matrix, with multiplication and division understood columnwise. Under this parameterization, the directional component is captured implicitly by normalizing each column of Wl + Xl Yl⊤ , while ml controls the corresponding scaling of each neuron.

k=1

While storing the dictionary can incur large memory overhead, E. A summary of efficient parameterizations Let us consider a case study to briefly compare the this can be mitigated by relying on structured matrices, e.g., parameterization approaches discussed so far. Specifically, Fourier bases, which can be generated on-the-fly [47]. consider a network with linear layers and associated weights Rotation-based fine-tuning. Aside from additive adapters, m×n L {W ∈ R } . l an alternative is to model fine-tuning updates as multiplicative l=1 This setup slightly idealizes the Transft pre n×n former architectures, where attention and multilayer perceptron W = W P with some P ∈ R . Different from additive (MLP) projections typically differ in dimensions. Nonetheless, updates, this choice reflects a distinct tradeoff. On the upside, ft pre pre it suffices to highlight the main ideas, and extend these the induced weight increment W − W = W (P − In ) ft results to heterogeneous shapes requires only minor algebraic can have high rank. On the downside, W is constrained pre adjustments. For an apples-to-apples comparison, all rankto lie in the column space of W , which can rule out related hyperparameters are standardized as r in both matrixupdates that additive LoRA can represent. Further, parameter and tensor-based parameterizations. The latter simply stacks the efficiency can be acquired by restricting P with additional m×n L layer weights {W ∈ R } to form a third-order tensor, l structure, such as a block-diagonal form. If one further imposes l=1 and does not consider alternative choices such as reshaping (approximate) orthogonality, this leads to orthogonal fine-tuning methods including OFT [148], BOFT [117], ETHER [13], and into higher-order tensors. Different parameterizations are HRA [227]. These approaches are popular in text-to-image summarized in Table II. A head-to-head empirical comparison generation, and can be interpreted as rotating the neurons, is deliberately omitted, as the performance of these methods is which are just the columns of Wpre . They differ mainly in how highly sensitive to the optimization strategy employed. Indeed, they constrain and parameterize P, offering different balance many parameterizations can be further improved when paired points between expressiveness and parameter efficiency. For with an optimizer tailored to their underlying structure. In example, OFT uses a block-diagonal parameterization of P, the next section, we review recent optimization advances that whereas BOFT is motivated by the Cooley–Tukey fast Fourier explicitly account for these adapter architectures. transform to represent a dense P as a product of sparse matrices V. E FFICIENT OPTIMIZATION OF L O RA via butterfly structures. Another way to express P implicitly is through a magBeyond adapter models, the optimization strategy is another nitude–direction decomposition of neurons (i.e., column of crucial ingredient for efficient fine-tuning. An example is Wl + ∆Wl ), as in DoRA [116]. While DoRA was originally depicted in Figure 4(c): without an optimizer that effectively presented through the lens of (vectorized) weight normalization, learns the adapter architecture, LoRA often underutilizes the a widely used technique in deep learning [157], it can also be available rank budget and results in limited expressiveness.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

While optimization theory for LoRA in full LLMs remains far from complete, it is already clear that off-the-shelf smooth optimization methods do not always translate cleanly selected parameterizations. In fact, even the one-layer testbed in (5) can exhibit an unfavorable landscape, which lacks the “nice” properties present in standard analyses of smooth optimization. Furthermore, LoRA introduces additional architectural structures, such as parametric symmetries (gauge invariances), that generic optimizers do not exploit. A common strategy for improving LoRA optimization is to first understand and explicitly characterize these overlooked properties on simplified testbeds; cf. (5). This serves as a principled sanity check: if an algorithm can hardly cope with a simplest setup, it is less likely to robustly carry over to large scale fine tuning. Let us first outline challenges and structural opportunities that one can leverage for efficient optimization. A. The simplest testbed problem Recall from Section III-D that fine-tuning a linear layer is intimately related to classical matrix sensing. To streamline the discussion, let us focus on the population form of (5) in the limit N → ∞. Under mild assumptions, the objective reduces to the matrix factorization problem 1 min n×r f (X, Y) := ∥A − XY⊤ ∥2F . (12) m×r 2 X∈R ,Y∈R

the loss landscape becomes even richer, exhibiting more intricate saddle structures and stronger curvature variability, which in turn poses remarkable optimization challenges. Despite these difficulties, (12) and LoRA fine-tuning share certain favorable topological features. For instance, (12) is known to admit no spurious local minima; i.e., all local minima are globally optimal. Likewise, LoRA has also been shown free of spurious local minima in the neural tangent kernel (NTK) regime [79], though little is known outside this regime. Architectural properties. An appealing optimization algorithm for LoRA (2), and its simplified variant (12), should not only cope with nonconvexity (e.g., by escaping saddles), but also maintain stability in the absence of global smoothness. In the prototype problem (12), we call an algorithm efficient if it reaches a global optimum within polynomial time. Classical theories, especially for local convergence, have been thoroughly developed; see the survey [27] and references therein. Beyond these well-studied aspects, there are also architecture-induced levers that can be explicitly exploited, as discussed next. Among the structural issues presented in the following subsections, the most salient one is the parametric symmetry, also known as gauge invariance. Both (2) and (12) exhibit this property in their parameterizations. Using Example 1 for illustration, points such as (x1 , y1 ) = ( 12 , 1) and (x2 , y2 ) = ( 18 , 4) are gauge-equivalent: they produce identical loss values, yet induce distinct local geometries, e.g., different Hessians (13). This discrepancy in curvature can lead to noticeably different optimization behaviors. For ease in exposition, we confine our discussion to an easy case, where the dimension r of X and Y is chosen identical to the target rank(A) := rA . In this setups, a zero training loss is achievable11 . For later use, the condition number of A is κ := σmax (A)/σmin (A). The layer index l is dropped hereafter for brevity.

Although the global optimum in (12) is characterized by the Eckart–Young–Mirsky theorem via a truncated SVD of A, the associated optimization landscape is non-trivial. Moreover, (12) already captures several structural features shared with LoRA fine-tuning that are underexploited by standard optimization approaches. The loss landscape. The bilinear parameterization XY⊤ introduces nonconvexity, and can violate standard regularity conditions presumed in smooth optimization analyses. Generally speaking, attractive global properties, including smoothness B. Initialization of LoRA or Polyak–Łojasiewicz (PL) inequality, no longer hold for f . The vanilla LoRA initializes X0 ∼ N (0, Θ(1/m)) and This can be readily illustrated by a simple example. Y0 = 0 for ∀l. This ensures Wl + X0 Y0⊤ = Wl , meaning that fine-tuning initiates exactly from the pre-trained weights. Example 1. Consider the simplest instance of (12), where While it is also possible to initialize X0 = 0 and Y0 ∼ m = n = r = 1, and A = 1; that is, N (0, Θ(1/r))12 , [61] shows that the vanilla LoRA initialization 1 2 enables larger learning rates, and leads to more stable feature f (x, y) = ∥xy − 1∥ . 2 learning in the regime of infinite network width. First, it is straightforward to see that (x, y) = (0, 0) is a But how does initialization affect optimization behavior? In saddle point, confirming nonconvexity. The presence of saddle classical smooth nonconvex optimization, initialization is anapoints precludes the global PL condition (gradient dominance) lyzed mainly through its distance to an optimum or a stationary ∥∇f ∥2 ≥ c (f − f ⋆ ) for global minimum f ∗ , and constant point: starting closer generally reduces the required iterations c > 0. To see that f is not globally smooth, check the Hessian for convergence [140]. However, because the objectives in (2) ï ò and (12) are nonconvex and not globally smooth, initialization y2 2xy − 1 Hf (x, y) = . (13) plays a critical role in a more complicated way than the simple 2xy − 1 x2 distance-to-solution. In particular, a class of “subspace aligned” Along the curve xy = 21 , the largest eigenvalue λmax (Hf ) = initializations have proven useful for both (12) and (2). max{x2 , y 2 }. Since x or y can be arbitrarily large, the For the one-layer adaptation in (12), recent work [106], curvature is unbounded, verifying that f is not globally smooth. [216] proves that the initialization scheme can have up to These unfavorable properties also persist in the LoRA fine11 In real-world fine-tuning, 0 training loss is often observed when applying tuning of LLMs. In particular, the induced objectives are LoRA to 7B models on e.g., the GLUE benchmark. nonconvex and lack global smoothness, with curvature varying 12 The default “fan-in” mode normalizes variance of neural network weights substantially across the parameter space. In higher dimensions, based on the number of input units (fan-in) to a layer.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

102 suboptimality

100 10 2 10 4 10 6 10 8

GD GD + Nystrom Init ScaledGD ScaledGD + Nystrom Init 40 60 80 100 iter

Fig. 8. Convergence of (12) with random Gaussian and Nyström initializations.

an exponential effect on the convergence rate with a given optimization algorithm. To be specific, a Nyström sketch [43], [190] is applied for initialization X0 = AΦ, Y0 = 0

(14)

Moving back to the LoRA fine-tuning, these theoretical insights suggest that an effective LoRA initialization should ideally capture the column and row spaces of the optimal update ∆W⋆ . However, the optimal subspace associated with ∆W⋆ is unknown a priori. Motivated by the empirical observation that well-performing updates ∆W often exhibit substantial subspace overlap with the pre-trained weights Wpre , a practical alternative is to use Wpre as a proxy to extract directional information for initialization [134], [17], [106]. Concretely, PiSSA [134] initializes X0 , Y0 using the top-r singular directions of W0 via truncated SVD, while OLoRA [17] utilizes a top-r QR decomposition for a similar goal. MiLoRA [196] instead leverages the bottom-r singular directions of W0 , and NoRA [106] employs a Nyström sketch to sample these directions. Finally, LoRA-GA [200] and LoRAOne [241] pursue subspace alignment through gradient-based initialization signals.

C. Optimization via asymmetric roles of X and Y

Another standard approach in SP for solving (12) (or (5)) where Φ ∈ Rr×r is a Gaussian random matrix. is alternating gradient descent (AltGD), sometimes discussed We first highlight a representative case, where initialization under the broader umbrella of alternating least squares (ALS). can have a pronounced impact. Consider the optimization algo- AltGD alternates between updating X and Y per iteration t t rithm ScaledGD [187], which is elaborated later in Section V-D. With step size η > 0, its iterates (indexed by t) are Xt+1 = Xt − ηx (Xt Yt⊤ − A)Yt (16a) Xt+1 = Xt − η(Xt Yt⊤ − A)Yt (Yt⊤ Yt )−1

−1 Yt+1 = Yt − η(Xt Yt⊤ − A)⊤ Xt (X⊤ . t Xt )

(15a)

(15b)

Theorem 4 (Impact of initialization for ScaledGD [80], [106], informal). With random initialization X0 ∼ N (0, σx2 ) and Y0 ∼ N (0, σy2 ) for some sufficiently small constants σx and σy , ScaledGD with proper step sizes applied to (12) requires O(log(1/ϵ)) iterations to reach global convergence f (Xt , Yt ) ≤ ϵ. However, ScaledGD under Nystrom initialization (14) achieves the same error in O(log log(1/ϵ)) iterations. Similarly, [216] establishes benefits of Nyström initialization for vanilla gradient descent (GD), which takes the form

Yt+1 = Yt − ηy (Xt+1 Yt⊤ − A)⊤ Xt+1

(16b)

where ηx > 0 and ηy > 0 are step sizes. Note that the gradients for Xt and Yt are evaluated at different points. For (12), it is known that such method converges linearly. Theorem 6 (Convergence of AltGD [203], informal). Consider AltGD in (16) initialized with X0 = AΦ and Y0 = Ψ, where Φ and Ψ are Gaussian random matrices of proper sizes. With proper step sizes, AltGD converges to global minima f (Xt , Yt ) ≤ ϵ in O(κ2 log(1/ϵ)) iterations.

Compared to GD with Nyström initialization in Theorem 5, AltGD achieves a comparable iteration complexity even when Xt+1 = Xt − η(Xt Yt⊤ − A)Yt the initialization only captures the column space of A (rather than both its column and row spaces). This suggests a Yt+1 = Yt − η(Xt Yt⊤ − A)⊤ Xt . practical advantage by treating the two factors asymmetrically. Theorem 5 (Impact of initialization for GD [222], [216], This asymmetry has been extended to ScaledGD (15). In informal). With random initialization X0 ∼ N (0, σx2 ) and particular, asymmetric ScaledGD variants [80], [118] show Y0 ∼ N (0, σy2 ) for some sufficiently small constants σx that updates (16) enable larger step size for faster convergence. and σy , GD with proper step sizes applied to (12) reBecause (16) requires two backpropagation passes per quires O(κ4 log(1/ϵ)) iterations to reach global convergence iteration, AltGD and related alternating schemes are not f (Xt , Yt ) ≤ ϵ. However, GD under Nystrom initialization (14) yet widely adopted in practical LoRA fine-tuning, while the achieves the same error in O(κ2 log(1/ϵ)) iterations. underlying principle of exploiting the asymmetric roles of X A numerical comparison is provided in Figure 8. The proofs and Y has been broadly recognized. For example, LoRA+ [62] are more involved, and we refer readers to the cited papers advocates for assigning different learning rates to X and for details. The intuition behind acceleration is that the sketch Y to reflect their asymmetric roles under standard LoRA initialization X0 = AΦ captures explicitly the column space initialization, which can improve stability and facilitate feature (and implicitly the row space) of A. Initializing the optimization learning. More broadly, several works explicitly study this with the optimal subspace, the algorithm bypasses the initial asymmetry. For instance, [245] shows that, under suitable search phase for subspace alignment, thereby substantially assumptions, fixing a random X and optimizing only Y can accelerating convergence. lead to improved generalization guarantees.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

point in the W space, and {(x, y)|xy = 2} as another. So from the perspective of W, the whole gauge orbit collapses to a single point, and the ambiguity disappears. However, explicitly forming and updating W is prohibitively expensive in the settings of interest. This can be overcome by exploiting more advanced optimization techniques under the mild assumption that W belongs to the fixed-rank manifold Mm×n := {W ∈ Rm×n | rank(W) = r} whichadmits r the quotient representation Mr := Rm×r × Rn×r GL(r). ∗ ∗ Here, / denotes the quotient under the natural group of GL(r) and Rm×r and Rn×r denote matrices with full column rank. ∗ ∗ The latter expression indicates that rather than optimizing over the full matrix W, we can continue working with the Fig. 9. Minimizing f (x, y) = 12 (xy − 1)2 . The manifold shown on the factorized variables (X, Y) ∈ Rm×r × Rn×r , provided that ∗ ∗ right is for visualization purposes only. the optimization procedure handles the quotient structure appropriately [3], [137], [14]. This yields a memory-efficient update that is, in essence, equivalent to optimizing over W D. Gauge-invariant optimization itself. A graphical illustration of this idea is shown in Figure 9. Standard optimization methods often ignore the “invariance” A useful preliminary observation is that Rm×r (and similarly ∗ inherent in LoRA parameterization. Consider a bilinear pair n×r R∗ ) is itself a manifold, namely the fixed-rank manifold (X, Y). The training objective depends on (X, Y) solely Mm×r . Although optimization over X and Y naturally invites r through their outer product XY⊤ , which is invariant to a Riemannian treatment, this manifold is, fortunately, simple invertible linear transformations; that is, for any invertible enough that Riemannian gradient descent almost always reduces Q ∈ Rr×r , the reparameterization X̃ = XQ and Ỹ = YQ−⊤ to vanilla gradient descent [3], [14]. leaves the product unchanged: X̃Ỹ⊤ = XY⊤ . Thus, (X, Y) With this background, handling gauge invariance requires and (X̃, Ỹ) attain the same objective value. In group theory only two additional ingredients beyond standard GD. The parlance, this property is known as gauge invariance [137]. first is to define a gauge-invariant metric, so that equivalent Crucially, while the objective value is invariant, the local points (XQ, YQ−⊤ ), ∀Q ∈ GL(r) share the same geometric geometry of the loss landscape at gauge-equivalent points can differ significantly. To see this, revisit Example 1 with structure in their tangent spaces. This metric induces the f (x, y) = 12 (xy − 1)2 , and compare two equivalent parameter- associated Riemannian gradient gradf that is used to perform izations (x, y) = (2, 2) and (x̃, ỹ) = (1, 4). It can be seen that update. The second, which can be optional, is to decompose gradf into vertical and horizontal components. The vertical the Hessian at these two points are component describes motion within an equivalence class, while ï ò ï ò 4 3 16 3 Hf (2, 2) = ; Hf (1, 4) = . (17) the horizontal component captures motion that genuinely 3 4 3 1 changes the underlying point on the quotient. For instance, The largest eigenvalues of these Hessians are different (7 versus in Example 1, vertical motion corresponds to moving along ∼ 19), indicating a discrepancy of local smoothness. Beyond the curve xy = 4. Projecting away the vertical component loss landscapes, the optimization behaviors can also differ can improve optimization efficiency, as it avoids updates along redundant move along the gauge orbits. considerably along the gauge orbit. Next, we make these ideas concrete by presenting several Example 2. After one GD update with stepsize η, initializing methods that fall under this unified framework. at (x, y) = (2, 2) yields (x+ , y + ) = (2 − 6η, 2 − 6η), whereas ScaledGD. Scaled gradient descent (ScaledGD) is among initialization (x̃, ỹ) = (1, 4) gives (x̃+ , ỹ + ) = (1−12η, 4−3η). the most widely used approaches for matrix sensing and Since x̃+ ỹ + ̸= x+ y + , these gauge-equivalent points can result completion [187], [234], [214]. While it admits several inin different objective changes and optimization trajectories. terpretations, we focus exclusively on its formulation as Given that gauge-equivalent points can exhibit markedly Riemannian optimization equipped with a specific gaugedisparate local geometries and optimization behaviors, it is invariant Riemannian metric. To be specific, for two tangent × Rn×r , natural to ask whether it is possible to systematically leverage vectors (ξX , ξY ) and (φX , φY ) at (X, Y) ∈ Rm×r ∗ ∗ the metric is defined as these differences for faster convergence. A general framework. The most direct way to avoid gauge symmetry is to abandon the factorized variables X and Y and g(X,Y) (ξX , ξY ), (φX , φY ) ⊤ instead optimize over the product W = XY . This is because := ⟨ξX (Y⊤ Y), φX ⟩F + ⟨ξY (X⊤ X), φY ⟩F . all factor pairs related by the transformation (XQ, YQ−⊤ ), with Q ∈ GL(r), produce exactly the same W. Note that GL(r) is the general linear group of degree r, i.e., the set of We refer readers to Appendix E for a detailed derivation of all r × r invertible matrices. In the scalar setting of Example 1, the gauge-invariance of this metric. Under this metric, the the entire hyperbola {(x, y)|xy = 4} can be viewed as one Riemannian gradients at (X, Y), also derived in detail in

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

Appendix E, are gradX f = ∇X f (X, Y)(Y⊤ Y)−1 ,

gradY f = ∇Y f (X, Y)(X⊤ X)−1 .

ScaledGD then updates (X, Y) by taking a step along the negative Riemannian gradient, i.e., Xt+1 = Xt − ηgradX f (Xt , Yt ),

Yt+1 = Yt − ηgradY f (Xt , Yt ). Next, we illustrate how ScaledGD copes with gauge invariance through Example 1. Example 3. Consider applying ScaledGD to the loss in Example 1. For any (x, y) satisfying xy = 4, the one-step ScaledGD update (x+ , y + ) renders 9 x y = 4 − 6η + η 2 . 4 + +

This confirms that, under ScaledGD, all gauge-equivalent points are updated coherently to a new equivalence class after one iteration, independent of the chosen factorization.

Fig. 10. Convergence comparison of LoRA [71], ScaledGD [187], and RefLoRA [238] for low-rank matrix factorization (12).

where (ξX , ξY ) and (φX , φY ) are tangential to the manifold at (X, Y). With the Riemannian gradient derived in Appendix F, RefLoRA updates are given by Xt+1 = Xt − η∇X f (Xt , Yt )S−1 t , Yt+1 = Yt − η∇Y f (Xt , Yt )St .

(19a) (19b)

The main benefit of this metric is that it eliminates the vertical It is worth noting that ScaledGD does not decompose the component in a computationally efficient manner, and the Riemannian gradient into vertical and horizontal components, Riemannian gradient thus lies entirely in the horizontal space; and therefore its update does not fully eliminate potential see Appendix F for a proof. As a consequence, RefLoRA not drift along gauge directions. While explicitly projecting onto only enjoys efficient update equivalence across gauge-related the horizontal space eliminates this drift, it typically requires parameterizations, but also yields a steeper decrease in the loss; solving a Sylvester equation [137], which is computationally see numerical tests in Figure 9, and the following example. expensive. Nevertheless, ScaledGD already proves highly effective for the one-layer LoRA problem (12). In particular, the Example 4. To apply RefLoRA to the loss in Example 1, for convergence rate in Theorem 4 does depend on the condition any (x, y) satisfying xy = 4, the one-step RefLoRA update number κ, which accounts for its popularity in ill-conditioned (x+ , y + ) renders matrix factorization and sensing tasks. x+ y + = (2 − 6η)2 . In the context of LoRA fine-tuning, ScaledGD-style gaugeaware optimization has been adopted in [232], where it is Similar to ScaledGD in Example 3, RefLoRA maps all gaugefurther shown to promote stable feature learning. LoRA- equivalent points with xy = 4 to the same equivalence class Pro [202] advances this approach by explicitly solving a after one step. Notwithstanding, when using the same (typically Sylvester equation to project the Riemannian gradient onto the small) step size η as ScaledGD, RefLoRA yields a noticeably horizontal space, which can speed up convergence at the cost larger decrease in the objective. Letting for instance η = 1/3, of extra per-iteration computations. ScaledGD gives f (x+ , y + ) = 2.25, while RefLoRA renders RefLoRA. Another method leveraging above viewpoint f (x+ , y + ) = 0. to tackle gauge invariance is refactored low-rank adaptation In fact, RefLoRA admits an alternative interpretation: among (RefLoRA) [238]. In contrast to ScaledGD, it relies on a all pairs in the equivalence class [(X, Y)] (i.e., all factorizations distinct gauge-invariant Riemannian metric to directly handle representing the same product XY⊤ ), it selects a particular one m×r n×r the gauge drift. For a point (X, Y) ∈ R∗ × R∗ , define their Gram matrices as PX := X⊤ X and PY := Y⊤ Y. such that one GD step from this point yields the steepest descent in the objective value [238]. This optimal pair X̃ = XS1/2 RefLoRA constructs a positive definite matrix and Ỹ = YS−1/2 is “balanced” in terms of their Gram 1/2 matrices X̃⊤ X̃ = Ỹ⊤ Ỹ. Furthermore, performing Riemannian −1/2 1/2 1/2 −1/2 S := PX PX PY PX PX optimization (19) with the gauge-invariant metric (18) is equivalent to standard GD on this balanced pair (X̃, Ỹ) in Euclidean space. In Example 4, this corresponds to performing which is also known as the (matrix) geometric mean of P−1 X GD along the trajectory of x = y. and PY . Using S, the gauge-invariant metric is defined as When applied to LLM fine-tuning, RefLoRA bypasses the g(X,Y) (ξX , ξY ), (φX , φY ) need for solving the Sylvester equation required by LoRA(18) Pro [202], and thus achieves faster per-iteration runtime. To := ⟨ξX S, φX ⟩F + ⟨ξY S−1 , φY ⟩F

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

grasp its efficiency, recall that the Gram matrices PX and Under this alternating update between “direction” (Xt , Yt ) PY are of size r × r, and hence computing S incurs merely and “magnitude” Θt , [114] shows that the required iterations O(m + n + r)r overhead, which is much smaller than the for convergence scales inversely with the rank r. O(m2 r) cost of solving a Sylvester equation. Theorem 7 ([114], informal). If m > n and r > rA , the Note that, although the above approaches are highly effective updates (21) for (20) converges to 12 ∥XT ΘT YT⊤ − A∥2F ≤ ϵ in practice, they incur additional computational cost, typically 2 4 r 4 rA κ r 2 rA κ4 1 requiring at least an inversion of a small r × r matrix. To shift within O (r−rA )8 + (r−rA )4 log ϵ iterations. the tradeoff toward greater computational efficiency, one may Recent work by [205] further shows that, in the matrixinstead account for only a subset of the full gauge invariance. sensing setting of (5), increasing the rank r not only accelerates A classical choice is scale invariance, corresponding to the optimization, but can also reduce sample complexity: the equivalence class {(cX, Y/c) | c ∈ R∗ }. Under this weaker number of measurements required to recover A decreases symmetry, the matrix St in the RefLoRA update (19) can be proportionally to 1/r. In practical LLM fine-tuning however, t ∥F replaced by ∥Y PoLAR does not strictly follow the alternating scheme in (21), ∥Xt ∥F I, yielding a cheaper update [238]. There are also other approaches that explicitly target and it replaces explicit retractions with retraction-free updates, gauge invariance in LoRA optimization. For example, LoRA- e.g., the landing family of methods [1], [46], [159], to improve RITE [224] factors out the “magnitude” components of Xt GPU efficiency. Nonetheless, Theorem 7 offers a plausible and Yt via polar decomposition, thereby focusing updates on explanation for why PoLAR tends to benefit from larger r. the subspace directions. ScaLoRA [240] copes with gauge Apart from PoLAR, the optimization of many other matrixinvariance under the HiRA [73] setting. Moreover, [105] based parameterizations (e.g., FedPara or KronA) is still shows that for scale-invariant two-factor problems (including not well understood. These architectures can introduce extra rank-1 LoRA), sharpness-aware minimization (SAM) [42], symmetries in addition to the standard gauge invariances, which [104] implicitly reduces the imbalance ∥X|2F − ∥Y∥2F . While may greatly affect optimization dynamics. We conjecture that this does not eliminate the full GL(r) gauge invariance, it explicitly analyzing these architectural properties holds strong suggests that part of SAM’s benefit may stem from implicitly potential for further improving downstream performance. regularizing a subset of gauge degrees of freedom. E. Optimization of SVD-type parameterization Previous subsections focused on advanced optimization schemes for the standard bilinear (BM) parameterization used in vanilla LoRA. We now turn to optimization schemes tailored to other architectures discussed in Section IV. This section focuses on SVD-type parameterizations, using PoLAR [114] as a representative example. Applying this parameterization to the testbed problem (12) yields 1 min ∥UΘV⊤ − A∥2F (20) U,V,Θ 2 s.t. U ∈ St(m, r), V ∈ St(n, r), Θ ∈ Rr×r

where St(m, r) := {U ∈ Rm×r | U⊤ U = Ir } stands for the Stiefel-manifold; cf. Section IV-A. Problem (20) also offers an additional explanation for leaving Θ unconstrained. Indeed, forcing Θ to be diagonal can slow down convergence, because it introduces non-strict saddles, whose Hessian has both 0 and negative eigenvalues, making them difficult to escape for first-order optimization algorithms [102]. As far as convergence, rather than working with quotient manifold as in the previous subsection, [114] views the constraint sets as embedded submanifolds of Euclidean space, and applies retraction-based Riemannian optimization. Let Et and Ft denote the Riemannian gradients with respect to Xt and Yt , respectively. Using the polar retraction, the Riemannian gradient descent (RGD) updates take the form −1/2 Xt+1 = (Xt − ηEt )(Ir + η 2 E⊤ t Et )

−1/2 Yt+1 = (Yt − ηFt )(Ir + η F⊤ . t Ft ) 2

(21a) (21b)

In addition, Θ is updated by a standard GD step Θt = Θt−1 − γ∇Θ f (Xt+1 , Θt , Yt+1 ).

(21c)

F. Optimization for tensor-based parameterization In general, advanced optimization for tensor-based adapters remains a largely underexplored frontier. We highlight several promising directions and open questions below. The SP community has developed a rich algorithmic toolbox for (approximate) tensor decomposition. Many tensor problems are traditionally tackled with alternating schemes, most notably alternating least squares (ALS) [228], block coordinate descent (BCD) [160], and splitting algorithms such as alternating direction method of multipliers (ADMM) [52]. In contrast, tensorized-LoRA adapters have not yet fully embraced these paradigms, plausibly due to the prohibitive cost of performing multiple backpropagation passes per iteration at the scale of LLMs. How to adapt the analytical insights and practical strengths of these methods to modern fine-tuning pipelines remains an open challenge. Turning to gradient-based methods in SP, existing theoretical understanding of tensor factorizations is still limited relative to the matrix case. For CP factorization, [50] shows that a slightly modified gradient flow (GF, the continuous-time counterpart of GD) can solve certain orthogonally decomposable tensor problems. Regarding Tucker parameterizations, [37], [119] have developed ScaledGD variants that aim to handle the richer gauge invariance therein, while [211] leverages optimization over the Grassmann manifold to exploit the Tucker-type structure. As for TT, the local convergence of Riemannian gradient descent is established in [18]. At a global level, optimization with TT parameterizations is closely related to deep linear networks, and may therefore suffer from similar depth-related bottlenecks as the tensor order grows [166]. Recent progress [239] alleviates such bottlenecks in deep linear networks, and may thus also offer useful insights here. So far,

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

a limited number of these optimization tools have been adopted in practical tensorized adapters for fine-tuning LLMs, leaving substantial room for further innovation. Last but not least, a critical open direction is to understand whether the parameter savings from exploiting cross-layer structure (see Section IV-C) come with hidden costs. This concern originates from SP: many tensor analogues of standard matrix concepts, including spectral norm and singular values, are NP-hard to compute [64]. Additionally, tensor problems often exhibit statistical–computational tradeoffs. For example, in spiked tensor PCA [7], gradient-based methods can require a substantially higher signal-to-noise ratio (SNR) than the information-theoretic threshold. It remains unclear whether analogous optimization bottlenecks arise for tensorized adapters in LLM fine-tuning, or whether these classical concerns in SP carry over to LLM settings, considering that fine-tuning only aims for good suboptimal solutions rather than the exact one. G. Summary of efficient optimization

Fig. 11. A comparison of LoRA (left) and QLoRA (right).

LoRA accounting for quantization error. LLMs are typically pre-trained and fine-tuned in FP16 or BF16 numerical representations. However, serving models at these precisions may not be scalable in cloud settings, especially under millions of concurrent user requests. Moreover, autoregressive generation requires repeated forward passes, which can lead to substantial end-to-end latency. A standard remedy is quantization, which runs the model at a lower precision (e.g., FP8/INT8 or 4-bit) to reduce memory footprint and improve inference throughput. However, quantization error can degrade downstream performance, that is noticeable at 4-bit precision. For instance, applying GPTQ 4-bit quantization [44] to LLaMA7B leads to roughly a 1-point performance drop in the 5-shot MMLU benchmark [63]. LoRA has been adopted as a lightweight approach to compensate for quantization error. QLoRA [33] for example, enables 4-bit quantization (NF4) with minimal performance loss. During fine-tuning, the quantized base model Q(Wl ), and the LoRA factors {Xl , Yl }l are learned so that the low-rank updates jointly account for quantization error and adapt the model to the downstream data

While the preceding discussion focuses on strategies that can be translated into practical algorithms, a growing body of work aims to uncover the fundamental principles governing LoRA’s success and identify broader optimization trends. For instance, it is analyzed in [231] that for fully connected networks LoRA can be highly expressive when the rank r is sufficiently large. From an optimization and generalization perspective, [79] shows that when r grows with the square root of data size, LoRA in the NTK regime avoids spurious local minima, and enjoys favorable generalization guarantees. For complementary perspectives centered on implicit regmin f ({Q(Wl ) + Xl Yl⊤ }l ). (22) ularization of optimization dynamics of GD/GF, cf., (12) Xl ,Yl and (5) [38], [81], [206]. In this context, regularization refers Here Q(Wl ) denotes the frozen quantized pre-trained weights, to the biases induced by the optimization procedure itself, while Xl and Yl are trained and stored in FP16/BF16. even in the absence of any explicit regularizer. While the This pipeline offers two benefits: i) the LoRA updates can underlying mathematical analyses are often highly technical, compensate for quantization-induced errors; and ii) keeping the core challenge remains practical: how to reliably and the base model quantized alleviates the memory bottleneck at empirically exploit them in real-world fine-tuning regimes, fine-tuning; see Figure 11 for an illustration. including choosing architectures and optimizers to induce LoftQ [110] aims at quantization-aware fine-tuning by desirable implicit biases, remains an open question. constructing a quantized base model whose quantization error is Lastly, note that most existing optimization techniques are more amenable with a low-rank correction. Specifically, before developed for the BM (or vanilla LoRA) parameterization. fine-tuning, it solves the per-layer l optimization Theory and practical algorithms for alternative parameterizations, such as SVD-type, high-rank, as well as tensor-based ∥Wl − Q(Wl ) − Xl Yl⊤ ∥2F . (23) min Q(Wl ),Xl ,Yl parameterizations, remain comparatively underexplored. We expect substantial room for further progress in these directions. To handle the discrete nature of quantization, LoftQ alternates between updating the quantized weight Q(Wl ) and refitting the low-rank factors (Xl , Yl ). Letting Ql,t denote the quantized VI. B ROADER SCOPE OF L O RA Having discussed adapter models and tailored optimization model weight at iteration t, LoftQ update takes the form strategies, this section deals with LoRA thrusts beyond the standard supervised fine-tuning on downstream tasks.

⊤ Ql,t = Q(Wl − Xl,t−1 Yl,t−1 )

Xl,t , Yl,t ∈ arg min ∥Wl − Ql,t − Xl Yl⊤ ∥2F . X,Y

A. Broader fine-tuning with LoRA LoRA improves LLM performance by adapting a pre-trained model to learn a target data distribution for downstream tasks. By interpreting “downstream tasks” and “data distribution” broadly, it already reveals a richer ecosystem of applications.

The resulting {Q(Wl ), Xl , Yl } are then used to initialize QLoRA fine-tuning in (22). IR-QLoRA [147] improves quantized fine-tuning from a complementary angle. Rather than calibrating quantization by minimizing an ℓ2 reconstruction loss in (23), it quantizes a

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

pre-trained LLM by minimizing an information-based loss. It suggests that a suitably chosen low-rank network can match the further introduces an additional, largely parameter-free feature performance of a large dense model, LORO [138] replaces the path via pooling/averaging input features. Together, these m × n linear weight matrix Wl with a low-rank factorization components stabilize ultra-low-bit (< 4-bit) quantized fine- Xl Yl⊤ , and trains the factors directly through Riemannian tuning. QA-LoRA [215] targets quantized deployment. Naively optimization. It is observed that this low-rank model can match merging Q(W) + XY⊤ produces an FP16/BF16-weighted the performance of full-parameter pre-training for a LLaMA-1B model, which cannot be directly deployed in low-bit form. model, while providing memory and runtime savings. To address this, QA-LoRA advocates a pooling-based LoRA A remotely related line of research that leverages low-rank variant that enables low-bit (e.g., INT4) merging of the adapted structure for pre-training is GaLore [243]. Unlike low-rank weights while preserving accuracy. parameterizations, GaLore keeps the model weights intact, but LoRA for long-context adaptation. LLMs are often pre- maintains a low-rank representation of the training dynamics. trained with a relatively short context window, typically 2K Concretely, it projects gradients into a low-rank subspace, to 8K tokens. For example, LLaMA2-7B supports a 4K performs the optimizer update (including momentum and context length [189]. However, many applications, such as second-moment states) in that compressed space, and then long-document understanding, retrieval over large corpora, and lifts the resulting update back to the original parameter space. multi-turn agents, require context lengths far beyond the pre- By storing and updating the compressed gradients and optimizer training limit [9], [174]. Viewing this extension of context states, GaLore achieves substantial memory savings. length as a distribution shift, one can apply fine-tuning to mitigate performance degradation. While the core remedy C. Serving LLMs with LoRA involves efficient attention mechanisms, which go beyond the LoRA can also benefit LLM serving. When deploying a scope of this paper, LoRA has been used as a lightweight tool to reduce the error induced by context extension. Empirically, model for a single downstream task, the adapted weights Wl + LongLoRA [26] reports extending LLaMA2-7B from 4K to Xl Yl⊤ can be pre-merged into a single matrix. As a result, serving a single LoRA-adapted model incurs no additional 100K context with the aid of LoRA. inference latency, as opposed to other PEFT methods such as prefix tuning [109]. B. Pre-training LLMs with LoRA However, LoRA’s serving advantage is most highlighted While LoRA was originally developed for PEFT, recent work when managing multiple downstream tasks. Consider the multihas also utilized it for pre-training to lower hardware barriers. domain serving setup where the users are from N domains, e.g., Since LoRA parameterizes updates in a rank-r subspace, a coding, mathematics, and health. A straightforward solution central challenge is how to recover the high-rank weight updates would be fine-tuning N separate LLMs, one per domain, which for pre-training with low-rank matrices ∆Wl . requires storing N full models in memory for serving. This One idea is to combine low rank with sparsity to obtain can be prohibitive when N or the base model size grows. effectively higher-rank updates, in the spirit of the low-rank- In contrast, LoRA allows for storing a single shared base plus-sparse models discussed in Section IV-B. SLTrain and model alongside N lightweight adapters, each specialized to a LOST [56], [107] fix the sparsity pattern before training to particular domain. The total storage requirement boils down to avoid the cost of storing a full-sized dense matrix. While N Sa + Sb , where Sa and Sb are the sizes of the base model SLTrain samples the nonzero entries at random, LOST enforces and LoRA adapter; typically Sa ≪ Sb . Thus, LoRA enables a randomly chosen column-wise sparsity pattern, and inserts an multi-domain serving in a highly memory-efficient manner. activation function between the two LoRA factors to account Building upon this, several works have further improved the for nonlinearity. efficiency of multi-LoRA serving from various perspectives. Another research line learns a sequence of low-rank updates SLoRA [173] leverages paging techniques to optimize memory s s {∆W P ℓ }.sAlthough each Wℓ is low-rank, their accumulation access for faster inference. LoRA-Inlaid [213] combines LoRA with quantization to efficiently serve multiple adapters. More s ∆Wℓ can be a high-rank matrix. This idea is exploited in ReLoRA [113], which repeatedly applies LoRA, merges the up- recently, EdgeLoRA [171] demonstrates LoRA-based serving date into the base weights, and restarts a new low-rank adapter in edge scenarios with tight resource constraints. during pre-training. CoLA [212] further links this mechanism These benefits are not limited to BM-based LoRA parameto Frank Wolfe method or greedy subspace learning [77], [103], terizations. For an SVD-style adapter ∆Wl = Ul Σl Vl⊤ , one where training proceeds through a sequence of subspaces, and can merge Ul Σl into a single matrix, making it equivalent each LoRA stage can be viewed as approximately solving to serving with standard LoRA. For tensor-based parameterian FW subproblem. Notably, the idea of learning low-rank zations, efficient serving remains less explored. Nevertheless, structure in a successive, stage-wise manner also has deep roots in the worst case, one can always “unpack” the tensor along in SP, for example via deflation [50]. Related viewpoints also the layer dimension and obtain L per-layer low-rank updates. appear in incremental learning [111] and, in certain nonconvex Lastly, for low-rank-plus-sparsity parameterizations, while the settings, saddle-to-saddle dynamics [76], [206]. low-rank component can be efficiently deployed, the sparse There are also approaches that rely on more advanced component requires leveraging specialized hardware, such as optimization to learn purely low-rank parameterizations. Moti- NVIDIA’s sparse engine (with N : M sparsity), to accelerate vated by the low-rank lottery ticket hypothesis [162], which computation, which remains an open area of research.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

Another important application of LoRA is batched serving, where K users simultaneously request a shared base model but K different LoRA adapters ∆Wlk = Xkl (Ylk )⊤ for their respective tasks. Let the input to the l-th layer be zkl ∈ Rm . Batching concatenates inputs to from matrix ⊤ Zl := [z1l , . . . , zK l ] , so that a single general matrix multiply (GeMM) can process all users’ requests in parallel, avoiding K separate kernel launches and improving throughput by amortizing overhead. With standard LoRA, batching applies cleanly to the pre-trained weight Wl , but user-specific adapters cannot be fused into the same batched GeMM. Consequently, the forward pass becomes ⊤ K K ⊤ Zl Wl +row-stack (z1l )⊤ X1l (Yl1 )⊤ , . . . , (zK l ) Xl (Yl ) | {z } | {z } batching

cannot be fused

where row-stack[·] stacks K row vectors into a matrix. FastLoRA [207], which predates HiRA from a different motivation, addresses this issue by reparameterizing adaptation with (8) so that the forward pass becomes Zl Wl + (24) 1 ⊤ 1 1 ⊤ K ⊤ K K ⊤ (zl ) (Wl ◦ Xl (Yl ) ), . . . , (zl ) (Wl ◦ Xl (Yl ) ) . This form admits a GEMM-friendly batch implementation; see also Appendix G. The tradeoff is that FastLoRA incurs extra O(mnr) computations compared to standard LoRA, making it most advantageous in the large-K regime, where throughput gains from improved batching outweigh the extra arithmetic.

D. Reinforcement Learning with LoRA At a high level, LLM alignment refers to post-training procedures, such as supervised fine-tuning and preferencebased optimization, that steer a pretrained model toward behavior preferred by humans. Recent evidence suggests that LoRA is surprisingly competitive for the post-training and alignment of LLMs via reinforcement learning (RL). For instance, TiNA [199] demonstrates that applying LoRA-based RL to a 1.5B parameter base model yields substantial reasoning improvements at a small cost of approximately $9 USD. Similarly, [180] investigated LoRA-based proximal policy optimization (PPO) alignment for LLaMA-7B. It finds that, LoRA-based RL from human feedback (RLHF) can match or even slightly outperform full fine-tuning while drastically reducing memory consumption. Recently, [163] reports that LoRA performs comparably to full-parameter updates in RL settings even with minimal ranks. Through an informationtheoretic lens it is argued that RL updates may inherently require relatively low effective capacity, which can be readily provided by lightweight LoRA adapters. The preceding four subsections collectively demonstrate that LoRA is a versatile tool applicable throughout the entire lifecycle of an LLM, from initial pre-training to downstream fine-tuning, alignment, and deployment/serving. By offering memory efficiency at every stage of the pipeline, LoRA effectively democratizes large-model development and deployment by lowering hardware barriers.

Fig. 12. Mix of multiple (two here) concepts. Figure taken from ZipLoRA [165].

E. Concept injection and merging with LoRA While the methodologies discussed are broadly applicable, this subsection focuses specifically on text-to-image generalization for ease of visualization. The additive nature of LoRA is often interpreted as a mechanism for encoding “addon knowledge.” This paradigm has proven highly effective in personalized text-to-image generation, where a diffusion model is adapted to generate images of novel subjects, such as a user’s own dog, using only a handful of (typically ≤ 10) reference examples [155]. Since the pre-trained model lacks prior exposure to these specific instances, attaching and fine-tuning a LoRA module provide a compact, modular representation of the new “concept.” A natural application extending this interpretation of additive concept encoder, is to combine N separately trained LoRAs, say {Xnl , Yln }N n=1 , to enable joint generation of multiple learned concepts. A representative example is subject–style fusion, where individual LoRAs encoding subject identity and artistic style are “merged” to produce diverse stylized outputs; see Figure 12 from ZipLoRA [165]. For such multi-concept generation, the central challenge lies in how to combine them to achieve high-fidelity generation. Interestingly, a simple linear P composition, ∆Wl = n wln Xnl (Yln )⊤ , is often sufficient for high-quality results in many tasks [72], [57]. Although the composition is linear at the weight level, the cumulative effect remains nonlinear due to the network depth. The mixing coefficients wln are optimized in a layer-wise manner to handle potential semantic conflicts and context dependence of concepts. For instance, the word “football” may refer to soccer or American football depending on whether the desired context is European or American. These conflict-resolution strategies draw clear parallels to the

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

established literature on ensemble learning [170], [121], [122], Vl := Wlaligned − Wlunaligned . As an example, the aligned which also has a long-standing history in SP, as well as to multi- model could be LLaMA-3.1-Chat, and the unaligned model task learning [164]. For example, MoLE [209] adopts a mixture- is the corresponding base LLaMA-3.1. The associated safety of-experts (MoE) framework, employing a gating network to projection matrix is dynamically predict the mixture coefficients {wln }. AlternaCl = Vl (Vl⊤ Vl )−1 Vl⊤ . (25) tively, [54] determines {wln } by minimizing the discrepancy between outputs of the merged model and the individual adapters Given a LoRA update ∆Wl = Xl Y⊤ , Safe LoRA conl on a calibration set. Concretely, let ∆Wln = Xnl (Yln )⊤ , and structs the projected update ∆Ŵl := Cl ∆Wl . The original PN define the merged update ∆Wl (wl ) := n=1 wln ∆Wln and update ∆Wl is replaced by ∆Ŵl whenever the projected calibration dataset D. The weights are optimized via update is sufficiently misaligned with the original update; that is, ⟨∆Ŵl , ∆Wl ⟩ ≤ τ ∥∆Ŵl ∥F ∥∆Wl ∥F for a threshold τ . N X Subsequent work [108] replaces Cl with a data-dependent min Ex∈D ∥ Wl + ∆Wl (wl ) x − Wl + ∆Wn x∥2 . wl formulation. Despite these promising results, there remains n=1 Other work has investigated learning mixing coefficients via room for improvement in safety-aware PEFT. For example, contrastive objectives [177], token- or region-wise weighting establishing theoretical robustness guarantees would provide schemes [219], or auxiliary hypernetworks that generate the stronger evidence that such methods improve safety. LoRA for privacy-aware fine-tuning. It is shown in [152] weights on the fly [172]. that LoRA fine-tuning does not inherently prevent privacy leakage, and hence membership inference attacks remain F. Responsible fine-tuning with LoRA effective. To address this issue, differential privacy (DP)–based As LLMs continue to raise noticeable societal concerns, optimizers inject carefully calibrated noise into gradient updates, it is natural to ask whether LoRA fine-tuning mitigates or providing a principled way to limit such leakage. Notably, [226] exacerbates issues including safety, privacy, and bias. suggests that DP optimizers such as DP-SGD can offer LoRA for uncertainty quantification. Uncertainty quan- better privacy–utility tradeoffs when applied to PEFT than tification is indispensable for safety-critical scenarios such as full-parameter update. Building on this, [181] observes that AI-assisted medical diagnosis and legal reasoning. While LLMs LoRA’s bilinear parameterization can cause DP noise to interact often exhibit reasonable calibration during pre-training [86], destructively with the two-factor updates. To mitigate this meaning that the model’s stated confidence roughly matches interference, the Xl factors are frozen per layer, and only the its actual correctness frequency, fine-tuning on domain-specific remaining components are trained. Together, these observations tasks often makes the model over-confident, especially when suggest that gauge invariance discussed in Section V-D may be the dataset [82], [186] is small. This motivates the pursuit of a natural complement to private LoRA, potentially improving Bayesian approaches that offer model uncertainty quantification. both training stability and model utility. Laplace-LoRA [217] first performs standard LoRA fine-tuning to obtain a maximum a posteriori (MAP) estimate the LoRA parameters, and then uses a Laplace approximation to the poste- G. Applications to multi-modal LLMs Multi-modal foundation models often recruit an LLM as the rior to assess uncertainty. For scalability, LoRA is viewed as a two-layer linear network, allowing Kronecker-factored (KFAC) backbone, mapping vision or audio inputs into “soft prompts” block approximation for the Fisher information matrix [132]. or prefix-like embeddings via an encoder plus a learnable An additional low-rank approximation is introduced to keep projection module. LoRA can be applied to jointly adapt the KFAC matrices tractable. BLoB [201] performs Bayesian the modality encoders and the LLM to improve cross-modal adaptation by jointly learning the mean and covariance of a low- alignment. For instance, [23] fine-tunes video encoders in rank variational distribution via backpropagation. By treating tandem with the LLM to improve representation parity between Xl as a variational random variable, it optimizes the resulting the two components; see Figure 13 for an example. There are several design choices for fine-tuning the backbone variational objective end-to-end. In addition, ScalaBL [158] and C-LoRA [151] shift the Bayesian/variational treatment to LLM. A straightforward strategy is to treat tokens from a much smaller set of parameters. In particular, they leverage different modalities uniformly, and apply LoRA in the standard SVD-type models, and only follow a Bayesian approach for modality-agnostic way [23]. Conversely, Phi-4-multi-modal [2] the singular value Σl ∈ Rr×r , to reduce variational overhead, leverages modality-specific LoRA modules coupled with dedicated routing mechanisms. This allows the model to switch and thus enhance scalability. LoRA for safety-aware fine-tuning. There are two pri- between different modes, such as text-only, vision-plus-text, mary directions along this line of research. One utilizes or audio-plus-text, while minimizing the burdensome crossLoRA to probe the fragility of alignment mechanisms. For modal interference. MoKA [204] proposes a hybrid approach example, [100] shows that adversarial LoRA or QLoRA that jointly models unimodal and cross-modal adaptation. It fine-tuning can substantially erode the refusal behavior of assigns each modality a unique LoRA factor {Xm l }m , mixes aligned chat models. The other direction leverages LoRA for modality-specific features via cross-attention, and subsequently safety fine-tuning. Safe LoRA [70] presumes that alignment applies a shared Yl factor across modalities; see Figure 13. fine-tuning identifies “safety directions” in weight space. It It remains unclear whether fully shared, modality-specific, or constrains LoRA weights to stay within those directions hybrid parameterizations offer the desired tradeoff in practice.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

Fig. 13. LoRA for fine-tuning multi-modal models.

The answer may depend on the modality mix, data regime, and target use case, leaving substantial room for further research towards principled multi-modal adapter designs. H. Additional directions As LoRA has gained widespread popularity and vast collections of LoRA checkpoints have become publicly available [244] (e.g., on HuggingFace), LoRA weights are increasingly being viewed as a new form of “data.” Inspired by generative modeling of images, recent work explores metamodels capable of directly generating LoRA weights. Thanks to the relatively low parameter count of LoRA adapters, several studies have employed diffusion-based generators to synthesize LoRA updates [210], [167]. Further, [146] incorporates the gauge invariance inherent in LoRA’s bilinear parameterization model and generation procedure. While these approaches are intriguing, they have so far been studied for task-specific personalization, and have not yet been demonstrated at larger scales of general-purpose foundation models. The generated adapters typically underperform those obtained via direct finetuning. Nevertheless, these preliminary results are encouraging: they achieve non-trivial performance across several settings and point toward toward faster, more reusable, and potentially “amortized” model adaptation. VII. R ESEARCH O UTLOOK The adapter architectures, optimization strategies, and broader scopes reviewed thus far indicate that substantial opportunities for research remain. While specific challenges have been noted throughout this survey, we wrap up by highlighting key areas where synergies between the SP and deep learning (DL) communities could lead to impactful advances. A. From SP to DL The SP community has developed a rich toolkit for leveraging low-rank structure. A number of SP tools can have high potential in the context of LLM fine-tuning too.

SP-transferred architectures. Architectures with provable efficiency in classical SP settings may offer significant merits when transferred to LLM adaptation. One example we have already seen is the SVD-type parameterizations discussed earlier in Section IV-A. While they are equally expressive as the BM factorization, SVD parameterizations can exhibit more favorable optimization behavior on matrix sensing (5); that is, one-layer LoRA. Moreover, several alternative parameterizations that are well-known in SP deserve exploration. For instance, [126] shows that in the rank-one case, a tensorized lift of (5) can lead to a more benign loss function landscape. In regimes where the restricted isometry property (RIP) fails and BM exhibits spurious local minima, the lifted formulation can turn these spurious minima into strict saddles. In the tensor regime, hierarchical Tucker decomposition [55] is also known to improve the efficiency of standard Tucker by reducing the parameters in the core. These examples raise an open question: do these architectures with established benefits in classical SP retain their advantages in the LLM scale? SP-inspired optimization. Alternating schemes such as ALS and splitting methods including ADMM have been successful in SP, yet they have seen limited adoption in LoRA finetuning. The major bottleneck may be that they require multiple backward passes at LLM scale. While a few recent works have begun to revisit such ideas in deployment-oriented settings [125], translating them into memory-efficient fine-tuning remains a challenge. Another pressing direction is to better understand optimization for tensor-based parameterizations. Although tensorized adapters can be more compact than matrixbased ones, it is less clear whether they introduce implicit costs. Tensor methods in SP are known to have computationalversus-statistical efficiency tradeoffs [37]. Whether analogous fundamental tradeoffs emerge with tensorized adapters in LLM fine-tuning remains a largely uncharted territory. Understanding fine-tuning dynamics. Understanding learning dynamics can reveal additional levers for efficient and reliable fine-tuning, and several SP tools may be particularly

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

useful in this regard. For example, it is well known that nuclearnorm regularization admits an equivalent BM variational form with Frobenius penalties [153], [79] through 1 ∥∆W∥∗ = min ∥X∥2F + ∥Y∥2F . XY ⊤ =∆W 2

yield simpler and more efficient algorithms, with examples already discussed in Section V-D. The rich factorization and recovery tasks in SP also exhibit rich—symmetry groups that the optimizer must implicitly navigate. Interestingly, for several SP problems even within the same gauge-equivalence class, not all parameterizations are “equally good” from an optimization standpoint. For instance, [48] demonstrates that balanced factorizations ∥X∥F ≈ ∥Y∥F enjoy more benign landscape properties, such as the absence of spurious local minima under suitable conditions. Moreover, flatter minima have been linked to better generalization in matrix recovery [34]. These insights motivate designing algorithms that explicitly respect the underlying symmetries and, among equivalent parameterizations, preferentially steer iterates toward wellconditioned representatives that offer favorable behaviors.

This relationship suggests that weight decay on LoRA factors implicitly biases the induced update ∆W toward a smaller nuclear norm. This offers a lens for interpreting why weight decay can stabilize training or improve generalization in practice. It remains to be seen how far this analogy carries beyond BM-style factorizations, e.g., to SVD-type and tensorized parameterizations, and whether comparable variational characterizations exist to explain their implicit biases. Sparsity with memory-efficient parameterization? Lowrank plus sparsity is a foundational modeling recipe in SP. However, current “sparse” LoRA variants typically fix the VIII. C ONCLUSION sparsity pattern in advance, as learning the sparse support endIn this survey, a comprehensive overview of low-rank to-end often necessitates applying an ℓ1 regularizer on a full matrix Sl ∈ Rm×n , which is difficult at LLM scale to reconcile adaptation (LoRA) for large models is provided to connect it with strict memory budgets. A key open problem is how to with well-established SP tools. By reframing LoRA as a modern make the sparsity pattern learnable without ever materializing rendition of classical low-rank models, this survey highlighted how architectural choices and optimization algorithms drive a dense Sl ∈ Rm×n . Broader scopes. Beyond optimization and parameterizations, the success of LoRA. Our discussion has emphasized that the information theory and coding theory can provide insightful relationship between SP and LoRA is not an one-way transfer guidance for quantization and compression, leading to more of knowledge. While SP offers a useful toolbox and rigorous principled designs. Moreover, subspace methods such as CCA analysis to improve and justify fine-tuning, the scale and and ICA could offer a fresh perspective on multi-view and complexity of foundation models introduce novel challenges multi-modal adaptation. For example, CCA could be used that push the boundaries of classical SP. As the field moves toward more sophisticated practical to learn low-dimensional shared representations, while ICA settings, the synergy between these two disciplines will be could identify disentangled, modality-specific latent factors to vital. Future research should not only continue bringing SP minimize cross-modal interference during joint fine-tuning. insights to bear on large-scale model adaptation, but also leverage the innovations developed for LoRA to inspire novel B. From DL back to SP approaches to fundamental SP problems, such as matrix sensing The innovations currently driving LLM fine-tuning are, and tensor completion. Ultimately, this cross-fertilization will in turn, proving useful for SP. Recent architectural and yield a more principled and efficient framework for artificial optimization paradigms, though motivated by LLM adaptation, general intelligence. are theoretically validated on classical SP testbeds such as matrix sensing and matrix completion. As a result, a number R EFERENCES of these parameterizations and algorithmic principles can be [1] P. Ablin and G. Peyré, “Fast and accurate optimization on the orthogonal transferred back to traditional low-rank recovery problems manifold without retraction,” in Proc. Int. Conf. on Artificial Intelligence with minimal modification. A salient example is the study and Statistics, Jan. 2022. of Hadamard decomposition [28], which is closely related to [2] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen et al., “Phi-4-mini technical FedPara [74] and HiRA [73]. The goal is to approximate a report: Compact yet powerful multimodal language models via mixturematrix as the Hadamard product of low-rank matrices min

X1 ,X2 ,Y1 ,Y2

∥X1 Y1⊤ ◦ X2 Y2⊤ − W∥2F .

This parameterization can be more compact than the standard BM factorization. Gradient-based solvers for this problem are developed in [28], [208]. Notably, the resulting optimization inherits additional symmetries beyond the usual factorization invariance within per-pair (Xi , Yi ), i = {1, 2}, and across Hadamard product, which further enriches the geometry and poses new challenges and opportunities for algorithm design. Another compelling direction involves explicitly accounting for symmetries in classical SP problems. Recent work in DL has highlighted that making such symmetries explicit can

of-loras,” arXiv preprint arXiv:2503.01743, 2025. [3] P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008. [4] A. Anandkumar, R. Ge, D. J. Hsu, S. M. Kakade, M. Telgarsky et al., “Tensor decompositions for learning latent variable models.” J. Mach. Learn. Res., vol. 15, no. 1, pp. 2773–2832, 2014. [5] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in Proc. Int. Conf. on Machine Learn., 2013. [6] S. Arora, N. Cohen, N. Golowich, and W. Hu, “A convergence analysis of gradient descent for deep linear neural networks,” in Proc. Int. Conf. on Learn. Representations, 2019. [7] G. B. Arous, S. Mei, A. Montanari, and M. Nica, “The landscape of the spiked tensor model,” Comm. on Pure and Applied Mathematics, vol. 72, no. 11, pp. 2282–2330, 2019. [8] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

[9] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou et al., “Longbench: A bilingual, multitask benchmark for long context understanding,” in Proc. Conf. Assoc. Comput. Linguist. Meet., 2024, pp. 3119–3137. [10] B. Baingana and G. B. Giannakis, “Embedding graphs under centrality constraints for network visualization,” arXiv preprint arXiv:1401.4408, 2014. [11] D. Bershatsky, D. Cherniuk, T. Daulbaev, A. Mikhalev, and I. Oseledets, “LoTR: Low tensor rank weight adaptation,” arXiv preprint arXiv:2402.01376, 2024. [12] K. Bhardwaj, N. P. Pandey, S. Priyadarshi, V. Ganapathy, M. Nagel, R. Esteves, S. Kadambi, S. Borse, P. Whatmough, R. Garrepalli, M. V. Baalen, and H. Teague, “Rapid switching and multi-adapter fusion via sparse high rank adapters,” in ICML Workshop on Foundation Models in the Wild, 2024. [13] M. Bini, K. Roth, Z. Akata, and A. Khoreva, “Ether: Efficient finetuning of large-scale models with hyperplane reflections,” arXiv preprint arXiv:2405.20271, 2024. [14] N. Boumal, An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, 2023. [15] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. HerbertVoss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Proc. Neural Info. Processing Syst., vol. 33, 2020, pp. 1877–1901. [16] S. Burer and R. D. Monteiro, “A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization,” Mathematical Programming, vol. 95, no. 2, pp. 329–357, 2003. [17] K. Büyükakyüz, “OLoRA: Orthonormal low-rank adaptation of large language models,” arXiv:2406.01775, 2024. [18] J.-F. Cai, J. Li, and D. Xia, “Provable tensor-train format tensor completion by riemannian optimization,” Journal of Machine Learning Research, vol. 23, no. 123, pp. 1–77, 2022. [19] E. J. Candès, X. Li, Y. Ma, and J. Wright, “Robust principal component analysis?” Journal of the ACM, vol. 58, no. 3, pp. 1–37, 2011. [20] ——, “Robust principal component analysis?” Journal of the ACM (JACM), vol. 58, no. 3, pp. 1–37, 2011. [21] E. J. Candes, T. Strohmer, and V. Voroninski, “Phaselift: Exact and stable signal recovery from magnitude measurements via convex programming,” Comm. on Pure and Applied Mathematics, vol. 66, no. 8, pp. 1241–1274, 2013. [22] E. J. Candes and T. Tao, “Near-optimal signal recovery from random projections: Universal encoding strategies?” IEEE Trans. Inf. Theory, vol. 52, no. 12, pp. 5406–5425, 2006. [23] U. Cappellazzo, M. Kim, H. Chen, P. Ma, S. Petridis, D. Falavigna, A. Brutti, and M. Pantic, “Large language models are strong audiovisual speech recognition learners,” in Proc. IEEE Int. Conf. Acoust., Speech, Sig. Process., 2025, pp. 1–5. [24] J. Chen, G. Wang, and G. B. Giannakis, “Graph multiview canonical correlation analysis,” IEEE Trans. Signal Processing, vol. 67, no. 11, pp. 2826–2838, 2019. [25] Y. Chen and M. J. Wainwright, “Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees,” arXiv preprint arXiv:1509.03025, 2015. [26] Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “LongLoRA: Efficient fine-tuning of long-context large language models,” in Proc. Int. Conf. on Learn. Representations, 2024. [27] Y. Chi, Y. M. Lu, and Y. Chen, “Nonconvex optimization meets lowrank matrix factorization: An overview,” IEEE Trans. Signal Processing, vol. 67, no. 20, pp. 5239–5269, 2019. [28] M. Ciaperoni, A. Gionis, and H. Mannila, “The hadamard decomposition problem,” Data Mining and Knowledge Discovery, vol. 38, no. 4, pp. 2306–2347, 2024. [29] A. Cichocki, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Caiafa, and H. A. Phan, “Tensor decompositions for signal processing applications: From two-way to multiway component analysis,” IEEE Sig. Process. Mag., vol. 32, no. 2, pp. 145–163, 2015. [30] P. Comon, “Independent component analysis, a new concept?” Signal processing, vol. 36, no. 3, pp. 287–314, 1994. [31] R. Csordás, C. D. Manning, and C. Potts, “Do language models use their depth efficiently?” in Proc. Neural Info. Processing Syst., 2025. [32] L. De Lathauwer, B. De Moor, and J. Vandewalle, “A multilinear singular value decomposition,” SIAM journal on Matrix Analysis and Applications, vol. 21, no. 4, pp. 1253–1278, 2000.

[33] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” in Proc. Neural Info. Processing Syst., vol. 36, 2023. [34] L. Ding, D. Drusvyatskiy, M. Fazel, and Z. Harchaoui, “Flat minima generalize for low-rank matrix recovery,” Information and Inference: A Journal of the IMA, vol. 13, no. 2, p. iaae009, 2024. [35] N. Ding, X. Lv, Q. Wang, Y. Chen, B. Zhou, Z. Liu, and M. Sun, “Sparse low-rank adaptation of pre-trained language models,” in Proc. Conf. on Empirical Methods in Natural Language Process., Dec. 2023, pp. 4133–4145. [36] Y. Ding, N. Krislock, J. Qian, and H. Wolkowicz, “Sensor network localization, euclidean distance matrix completions, and graph realization,” in Proc. of the first ACM international workshop on Mobile entity localization and tracking in GPS-less environments, 2008, pp. 129–134. [37] H. Dong, T. Tong, C. Ma, and Y. Chi, “Fast and provable tensor robust principal component analysis via scaled gradient descent,” Information and Inference: A Journal of the IMA, vol. 12, no. 3, pp. 1716–1758, 2023. [38] S. S. Du, W. Hu, and J. D. Lee, “Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced,” in Proc. Neural Info. Processing Syst., vol. 31, 2018. [39] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [40] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh, “Krona: Parameter-efficient tuning with kronecker adapter,” in Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, 2025, pp. 49–65. [41] J. Fang, H. Jiang, K. Wang, Y. Ma, S. Jie, X. Wang, X. He, and T.-S. Chua, “Alphaedit: Null-space constrained knowledge editing for language models,” arXiv preprint arXiv:2410.02355, 2024. [42] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in Proc. Int. Conf. on Learn. Representations, 2021. [43] Z. Frangella, J. A. Tropp, and M. Udell, “Randomized nyström preconditioning,” SIAM Journal on Matrix Analysis and Applications, vol. 44, no. 2, pp. 718–752, 2023. [44] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “OPTQ: Accurate quantization for generative pre-trained transformers,” in Proc. Int. Conf. on Learn. Representations, 2023. [45] S. Gandy, B. Recht, and I. Yamada, “Tensor completion and low-n-rank tensor recovery via convex optimization,” Inverse problems, vol. 27, no. 2, p. 025010, 2011. [46] B. Gao, S. Vary, P. Ablin, and P.-A. Absil, “Optimization flows landing on the stiefel manifold,” International Symposium on Mathematical Theory of Networks and Systems MTNS, vol. 55, no. 30, pp. 25–30, 2022. [47] Z. Gao, Q. Wang, A. Chen, Z. Liu, B. Wu, L. Chen, and J. Li, “Parameterefficient fine-tuning with discrete fourier transform,” in Proc. Int. Conf. on Machine Learn., 2024. [48] R. Ge, C. Jin, and Y. Zheng, “No spurious local minima in nonconvex low rank problems: A unified geometric analysis,” in Proc. Int. Conf. on Machine Learn., 2017, pp. 1233–1242. [49] R. Ge, J. D. Lee, and T. Ma, “Matrix completion has no spurious local minimum,” Proc. Neural Info. Processing Syst., vol. 29, 2016. [50] R. Ge, Y. Ren, X. Wang, and M. Zhou, “Understanding deflation process in over-parametrized tensor decomposition,” in Proc. Neural Info. Processing Syst., M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, 2021, pp. 1299–1311. [51] Gemma-team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al., “Gemma: Open models based on gemini research and technology,” arXiv:2403.08295, 2024. [52] G. B. Giannakis, Q. Ling, G. Mateos, I. D. Schizas, and H. Zhu, Decentralized Learning for Wireless Communications and Networking. Springer International Publishing, 2016, pp. 461–497. [53] D. Gross, Y.-K. Liu, S. T. Flammia, S. Becker, and J. Eisert, “Quantum state tomography via compressed sensing,” Physical review letters, vol. 105, no. 15, p. 150401, 2010. [54] Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu et al., “Mix-of-show: Decentralized lowrank adaptation for multi-concept customization of diffusion models,” in Proc. Neural Info. Processing Syst., vol. 36, 2023. [55] W. Hackbusch and S. Kühn, “A new scheme for the tensor representation,” Journal of Fourier analysis and applications, vol. 15, no. 5, pp. 706–722, 2009.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

[56] A. Han, J. Li, W. Huang, M. Hong, A. Takeda, P. K. Jawanpuria, and B. Mishra, “SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 118 267–118 295. [57] L. Han, Y. Li, H. Zhang, P. Milanfar, D. Metaxas, and F. Yang, “Svdiff: Compact parameter space for diffusion fine-tuning,” in Proc. Int. Conf. on Computer Vision, 2023, pp. 7323–7334. [58] Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-efficient fine-tuning for large models: A comprehensive survey,” arXiv preprint arXiv:2403.14608, 2024. [59] M. Hardt and M. Wootters, “Fast matrix completion without the condition number,” in Conf. on learning theory, 2014, pp. 638–678. [60] J. Haupt, Q. Lu, Y. Shen, J. Chen, Y. Dong, D. McCreary, M. Akçakaya, and G. B. Giannakis, “Deploying AI for signal processing education: Selected challenges and intriguing opportunities,” IEEE Sig. Process. Mag., Special Issue on Artificial Intelligence for Education: A Signal Processing Perspective, vol. 43, no. 1, pp. 32–46, 2026. [61] S. Hayou, N. Ghosh, and B. Yu, “The impact of initialization on lora finetuning dynamics,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 117 015–117 040. [62] ——, “LoRA+: Efficient low rank adaptation of large models,” in Proc. Int. Conf. on Machine Learn., 2024. [63] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in Proc. Int. Conf. on Learn. Representations, 2021. [64] C. J. Hillar and L.-H. Lim, “Most tensor problems are np-hard,” Journal of the ACM (JACM), vol. 60, no. 6, pp. 1–39, 2013. [65] F. L. Hitchcock, “The expression of a tensor or a polyadic as a sum of products,” Journal of Mathematics and Physics, vol. 6, no. 1-4, pp. 164–189, 1927. [66] ——, “Multiple invariants and generalized rank of a p-way matrix or tensor,” Journal of Mathematics and Physics, vol. 7, no. 1-4, pp. 39–79, 1928. [67] H. Hotelling, “Relations between two sets of variates,” in Breakthroughs in statistics: methodology and distribution, 1992, pp. 162–190. [68] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proc. Int. Conf. on Machine Learn., 2019, pp. 2790–2799. [69] I. Hounie, C. Kanatsoulis, A. Tandon, and A. Ribeiro, “Lorta: Low rank tensor adaptation of large language models,” arXiv preprint arXiv:2410.04060, 2024. [70] C.-Y. Hsu, Y.-L. Tsai, C.-H. Lin, P.-Y. Chen, C.-M. Yu, and C.-Y. Huang, “Safe lora: The silver lining of reducing safety risks when finetuning large language models,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 65 072–65 094. [71] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. Int. Conf. on Learn. Representations, 2022. [72] C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin, “Lorahub: Efficient cross-task generalization via dynamic lora composition,” arXiv preprint arXiv:2307.13269, 2023. [73] Q. Huang, T. Ko, Z. Zhuang, L. Tang, and Y. Zhang, “Hira: Parameterefficient hadamard high-rank adaptation for large language models,” in Proc. Int. Conf. on Learn. Representations, 2025. [74] N. Hyeon-Woo, M. Ye-Bin, and T.-H. Oh, “Fedpara: Low-rank hadamard product for communication-efficient federated learning,” in Proc. Int. Conf. on Learn. Representations, 2022. [75] A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural networks, vol. 13, no. 4-5, pp. 411–430, 2000. [76] A. Jacot, F. Ged, B. Şimşek, C. Hongler, and F. Gabriel, “Saddle-tosaddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity,” arXiv preprint arXiv:2106.15933, 2021. [77] M. Jaggi, “Revisiting frank-wolfe: Projection-free sparse convex optimization,” in Proc. Int. Conf. on Machine Learn., 2013, pp. 427–435. [78] P. Jain, P. Netrapalli, and S. Sanghavi, “Low-rank matrix completion using alternating minimization,” in Proc. of the forty-fifth annual ACM symposium on Theory of computing, 2013, pp. 665–674. [79] U. Jang, J. D. Lee, and E. K. Ryu, “LoRA training in the NTK regime has no spurious local minima,” in Proc. Int. Conf. on Machine Learn., 2024. [80] X. Jia, H. Wang, J. Peng, X. Feng, and D. Meng, “Preconditioning matters: Fast global convergence of non-convex matrix factorization via scaled gradient descent,” in Proc. Neural Info. Processing Syst., 2023. [81] L. Jiang, Y. Chen, and L. Ding, “Algorithmic regularization in modelfree overparametrized asymmetric matrix factorization,” SIAM Journal on Mathematics of Data Science, vol. 5, no. 3, pp. 723–744, 2023.

[82] P. Jiang, D. Ergu, F. Liu, Y. Cai, and B. Ma, “Can we trust you? on calibration of a probabilistic object detector for autonomous driving,” in Proc. IEEE/RSJ International Conf. on Intelligent Robots and Systems, 2021, pp. 3188–3194. [83] T. Jiang and N. D. Sidiropoulos, “Kruskal’s permutation lemma and the identification of candecomp/parafac and bilinear models with constant modulus constraints,” IEEE Trans. Signal Processing, vol. 52, no. 9, pp. 2625–2636, 2004. [84] S. Jie and Z.-H. Deng, “Fact: Factor-tuning for lightweight adaptation on vision transformer,” in Proc. AAAI Conf. Artif. Intel., vol. 37, no. 1, 2023, pp. 1060–1068. [85] K. H. Jin and J. C. Ye, “Annihilating filter-based low-rank hankel matrix approach for image inpainting,” IEEETIP, vol. 24, no. 11, pp. 3498–3511, 2015. [86] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson et al., “Language models (mostly) know what they know,” arXiv preprint arXiv:2207.05221, 2022. [87] D. S. Kalogerias and A. P. Petropulu, “Matrix completion in colocated mimo radar: Recoverability, bounds & theoretical guarantees,” IEEE Trans. Signal Processing, vol. 62, no. 2, pp. 309–321, 2013. [88] C. I. Kanatsoulis, X. Fu, N. D. Sidiropoulos, and M. Akçakaya, “Tensor completion from regular sub-nyquist samples,” IEEE Trans. Signal Processing, vol. 68, pp. 1–16, 2019. [89] R. H. Keshavan, A. Montanari, and S. Oh, “Matrix completion from a few entries,” IEEE Trans. Signal Processing, vol. 56, no. 6, pp. 2980–2998, 2010. [90] C. G. Khatri and C. R. Rao, “Solutions to some functional equations and their applications to characterization of probability distributions,” Sankhyā: The Indian Journal of Statistics, Series A, pp. 167–180, 1968. [91] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM review, vol. 51, no. 3, pp. 455–500, 2009. [92] S. A. Koohpayegani, N. K. L, P. Nooralinejad, S. Kolouri, and H. Pirsiavash, “NOLA: Compressing loRA using linear combination of random basis,” in Proc. Int. Conf. on Learn. Representations, 2024. [93] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano, “VeRA: Vector-based random matrix adaptation,” in Proc. Int. Conf. on Learn. Representations, 2024. [94] Y. Koren, R. Bell, and C. Volinsky, “Matrix factorization techniques for recommender systems,” Computer, vol. 42, no. 8, pp. 30–37, 2009. [95] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in Proc. Int. Conf. on Machine Learn. PMlR, 2019, pp. 3519–3529. [96] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proc. of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [97] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” in Proc. Int. Conf. on Learn. Representations, 2020. [98] C. Lanczos, “An iteration method for the solution of the eigenvalue problem of linear differential and integral operators,” Journal of research of the National Bureau of Standards, vol. 45, no. 4, pp. 255–282, 1950. [99] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” nature, vol. 401, no. 6755, pp. 788–791, 1999. [100] S. Lermen, C. Rogers-Smith, and J. Ladish, “Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b,” arXiv preprint arXiv:2310.20624, 2023. [101] B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proc. Conf. on Empirical Methods in Natural Language Process., 2021, pp. 3045–3059. [102] E. Levin, J. Kileel, and N. Boumal, “The effect of smooth parametrizations on nonconvex optimization landscapes,” Mathematical Programming, Mar. 2024. [103] B. Li, M. Coutino, G. B. Giannakis, and G. Leus, “A momentum-guided frank-wolfe algorithm,” IEEE Trans. Signal Processing, vol. 69, pp. 3597–3611, 2021. [104] B. Li and G. B. Giannakis, “Enhancing sharpness-aware optimization through variance suppression,” in Proc. Neural Info. Processing Syst., vol. 36, 2023. [105] B. Li, L. Zhang, and N. He, “Implicit regularization of sharpnessaware minimization for scale-invariant problems,” in Proc. Neural Info. Processing Syst., 2024. [106] B. Li, L. Zhang, A. Mokhtari, and N. He, “On the crucial role of initialization for matrix factorization,” in Proc. Int. Conf. on Learn. Representations, 2025.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

[107] J. Li, L. Yin, L. Shen, J. Xu, L. Xu, T. Huang, W. Wang, S. Liu, and X. Wang, “Lost: Low-rank and sparse pre-training for large language models,” arXiv preprint arXiv:2508.02668, 2025. [108] M. Li, W. M. Si, M. Backes, Y. Zhang, and Y. Wang, “Salora: Safety-alignment preserved low-rank adaptation,” arXiv preprint arXiv:2501.01765, 2025. [109] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proc. Conf. Assoc. Comput. Linguist. Meet., 2021, pp. 4582–4597. [110] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “LoftQ: LoRA-fine-tuning-aware quantization for large language models,” in Proc. Int. Conf. on Learn. Representations, 2024. [111] Z. Li, Y. Luo, and K. Lyu, “Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning,” arXiv preprint arXiv:2012.09839, 2020. [112] Z. Li, S. Sajadmanesh, J. Li, and L. Lyu, “StelLA: Subspace learning in low-rank adaptation using stiefel manifold,” in Proc. Neural Info. Processing Syst., 2025. [113] V. Lialin, S. Muckatira, N. Shivagunde, and A. Rumshisky, “ReLoRA: High-rank training through low-rank updates,” in Proc. Int. Conf. on Learn. Representations, 2024. [114] K. Lion, L. Zhang, B. Li*, and N. He*, “Polar: Polar-decomposed low-rank adapter representation,” in Proc. Neural Info. Processing Syst., 2025, (*equal supervision). [115] A. E. Litvak and K. E. Tikhomirov, “Singularity of sparse bernoulli matrices,” Duke Mathematical Journal, vol. 171, no. 5, pp. 1135–1233, 2022. [116] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.T. Cheng, and M.-H. Chen, “DoRA: Weight-decomposed low-rank adaptation,” in Proc. Int. Conf. on Machine Learn., 2024. [117] W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng, Z. Liu, J. Heo, S. Peng et al., “Parameter-efficient orthogonal finetuning via butterfly factorization,” arXiv preprint arXiv:2311.06243, 2023. [118] Z. Liu, Z. Han, Y. Tang, S. Tang, and Y. Wang, “Efficient overparameterized matrix sensing from noisy measurements via alternating preconditioned gradient descent,” arXiv preprint arXiv:2502.00463, 2025. [119] Z. Liu, Z. Han, Y. Tang, X.-L. Zhao, and Y. Wang, “Low-tubal-rank tensor recovery via factorized gradient descent,” IEEE Trans. Signal Processing, 2024. [120] J. Lopez-Piqueres, P. Deshpande, A. Ray, M. J. Villani, M. Pistoia, and N. Kumar, “Metatt: A global tensor-train adapter for parameter-efficient fine-tuning,” arXiv preprint arXiv:2506.09105, 2025. [121] Q. Lu, G. Karanikolas, Y. Shen, and G. B. Giannakis, “Ensemble gaussian processes with spectral features for online interactive learning with scalability,” in Proc. Int. Conf. on Artificial Intelligence and Statistics, 2020, pp. 1910–1920. [122] Q. Lu, K. D. Polyzos, B. Li, and G. B. Giannakis, “Surrogate modeling for bayesian optimization beyond a single gaussian process,” IEEEPAMI, vol. 45, no. 9, pp. 11 283–11 296, 2023. [123] Y. Lu, “How much VRAM do i need for llm model finetuning?” Sep. 2024. [Online]. Available: https://modal.com/blog/ how-much-vram-need-fine-tuning [124] C. Ma, K. Wang, Y. Chi, and Y. Chen, “Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval and matrix completion,” in Proc. Int. Conf. on Machine Learn., 2018, pp. 3345–3354. [125] H. Ma, M. I. Bal, L. Zhang, B. Li, N. He, M. Zeilinger, and M. Muehlebach, “Salaad: Sparse and low-rank adaptation via admm,” arXiv preprint arXiv:2602.00942, 2026. [126] Z. Ma, I. Molybog, J. Lavaei, and S. Sojoudi, “Over-parametrization via lifting for low-rank matrix sensing: Conversion of spurious solutions to strict saddle points,” in Proc. Int. Conf. on Machine Learn., 2023, pp. 23 373–23 387. [127] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora, “Fine-tuning language models with just forward passes,” in Proc. Neural Info. Processing Syst., vol. 36, 2023, pp. 53 038–53 075. [128] S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, B. Bossan, and M. Tietz, “PEFT: State-of-the-art parameter-efficient fine-tuning methods,” https://github.com/huggingface/peft, 2022. [129] Y. Mao, Y. Ge, Y. Fan, W. Xu, Y. Mi, Z. Hu, and Y. Gao, “A survey on lora of large language models,” Frontiers of Computer Science, vol. 19, no. 7, p. 197605, 2025. [130] M. Mardani, G. Mateos, and G. B. Giannakis, “Recovery of low-rank plus compressed sparse matrices with application to unveiling traffic anomalies,” IEEE Trans. Inf. Theory, vol. 59, no. 8, pp. 5186–5205, 2013.

[131] ——, “Subspace learning and imputation for streaming big data matrices and tensors,” IEEE Trans. Signal Processing, vol. 63, no. 10, pp. 2663– 2677, 2015. [132] J. Martens and R. Grosse, “Optimizing neural networks with kroneckerfactored approximate curvature,” in Proc. Int. Conf. on Machine Learn., 2015, pp. 2408–2417. [133] G. Mateos and G. B. Giannakis, “Robust PCA as bilinear decomposition with outlier-sparsity regularization,” IEEE Trans. Signal Processing, vol. 60, no. 10, pp. 5176–5190, 2012. [134] F. Meng, Z. Wang, and M. Zhang, “Pissa: Principal singular values and singular vectors adaptation of large language models,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 121 038–121 072. [135] E. Million, “The hadamard product,” Course Notes, vol. 3, no. 6, pp. 1–7, 2007. [136] Z. Min and X. Wang, “Docs: Quantifying weight similarity for deeper insights into large language models,” arXiv preprint arXiv:2501.16650, 2025. [137] B. Mishra, G. Meyer, S. Bonnabel, and R. Sepulchre, “Fixed-rank matrix factorizations and riemannian low-rank optimization,” Computational Statistics, vol. 29, no. 3, pp. 591–621, 2014. [138] Z. Mo, L.-K. Huang, and S. J. Pan, “Parameter and memory efficient pretraining via low-rank riemannian optimization,” in Proc. Int. Conf. on Learn. Representations, 2025. [139] S. Negahban and M. J. Wainwright, “Estimation of (near) low-rank matrices with noise and high-dimensional scaling,” The Annals of Statistics, vol. 39, no. 2, pp. 1069–1097, 2011. [140] Y. Nesterov, Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2013, vol. 87. [141] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li, “Large language diffusion models,” arXiv preprint arXiv:2502.09992, 2025. [142] M. Nikdan, S. Tabesh, and D. Alistarh, “RoSA: Accurate parameterefficient fine-tuning via robust adaptation,” in Proc. Int. Conf. on Machine Learn., 2024. [143] B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, vol. 381, no. 6583, pp. 607–609, 1996. [144] V. Papyan, J. Sulam, and M. Elad, “Working locally thinking globally: Theoretical guarantees for convolutional sparse coding,” IEEETSP, vol. 65, no. 21, pp. 5687–5701, 2017. [145] K. Pearson, “On lines and planes of closest fit to systems of points in space,” The London, Edinburgh, and Dublin philosophical magazine and journal of science, vol. 2, no. 11, pp. 559–572, 1901. [146] T. Putterman, D. Lim, Y. Gelberg, S. Jegelka, and H. Maron, “Learning on loras: Gl-equivariant processing of low-rank weight spaces for large finetuned models,” arXiv:2410.04207, 2024. [147] H. Qin, X. Ma, X. Zheng, X. Li, Y. Zhang, S. Liu, J. Luo, X. Liu, and M. Magno, “Accurate LoRA-finetuning quantization of LLMs via information retention,” in Proc. Int. Conf. on Machine Learn., 2024. [148] Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf, “Controlling text-to-image diffusion by orthogonal finetuning,” in Proc. Neural Info. Processing Syst., vol. 36, 2023, pp. 79 320–79 362. [149] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018. [150] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. [151] A. H. Rahmati, S. Jantre, W. Zhang, Y. Wang, B.-J. Yoon, N. Urban, and X. Qian, “C-loRA: Contextual low-rank adaptation for uncertainty estimation in large language models,” in Proc. Neural Info. Processing Syst., 2025. [152] D. Ran, X. He, T. Cong, A. Wang, Q. Li, and X. Wang, “Lora-leak: Membership inference attacks against lora fine-tuned language models,” arXiv preprint arXiv:2507.18302, 2025. [153] B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM review, vol. 52, no. 3, pp. 471–501, 2010. [154] A. Renduchintala, T. Konuk, and O. Kuchaiev, “Tied-lora: Enhancing parameter efficiency of lora with weight tying,” in Proc. Conf. Assoc. Comput. Linguist. Meet., 2024, pp. 8694–8705. [155] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subjectdriven generation,” in Proc. Conf. Computer Vision and Pattern Recognition, 2023, pp. 22 500–22 510.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

[156] M. Sadeghi, M. G. Nejad, M. J. Asl, Y. Gu, Y. Yu, M. Asgharian, and V. P. Nia, “Moka: Mixture of kronecker adapters,” arXiv preprint arXiv:2508.03527, 2025. [157] T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” vol. 29, 2016. [158] C. Samplawski, A. D. Cobb, M. Acharya, R. Kaur, and S. Jha, “Scalable bayesian low-rank adaptation of large language models via stochastic variational subspace inference,” in Proc. Conf. on Uncertainty in Artificial Intelligence, vol. 286, 21–25 Jul 2025, pp. 3587–3604. [159] S. Schechtman, D. Tiapkin, M. Muehlebach, and E. Moulines, “Orthogonal directions constrained gradient method: from non-linear equality constraints to Stiefel manifold,” in Proc. Annual Conf. on Learn. Theory, 2023. [160] I. D. Schizas and G. B. Giannakis, “Covariance eigenvector sparsity for compression and denoising,” IEEE Trans. Signal Processing, vol. 60, no. 5, pp. 2408–2421, 2012. [161] B. Schölkopf, A. Smola, and K.-R. Müller, “Kernel principal component analysis,” in Proc. International Conf. on artificial neural networks, 1997, pp. 583–588. [162] S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco, “Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations,” in Proc. Neural Info. Processing Syst., vol. 35, 2022, pp. 20 051–20 063. [163] J. Schulman and T. M. Lab, “Lora without regret,” Thinking Machines Lab: Connectionism, 2025, https://thinkingmachines.ai/blog/lora/. [164] O. Sener and V. Koltun, “Multi-task learning as multi-objective optimization,” in Proc. Neural Info. Processing Syst., vol. 31, 2018. [165] V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani, “Ziplora: Any subject in any style by effectively merging loras,” 2024, pp. 422–438. [166] O. Shamir, “Exponential convergence time of gradient descent for onedimensional deep linear neural networks,” in Proc. Annual Conf. on Learn. Theory, 2019, pp. 2691–2713. [167] Y. Shao, M. Yan, Y. Liu, S. Chen, W. Chen, X. Long, Z. Yan, L. Li, C. Zhang, N. Sebe et al., “In-context meta lora generation,” arXiv preprint arXiv:2501.17635, 2025. [168] F. Sheikholeslami, D. Berberidis, and G. B. Giannakis, “Large-scale kernel-based feature extraction via low-rank subspace tracking on a budget,” IEEE Trans. Signal Processing, vol. 66, no. 8, pp. 1967–1981, 2018. [169] Y. Shen, B. Baingana, and G. B. Giannakis, “Tensor decompositions for identifying directed graph topologies and tracking dynamic networks,” IEEE Trans. Signal Processing, vol. 65, no. 14, pp. 3675–3687, 2017. [170] Y. Shen, T. Chen, and G. B. Giannakis, “Random feature-based online multi-kernel learning in environments with unknown dynamics,” Journal of Machine Learning Research, vol. 20, no. 22, pp. 1–36, 2019. [171] Z. Shen, Y. He, Z. Wang, Y. Zhang, G. Sun, W. Ye, and A. Li, “Edgelora: An efficient multi-tenant llm serving system on edge devices,” in Proc. of the Annual International Conf. on Mobile Systems, Applications and Services, 2025, pp. 138–153. [172] D. Shenaj, O. Bohdal, M. Ozay, P. Zanuttigh, and U. Michieli, “Lora. rar: Learning to merge loras via hypernetworks for subject-style conditioned image generation,” in Proc. Int. Conf. on Computer Vision, 2025, pp. 16 132–16 142. [173] Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica, “S-LoRA: Serving thousands of concurrent lora adapters,” in Proc. Conf. on Machine Learn. and Syst., 2024. [174] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Proc. Neural Info. Processing Syst., vol. 36, 2023, pp. 8634–8652. [175] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos, “Tensor decomposition for signal processing and machine learning,” IEEE Trans. Signal Processing, vol. 65, no. 13, pp. 3551–3582, 2017. [176] N. D. Sidiropoulos, G. B. Giannakis, and R. Bro, “Blind parafac receivers for ds-cdma systems,” IEEE Trans. Signal Processing, vol. 48, no. 3, pp. 810–823, 2002. [177] E. Simsar, T. Hofmann, F. Tombari, and P. Yanardag, “Loraclr: Contrastive adaptation for customization of diffusion models,” in Proc. Conf. Computer Vision and Pattern Recognition, 2025, pp. 13 189– 13 198. [178] R. Singhal, K. Ponkshe, R. Vartak, and P. Vepakomma, “ABBA: Highly expressive hadamard product adaptation for large language models,” arXiv preprint arXiv:2505.14238, 2025.

[179] G. P. Styan, “Hadamard products and multivariate statistical analysis,” Linear algebra and its applications, vol. 6, pp. 217–240, 1973. [180] S. Sun, D. Gupta, and M. Iyyer, “Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf,” arXiv preprint arXiv:2309.09055, 2023. [181] Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving lora in privacy-preserving federated learning,” arXiv preprint arXiv:2403.12313, 2024. [182] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière et al., “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786, 2025. [183] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé et al., “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024. [184] Q. Team, “Qwen3-max: Just scale it,” Sep. 2025. [185] J. B. Tenenbaum, V. de Silva, and J. C. Langford, “A global geometric framework for nonlinear dimensionality reduction,” Science, vol. 290, no. 5500, pp. 2319–2323, 2000. [186] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,” in Proc. Conf. on Empirical Methods in Natural Language Process., 2023, pp. 5433–5442. [187] T. Tong, C. Ma, and Y. Chi, “Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent,” J. Mach. Learn. Res., vol. 22, no. 150, pp. 1–63, 2021. [188] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv:2302.13971, 2023. [189] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv:2307.09288, 2023. [190] J. A. Tropp, A. Yurtsever, M. Udell, and V. Cevher, “Fixed-rank approximation of a positive-semidefinite matrix from streaming data,” in Proc. Neural Info. Processing Syst., vol. 30, 2017. [191] L. R. Tucker et al., “The extension of factor analysis to threedimensional matrices,” Contributions to mathematical psychology, vol. 110119, pp. 110–182, 1964. [192] M. Turk and A. Pentland, “Eigenfaces for recognition,” Journal of cognitive neuroscience, vol. 3, no. 1, pp. 71–86, 1991. [193] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “Dylora: Parameter-efficient tuning of pre-trained models using dynamic searchfree low-rank adaptation,” in Proc. Conf. Assoc. Comput. Linguist. Meet., 2023, pp. 3274–3287. [194] C. F. Van Loan, “The ubiquitous kronecker product,” Journal of computational and applied mathematics, vol. 123, no. 1-2, pp. 85– 100, 2000. [195] L. Veeramacheneni, M. Wolter, H. Kuehne, and J. Gall, “Canonical rank adaptation: An efficient fine-tuning strategy for vision transformers,” in Proc. Int. Conf. on Machine Learn., 2025. [196] H. Wang, Y. Li, S. Wang, G. Chen, and Y. Chen, “Milora: Harnessing minor singular components for parameter-efficient llm finetuning,” in Proc. Conf. Assoc. Comput. Linguist. Meet., 2025, pp. 4823–4836. [197] L. Wang, S. Chen, L. Jiang, S. Pan, R. Cai, S. Yang, and F. Yang, “Parameter-efficient fine-tuning in large language models: a survey of methodologies,” Artificial Intelligence Review, vol. 58, no. 8, p. 227, 2025. [198] R. Wang, Introduction to orthogonal transforms: with applications in data processing and analysis. Cambridge University Press, 2012. [199] S. Wang, J. Asilis, Ö. F. Akgül, E. B. Bilgin, O. Liu, and W. Neiswanger, “Tina: Tiny reasoning models via lora,” arXiv preprint arXiv:2504.15777, 2025. [200] S. Wang, L. Yu, and J. Li, “Lora-ga: Low-rank adaptation with gradient approximation,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 54 905–54 931. [201] Y. Wang, H. Shi, L. Han, D. N. Metaxas, and H. Wang, “BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models,” Proc. Neural Info. Processing Syst., 2024. [202] Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan, “LoRA-pro: Are low-rank adapters properly optimized?” in Proc. Int. Conf. on Learn. Representations, 2025. [203] R. Ward and T. Kolda, “Convergence of alternating gradient descent for matrix factorization,” in Proc. Neural Info. Processing Syst., vol. 36, 2023, pp. 22 369–22 382. [204] Y. Wei, Y. Miao, D. Zhou, and D. Hu, “Moka: Multimodal low-rank adaptation for MLLMs,” in Proc. Neural Info. Processing Syst., 2025.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

[205] Y. Wei, L. Zhang, B. Li, and N. He, “On the benefits of weight normalization for overparameterized matrix sensing,” arXiv preprint arXiv:2510.01175, 2025. [206] ——, “On direction–magnitude decomposition in low-rank matrix optimization,” sumitted to SIMODS, 2026. [207] Y. Wen and S. Chaudhuri, “Batched low-rank adaptation of foundation models,” arXiv preprint arXiv:2312.05677, 2023. [208] S. Wertz, A. Vandaele, and N. Gillis, “Efficient algorithms for the hadamard decomposition,” in Proc. IEEE International Workshop on Machine Learning for Signal Processing, 2025, pp. 1–6. [209] X. Wu, S. Huang, and F. Wei, “Mixture of lora experts,” arXiv preprint arXiv:2404.13628, 2024. [210] Y. Wu, Y. Shi, J. Wei, C. Sun, Y. Yang, and H. T. Shen, “Difflora: Generating personalized low-rank adaptation weights with diffusion,” arXiv preprint arXiv:2408.06740, 2024. [211] D. Xia and M. Yuan, “On polynomial time methods for exact low-rank tensor completion,” Foundations of Computational Mathematics, vol. 19, no. 6, pp. 1265–1313, 2019. [212] W. Xia, C. Qin, and E. Hazan, “Chain of LoRA: Efficient fine-tuning of language models via residual learning,” in ICML 2024 Workshop on LLMs and Cognition, 2024. [213] Y. Xia, F. Fu, W. Zhang, J. Jiang, and B. Cui, “Efficient multi-task llm quantization and serving for multiple lora adapters,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 63 686–63 714. [214] X. Xu, Y. Shen, Y. Chi, and C. Ma, “The power of preconditioning in overparameterized low-rank matrix sensing,” in Proc. Int. Conf. on Machine Learn., 2023, pp. 38 611–38 654. [215] Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian, “QA-loRA: Quantization-aware low-rank adaptation of large language models,” in Proc. Int. Conf. on Learn. Representations, 2024. [216] Z. Xu, Y. Wang, T. Zhao, R. Ward, and M. Tao, “Provable acceleration of nesterov’s accelerated gradient for rectangular matrix factorization and linear neural networks,” in Proc. Neural Info. Processing Syst., 2024. [217] A. X. Yang, M. Robeyns, X. Wang, and L. Aitchison, “Bayesian Lowrank Adaptation for Large Language Models,” Proc. Int. Conf. on Learn. Representations, 2024. [218] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [219] Y. Yang, W. Wang, L. Peng, C. Song, Y. Chen, H. Li, X. Yang, Q. Lu, D. Cai, B. Wu et al., “Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models,” arXiv preprint arXiv:2403.11627, 2024. [220] Y. Yang, J. Zhou, N. Wong, and Z. Zhang, “Loretta: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models,” in Proc. North American Chapter of the Association for Computational Linguistics (NAACL), 2024, pp. 3161–3176. [221] Y. Yao and G. Giannakis, “On regularity and identifiability of blind source separation under constant-modulus constraints,” IEEE Trans. Signal Processing, vol. 53, no. 4, pp. 1272–1281, 2005. [222] T. Ye and S. S. Du, “Global convergence of gradient descent for asymmetric low-rank matrix factorization,” in Proc. Neural Info. Processing Syst., vol. 34, 2021, pp. 1429–1439. [223] S.-Y. YEH, Y.-G. Hsieh, Z. Gao, B. B. W. Yang, G. Oh, and Y. Gong, “Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation,” in Proc. Int. Conf. on Learn. Representations, 2024. [224] J.-N. Yen, S. Si, Z. Meng, F. Yu, S. S. Duvvuri, I. S. Dhillon, C.-J. Hsieh, and S. Kumar, “LoRA done RITE: Robust invariant transformation equilibration for loRA optimization,” in Proc. Int. Conf. on Learn. Representations, 2025. [225] F. W. Young and R. M. Hamer, Multidimensional scaling: History, theory, and applications. Psychology Press, 2013. [226] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz et al., “Differentially private finetuning of language models,” arXiv preprint arXiv:2110.06500, 2021. [227] S. Yuan, H. Liu, and H. Xu, “Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation,” in Proc. Neural Info. Processing Syst., vol. 37, 2024, pp. 113 484–113 518. [228] D. Zachariah, M. Sundin, M. Jansson, and S. Chatterjee, “Alternating least-squares for low-rank matrix reconstruction,” IEEE Sig. Process. Lett., vol. 19, no. 4, pp. 231–234, 2012. [229] E. B. Zaken, Y. Goldberg, and S. Ravfogel, “Bitfit: Simple parameterefficient fine-tuning for transformer-based masked language-models,” in Proc. Conf. Assoc. Comput. Linguist. Meet., 2022, pp. 1–9.

[230] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “HellaSwag: Can a machine really finish your sentence?” in Proc. Conf. Assoc. Comput. Linguist. Meet., May 2019. [231] Y. Zeng and K. Lee, “The expressive power of low-rank adaptation,” in Proc. Int. Conf. on Learn. Representations, 2024. [232] F. Zhang and M. Pilanci, “Riemannian preconditioned LoRA for finetuning foundation models,” in Proc. Int. Conf. on Machine Learn., 2024. [233] F. Zhang, L. Li, J. Chen, Z. Jiang, B. Wang, and Y. Qian, “Increlora: Incremental parameter allocation method for parameter-efficient finetuning,” arXiv preprint arXiv:2308.12043, 2023. [234] J. Zhang, S. Fattahi, and R. Y. Zhang, “Preconditioned gradient descent for over-parameterized nonconvex matrix factorization,” in Proc. Neural Info. Processing Syst., vol. 34, 2021, pp. 5985–5996. [235] L. Zhang, B. Li, K. K. Thekumparampil, S. Oh, and N. He, “Dpzero: Private fine-tuning of language models without backpropagation,” arXiv preprint arXiv:2310.09639, 2023. [236] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” in Proc. Int. Conf. on Learn. Representations, 2023. [237] Y. Zhang, P. Li, J. Hong, J. Li, Y. Zhang, W. Zheng, P.-Y. Chen, J. D. Lee, W. Yin, M. Hong, Z. Wang, S. Liu, and T. Chen, “Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark,” in Proc. Int. Conf. on Machine Learn., vol. 235, 21–27 Jul 2024, pp. 59 173–59 190. [238] Y. Zhang, B. Li, and G. B. Giannakis, “RefLoRA: Refactored low-rank adaptation for efficient fine-tuning of large models,” in Proc. Neural Info. Processing Syst., 2025, in submission. [239] Y. Zhang, B. Li, N. He, and G. B. Giannakis, “ANCRe: Adaptive neural connection reassignment for efficient depth scaling,” arXiv preprint arXiv:2602.09009, 2026. [240] Y. Zhang, X. Yang, Y. Cai, and G. B. Giannakis, “Scalora: Optimally scaled low-rank adaptation for efficient high-rank fine-tuning,” arXiv preprint arXiv:2510.23818, 2025. [241] Y. Zhang, F. Liu, and Y. Chen, “LoRA-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently,” in Proc. Int. Conf. on Machine Learn., 2025. [242] Z. Zhang and S. Aeron, “Exact tensor completion using t-svd,” IEEE Trans. Signal Processing, vol. 65, no. 6, pp. 1511–1526, 2016. [243] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian, “GaLore: Memory-efficient LLM training by gradient low-rank projection,” in Proc. Int. Conf. on Machine Learn., 2024. [244] Z. Zhao, L. Gan, G. Wang, Y. Hu, T. Shen, H. Yang, K. Kuang, and F. Wu, “Retrieval-augmented mixture of lora experts for uploadable machine learning,” arXiv preprint arXiv:2406.16989, 2024. [245] J. Zhu, K. Greenewald, K. Nadjahi, H. S. d. O. Borde, R. B. Gabrielsson, L. Choshen, M. Ghassemi, M. Yurochkin, and J. Solomon, “Asymmetry in low-rank adapters of foundation models,” in Proc. Int. Conf. on Machine Learn., 2024.

A PPENDIX A. Proof of Proposition 1 Proof. First, we demonstrate that ∆W can express any matrix of rank up to r. Let X2 = [1m , 0, . . . , 0] and Y2 = [1n , 0, . . . , 0], where 1 denotes the all-ones vector of the specified dimension. This yields ∆W = (X1 Y1⊤ ) ⊙ (X2 Y2⊤ ) = X1 Y1⊤ . This recovers the BM factorization used in vanilla LoRA, which spans all matrices of rank at most r. Second, we evaluate the expressiveness for higher-rank matrices. Using Theorem 1, it is clear that rank(∆W) ≤ r2 . Recall that the dimension of a rank-r matrix manifold in Rm×n is r(m + n − r). As ∆W is parameterized by two manifolds of rank at most r, it has no more than 2r(m + n − r) degrees of freedom (DoF). The dimension of a rank-r2 manifold may exceed this DoF via r2 (m + n − r2 ) > 2r(m + n − r). As a consequence, the parameterization cannot be surjective. That says, ∆W can express only a subset of matrices of rank within (r, r2 ].

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

B. Proof of Proposition 2 Proof. From Theorem 1, the upper bound of rank is rmax := min{rank(W) · r, m, n}. To prove the proposition, it suffices to show that there exists certain W, such that ∆W = W ⊙ (XY⊤ ) can only cover a subset of matrices with rank in (0, r] and (r, rmax ]. A simple example is W = I, which results in diagonal ∆W.

space Rn×r × Rm×r . For conciseness, denote the Euclidean (Frobenius) gradients by GX := ∇X F (X, Y),

GY := ∇Y F (X, Y).

Next, we derive the Riemannian gradient from the Riemannian metric g(X,Y) (ξX , ξY ), (φX , φY ) := ⟨ξX (Y⊤ Y), φX ⟩F + ⟨ξY (X⊤ X), φY ⟩F .

C. Proof of Proposition 3

The Riemannian gradient gradF (X, Y) = Proof. From Thoerem 2, the upper bound of rank(A ⊗ B) is (grad F, grad F ) is defined as the unique tangent vector 2 X Y r . Next, we construct a counterexample to show that exists certain ∆W which cannot be expressed as A ⊗ B. Specifically, satisfying, for any tangent direction (φX , φY ), that consider d1 = d2 = d3 = d4 = 2. It follows that g(X,Y) (gradX F, gradY F ), (φX , φY ) ï ò a B a12 B = ⟨GX , φX ⟩F + ⟨GY , φY ⟩F . (26) ∆W = 11 . a21 B a22 B Substituting the definition of g(X,Y) into (26) yields Hence the block structure cannot represent matrices such as   ⟨gradX F Y⊤ Y, φX ⟩F +⟨gradY F X⊤ X, φY ⟩F 1 0 0 0 = ⟨GX , φX ⟩F + ⟨GY , φY ⟩F . (27) 0 0 0 0   0 0 0 1 Since φX and φY are arbitrary, (27) implies 0 0 0 0 gradY F (X⊤ X) = GY . gradX F (Y⊤ Y) = GX , which suggests the Kronecker product can only express a subset As X and Y have full column rank so that X⊤ X and Y⊤ Y of matrices of rank up to r2 . are invertible, it follows D. Proof of Proposition 4

gradX F = GX (Y⊤ Y)−1 ,

Proof. First, setting S = 0 recovers LoRA, thus enabling parameterizing all matrices of rank no more than r. Second, using Theorem 3 leads to the upper bound rank(∆W) ≤ min{rank(XY⊤ ) + rank(S), m, n} ≤ min{m, n}

gradY F = GY (X⊤ X)−1 .

F. RefLoRA derivation Similar to ScaledGD, we first illustrate that the metric of RefLoRA is gauge-invariant. Define geometric mean S̃ := 1/2 −1/2 1/2 1/2 −1/2 P̃X P̃X P̃Y P̃X P̃X where P̃X := X̃⊤ X̃ and P̃Y := Ỹ⊤ Ỹ. Using (X̃, Ỹ) = (XQ, YQ−⊤ ), Q ∈ GL(r), we have −⊤ 2 Q (P̃X P̃Y )1/2 Q⊤ = Q−⊤ P̃X P̃Y Q⊤

which concludes the proof. E. ScaledGD derivation

First, we show that ScaledGD’s metric is gauge-invariant. = PX PY Consider two gauge-equivalent points (X, Y) and (X̃, Ỹ) = 1/2 ⇒ (P̃X P̃Y ) = Q⊤ (PX PY )1/2 Q−⊤ (XQ, YQ−⊤ ), Q ∈ GL(r). Let (ξX , ξY ), (φX , φY ) ∈ TX,Y be any two points on the tangent space at (X, Y). By Then, it follows from [238, Lemma 7] that pushforward, the corresponding gauge-invariant points on 1/2 S̃ = P̃−1 X (P̃X P̃Y ) the tangent space TX̃,Ỹ are (ξ̃X , ξ̃Y ) = (ξX Q, ξY Q−⊤ ) ⊤ 1/2 −⊤ = P̃−1 Q and (φ̃X , φ̃Y ) = (φX Q, φY Q−⊤ ). Using the definition of X Q (PX PY ) −1 −1 1/2 ScaledGD’s metric, it follows that = Q PX (PX PY ) Q−⊤ = Q−1 SQ−⊤ . (28) g(X̃,Ỹ) (ξ̃X , ξ̃Y ), (φ̃X , φ̃Y ) = ⟨ξ̃X (Ỹ⊤ Ỹ), φ̃X ⟩F + ⟨ξ̃Y (X̃⊤ X̃), φ̃Y ⟩F

= ⟨ξX Q(Q

−1

⊤

Y YQ

−⊤

), φX Q⟩F +

⟨ξY Q−⊤ (Q⊤ X⊤ XQ), φY Q−⊤ ⟩F

= ⟨ξX (Y⊤ Y), φX ⟩F + ⟨ξY (X⊤ X), φY ⟩F = g(X,Y) (ξX , ξY ), (φX , φY ) which proves the gauge-invariance of the metric. Next, we drive the Riemannian gradient under the metric. Let F (X, Y) := f (X, Y) be the lifted objective on the total

Then by definition, the metric of RefLoRA satisfies g(X̃,Ỹ) (ξ̃X , ξ̃Y ), (φ̃X , φ̃Y ) = ⟨ξ̃X S̃, φ̃X ⟩F + ⟨ξ̃Y S̃−1 , φ̃Y ⟩F (a)

= ⟨ξX Q(Q−1 SQ−⊤ ), φX Q⟩F +

⟨ξY Q−⊤ (Q⊤ S−1 Q), φY Q−⊤ ⟩F

= ⟨ξX S, φX ⟩F + ⟨ξY S−1 , φY ⟩F = g(X,Y) (ξX , ξY ), (φX , φY )

IEEE TRANSACTIONS ON SIGNAL PROCESSING (SUBMITTED)

where (a) utilizes (28). This thus proves the gauge-invariance of RefLoRA’s metric. Next, we show that the Riemannian metric (18) leads to RefLoRA updates (19). By definition (26), the Riemannian gradient (gradX F, gradY F ) should satisfy ⟨gradX F S, ξX ⟩F +⟨gradY F S−1 , φY ⟩F

= ⟨GX , φX ⟩F + ⟨GY , φY ⟩F .

for any (φX , φY ) ∈ Rn×r × Rm×r . The Riemannian gradient thus satisfies gradX F S = GX ,

gradY F S−1 = GY .

gradY F = GY S.

Next, we prove that RefLoRA’s Riemmanian metric (18) guarantees the update of (X, Y) is always on the horizontal space, and has no gauge drift. vert First, the vertical space at (X, Y) is T(X,Y) = ⊤ r×r {(XU, −YU ) | U ∈ R }, which can be verified via (X + ϵXU)(Y − ϵYU⊤ )⊤ = XY⊤ + O(ϵ2 ).

For update (∆X, ∆Y) to be purely horizontal, we must have g(X,Y) ((XU, −YU⊤ ), (∆X, ∆Y)) = 0, ∀U ∈ Rr×r . Plugging in the update ∆X = −ηgradX F = −ηGX S−1 and ∆Y = −ηgradY F = −ηGY S gives g(X,Y) ((XU, −YU⊤ ), (∆X, ∆Y))

= ⟨XUS, −ηGX S−1 ⟩F + ⟨−YU⊤ S−1 , −ηGY S⟩F (a)

= ⟨XUS, −ηGW YS−1 ⟩F + ⟨−YU⊤ S−1 , −ηG⊤ W XS⟩F

where (a) comes from the chain rule with GW := ∇W F (X, Y). This demonstrates that the gauge component vanishes, thus completing the proof. G. FastLoRA for batched serving Recall from (24) that the first term Zl Wl already allows batching. Therefore we only consider the implementation of the second term in (24). For notational simplicity, the layer index l is omitted. Consider the output of k-th adapter Ñ é r X zk W ◦ (Xk Yk⊤ ) = zk W ◦ xkj yjk⊤ j=1

r X j=1

zk W ◦ xkj yjk⊤

zk diag(xkj )W ◦ diag(yjk )

xK⊤ j

zk ◦ xkj W ◦ diag(yjk ).

This is batching friendly, as one can simply stack all k related vectors together.

yjK⊤

Then the K outputs can be efficiently computed in batched form via  1  z (W ◦ X1 Y1⊤ ) r   X .. (Z ◦ X̃j )W ◦ Ỹj .  = . zK (W ◦ Xr Yr⊤ )

It then follows that gradX F = GX S−1 ,

Let Z = [z1⊤ , . . . , zK⊤ ]⊤ ∈ RK×m , and for each j ∈ [r] stack the j-th columns across users as  1⊤   1⊤  xj yj  ..   ..  j K×m j X̃ =  .  ∈ R , Ỹ =  .  ∈ RK×n .

j=1

Record · ID 126498 · SHA-256 22f1de5f354f8724
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.