LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

BRAIn Lab

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates Dmitrii Andriianov1 , Andrey Veprikov1,2 , Aleksandr Beznosikov1,3 1 Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) 2 SB AI Lab

arXiv:2609.02734v1 [cs.LG] 2 Sep 2026

3 Innopolis University

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to 2.8× cheaper than the truncated-SVD retraction used by prior manifold methods. We prove that the Frobenius-norm version of our surrogate recovers LoRA-Pro, and we identify the tangent-projected gradient, the Riemannian gradient of the manifold, as the stationarity measure natural to LoRA training and computable from the factor gradients alone. Under this measure we give the first global convergence guarantees for both LoRAPro and LoRA-TSD, with rates that drive the factor-gradient norms to zero. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B and Qwen3-32B, LoRA-TSD outperforms every competing LoRA optimizer and stays robust to the adapter rank. Code is available at https://github.com/brain-lab-research/LoRA-TSD.

1

Introduction

Full fine-tuning adapts all parameters of a pretrained language model, but its memory and storage costs become prohibitive at scale. Training requires gradients and optimizer states for every parameter, and each downstream task produces a separate model copy. Parameter-efficient fine-tuning (PEFT) reduces these costs by updating a small set of parameters while keeping the pretrained weights frozen [Han et al., 2024]. Among PEFT methods, low-rank adaptation (LoRA) has become a common approach to adapting foundation models. Its compact task-specific updates can be merged into the backbone without additional inference latency, while multiple adapters can share the same frozen model [Hu et al., 2022]. This structure is also used in deployed systems, including multi-adapter serving and domain-specific language models for customer-facing applications [Nguyen et al., 2024; Scozzafava et al., 2025]. For a frozen weight W0 ∈ Rm×n , LoRA represents the update as ∆W = BA, where B ∈ Rm×r , A ∈ Rr×n , and r ≪ min(m, n). This parameterization has motivated a broad line of subsequent work. Existing methods reduce training memory, allocate the rank budget adaptively, improve initialization, modify factor scaling or weight parameterization, and select the most useful adapters [Dettmers et al., 2023; Zhang et al., 2023; Meng et al., 2024; Hayou et al., 2024; Liu et al., 2024; Veprikov et al., 2026]. Despite this progress, the optimization of the LoRA factors remains less settled. Standard optimizers such as Adam [Kingma and Ba, 2015] update A and B as independent parameter blocks, even though the model depends on their product BA. Consequently, a well-scaled step in factor space can produce a poorly scaled or misaligned update in weight space. LoRA-specific optimizers address this mismatch by correcting the induced 1

weight update, removing its dependence on the chosen factorization, or optimizing the low-rank weight directly through fixed-rank manifold geometry [Wang et al., 2025; Tastan et al., 2025; Yen et al., 2024; Bonnabel, 2013; Zhang and Pilanci, 2024; Bogachev et al., 2025; Cesista et al., 2026]. Section 2 reviews these approaches and explains why applying a full-matrix optimizer such as Muon within the LoRA parameterization remains nontrivial. We propose LoRA-TSD (Tangent-Space Spectral Descent), which brings the spectral update of Muon [Jordan et al., 2024] to LoRA. Muon balances the singular directions of a matrix update instead of allowing a few directions to dominate. LoRA-TSD computes the corresponding update for the low-rank weight and maps it back to A and B. This mapping gives a factor-induced retraction that follows directly from the LoRA parametrization. Both our retraction and the truncated-SVD retraction used by prior manifold methods admit low-rank implementations. Ours avoids the truncated SVD and is faster in practice. Contributions. • We develop LoRA-TSD, a LoRA-native extension of the tangent spectral iteration proposed by Riemannion [Bogachev et al., 2025]. It operates through standard LoRA factors and uses a factor-induced retraction without materializing m × n matrices. • In the L-smooth setting, we prove convergence to stationarity for momentum-free LoRA-TSD. • We identify LoRA-Pro [Wang et al., 2025] as the Frobenius counterpart of LoRA-TSD and prove its convergence to stationarity. • Across the evaluated benchmarks, LoRA-TSD outperforms generic and LoRA-specific baselines and is the most stable method across adapter ranks.

2

Related Work

LoRA-specific optimizers. The induced weight update ∆(BA) = ∆B A + B ∆A is bilinear in the factors, so a step that is well-scaled for A and B can be misaligned in weight space: it depends on the factor norms and conditioning and generally differs from direct descent on BA. LoRA-specific optimizers address this mismatch. One line of work improves the parametrization or its initialization. LoRA+ [Hayou et al., 2024] assigns separate learning rates to the two factors, removing a width-dependent inefficiency of naive LoRA. DoRA [Liu et al., 2024] decomposes each pretrained weight into separately trainable magnitude and directional components, using LoRA to parameterize the directional update. PiSSA [Meng et al., 2024] and LoRA-GA [Wang et al., 2024] instead pick the initial factors from the leading singular directions of the weight or of its gradient, aligning the first LoRA step with full fine-tuning. A second line keeps the parametrization fixed and corrects the update itself. LoRA-Pro [Wang et al., 2025] is the central example. Let L denote the training loss. LoRA-Pro chooses the factor updates
2 (∆A⋆ , ∆B ⋆ ) = arg min ∇W L − B∆A + ∆BA F . (1) ∆A,∆B

LoFT [Tastan et al., 2025] pursues a related correction at the optimizer level by projecting the Adam moments into the low-rank subspace. The same factor ambiguity also creates an optimization problem: if two factorizations of the same adapter lead to different weight-space steps, training can become sensitive to the arbitrary scale and rotation of the LoRA factors, with one factor moving while the other stays nearly fixed [Yen et al., 2024]. LoRA-Rite formalizes the desired consistency as transformation invariance, meaning that replacing (B, A) by (BR−1 , RA) for any invertible R ∈ Rr×r does not change the induced update to BA. LoRA-TSD satisfies the same invariance, but chooses the target direction through spectral-norm tangent geometry. In the Frobenius-norm case, our formulation recovers the LoRA-Pro direction, with the step scale set by the optimizer learning rate.

2

Orthogonalized and spectral-norm optimizers. Muon [Jordan et al., 2024] orthogonalizes the momentum update before the step, which equalizes its singular values and prevents it from collapsing onto a few dominant directions. Bernstein and Newhouse [2024] identify this update as steepest descent under the spectral norm, placing Muon within a broader family of norm-constrained optimizers. Muon assumes access to the full weight matrix, so transferring its geometry to the low-rank setting is not immediate. LoRA-TSD does so by solving the spectral-norm oracle inside the tangent space of the fixed-rank manifold. Riemannian optimization for LoRA. Riemannian SGD projects the Euclidean gradient onto the tangent space and retracts the resulting step back to the manifold [Bonnabel, 2013; Absil et al., 2008; Vandereycken, 2013]. Zhang and Pilanci [2024] keep this scheme but replace the Euclidean metric with an r × r preconditioner that rescales each factor update by the Gram matrix of the other factor. Both descend along the ordinary tangent gradient, whereas Riemannion and LoRA-Muon, like our method, orthogonalize this tangent step to obtain a spectral-norm update. The closest work to ours is Riemannion [Bogachev et al., 2025], which applies Muon-like orthogonalized updates to low-rank adapters using the Riemannian geometry of fixed-rank matrices. We share the tangent-space motivation but differ in how the spectral-norm tangent update is approximated and how the tangent direction is mapped back to the factors. Section 3 makes this comparison precise. A closely related optimizer, LoRA-Muon [Cesista et al., 2026], develops the same spectral steepest descent view of the LoRA tangent space, together with a gauge-invariance analysis and a matched split weight-decay rule. To make the tangent problem tractable in closed form, it decouples the two tangent components ∆B A and B ∆A, splitting the trust region evenly between them and orthogonalizing each half with a single matrix sign step. The decoupling buys a closed form but discards the interaction between the two components, and the even split approximates the spectral-norm constraint only while the two stay aligned during training. The resulting step is a single closed-form expression with no mechanism to refine the direction toward the constrained spectral solution. LoRA-TSD retains the interaction through the tangent projector and reaches this solution by the alternating projection of Eq. (12). Concurrently, PoLoRA [Ghosh et al., 2026] derives a product-aware LMO with a curvature-weighted spectral constraint. Like LoRA-Muon, it decouples the constraint on B∆A + ∆BA into two factorwise constraints, then adds curvature preconditioning and magnitude control. LoRA-TSD instead retains the coupled tangent update.

3 3.1

Method Notation

We derive the update for a single adapted layer. Let W ∈ Rm×n be its weight matrix, and let L(W ) denote the training loss with all other parameters fixed. The same update is applied independently to each adapted layer. LoRA parametrization and gradients. LoRA keeps the pretrained matrix W0 fixed and trains a rank-r residual, Wk = W0 + Bk Ak , (2) where Bk ∈ Rm×r , Ak ∈ Rr×n , and r ≪ min(m, n). At iteration k, write GkW = ∇W L(Wk ),

GkA = ∇A L(Wk ),

GkB = ∇B L(Wk ).

(3)

The chain rule gives the relations between the full-space gradient and the observable factor gradients, GkA = Bk⊤ GkW ,

GkB = GkW A⊤ k.

(4)

The matrix GkW is the gradient we would obtain under full fine-tuning, but LoRA training exposes only GkA and GkB . 3

Spectral smoothness. Muon [Jordan et al., 2024] is steepest descent in the spectral-norm geometry [Bernstein and Newhouse, 2024], so we use the same norm for the local model. Assume that ∇W L(W ) is 1/λ-Lipschitz continuous with respect to the spectral norm. Then, for any perturbation ∆Wk , L(Wk + ∆Wk ) ≤ L(Wk ) + ⟨∇L(Wk ), ∆Wk ⟩ +

1 ∥∆Wk ∥22 . 2λ

(5)

First-order LoRA steps. A transition from (Bk , Ak ) to (Bk + ∆Bk , Ak + ∆Ak ) induces the weight change ∆WkLoRA = ∆Bk Ak + Bk ∆Ak + ∆Bk ∆Ak .

(6)

The first two terms are linear in the factor increments. The product ∆Bk ∆Ak is second order. Following standard LoRA optimizer derivations [Wang et al., 2025; Tastan et al., 2025; Bogachev et al., 2025], we choose the direction from the first-order transition ∆Bk Ak + Bk ∆Ak . Fixed-rank manifold and tangent space. The reachable LoRA residuals BA have rank at most r. We work on the rank-exact stratum because every matrix of rank at most r can be approximated arbitrarily well by rank-r matrices. The residual then lies on the smooth fixed-rank manifold Mr ⊂ Rm×n [Absil et al., 2008; Vandereycken, 2013]. The affine shift by W0 does not change the tangent geometry. The first-order LoRA updates at Wk = W0 + Bk Ak form the tangent space  TWk Mr = ∆Bk Ak + Bk ∆Ak | ∆Bk ∈ Rm×r , ∆Ak ∈ Rr×n . (7) After the second-order term is dropped, this is the exact search space of LoRA. It depends only on the column space of Bk and the row space of Ak , not on a particular scaling of the factors. When the iteration index is clear, write T = TWk Mr and denote by PT the orthogonal projector onto this tangent space. Linear minimization oracle. Bernstein and Newhouse [2024] derive connections between many optimizers and linear minimization oracles over norm balls. For a norm ball B and a gradient G, the oracle returns LMOB (G) ∈ arg min⟨G, X⟩. X∈B

For the spectral-norm ball, the oracle is the negative polar factor of G, which is the matrix-sign direction used by Muon.

3.2

Deriving the optimization problem

Apply the spectral smoothness bound in Eq. (5) to the first-order LoRA transition:
1 L Wk + ∆Bk Ak + Bk ∆Ak ≤ L(Wk ) + ⟨GkW , ∆Bk Ak + Bk ∆Ak ⟩ + ∥∆Bk Ak + Bk ∆Ak ∥22 . 2λ This yields the local surrogate   1 k 2 min ⟨GW , ∆Bk Ak + Bk ∆Ak ⟩ + ∥∆Bk Ak + Bk ∆Ak ∥2 . ∆Ak ,∆Bk 2λ

(8)

(9)

The factor increments are not unique. Different pairs (∆Bk , ∆Ak ) can induce the same weight-space change, and the network only sees this change. We therefore optimize directly over X = ∆Bk Ak + Bk ∆Ak . Using Eq. (7), Eq. (9) becomes   1 k 2 min ⟨GW , X⟩ + ∥X∥2 . (10) X∈TWk Mr 2λ The tangent-space surrogate of Eq. (10) is the common formulation behind several LoRA optimizers. Its Frobenius-norm version recovers the LoRA-Pro direction, as proved in Section 5. LoRA-TSD instead retains the spectral norm, like Riemannion [Bogachev et al., 2025], and uses the LMO formulation from Section 3.1 to obtain a Muon-style tangent update. 4

3.3

Tangent-space spectral descent

We now solve Eq. (10) in the spectral norm, matching the geometry of Muon. The quadratic term sets the step scale, while the linear term sets the descent direction. Since the spectral norm is homogeneous, the direction of the penalized problem can be obtained from the corresponding trust-region form: for a fixed radius, minimize the same linear term over the spectral-norm ball. This direction-scale separation is standard in normalized deep-learning optimizers. Sign and matrix-sign methods first choose a norm-steepest direction and then set the step length with a learning rate. Bernstein and Newhouse [2024] derive this LMO view for a broad class of optimizers, including the spectral-norm update underlying Muon. For LoRA, the trust region must also respect the tangent constraint X ∈ TWk Mr , giving min

X∈TWk Mr ∥X∥2 ≤ρ

⟨GkW , X⟩.

(11)

This move isolates the descent direction from the step scale. The scale is then set by the learning rate, while the convergence analysis only requires a positive descent margin for the direction returned by the constrained oracle. We approximate the constrained oracle with alternating projections [Bauschke and Borwein, 1996]: a tangent projection followed by an orthogonalization toward the spectral-norm ball,
Xt+1 = PT msign(Xt ) , X0 = PT (GkW ). (12) where msign(Z) = U V ⊤ for Z = U ΣV ⊤ , the polar factor. The limit X∞ is the flattest matrix in T consistent with PT (GkW ). This procedure is close to the tangent-space orthogonalization of Riemannion [Bogachev et al., 2025], but applies the projection iteratively as a refinement toward Eq. (11). Appendix C analyzes the update spectrum and shows that its first 2r singular values are almost equalized. This is the analogue of Muon for the LoRA tangent space. Theorem 1 establishes convergence for this update under the stated assumptions. Closed form for PT (GkW ). We omit the iteration index k for brevity. For PB = B(B ⊤ B)−1 B ⊤ and PA = A⊤ (AA⊤ )−1 A the projector onto the tangent space is PT (Z) = PB Z + ZPA − PB ZPA .

(13)

Substituting Z = GW , the unobserved full-space gradient cancels, and PT (GW ) is expressed purely through the factor gradients GA = B ⊤ GW and GB = GW A⊤ : PT (GW ) = B(B ⊤ B)−1 GA + GB (AA⊤ )−1 A − B(B ⊤ B)−1 (GA A⊤ )(AA⊤ )−1 A.

(14)

Thus PT (GW ) can be computed without forming GW , which LoRA does not expose (Section 3.1). Reconstructing the LoRA factor updates. The alternating projections in Eq. (12) yield a full-space update ∆W = ∆BA + B∆A ∈ T . To apply the update in the LoRA factors, we recover ∆A and ∆B. The split is not unique, so we use the following canonical choice. Let PA = A⊤ (AA⊤ )−1 A be the orthogonal projector onto row(A). As row(∆BA) ⊆ row(A) we set

so that

∆BA = ∆W PA = ∆W A⊤ (AA⊤ )−1 A,

(15)

∆B = ∆W A⊤ (AA⊤ )−1 .

(16)

B∆A = ∆W − ∆BA,

(17)

∆A = (B ⊤ B)−1 B ⊤ (∆W − ∆BA).

(18)

The residual is assigned to B∆A, and hence

5

Algorithm 1. Algorithm 1 gives the conceptual form of LoRA-TSD. It contains the full mathematical update: compute the tangentprojected gradient, refine it with the tangent spectral step, and map the resulting tangent update back to LoRA factors through Eqs. (16) and (18). It omits momentum and engineering details, but this is the form analyzed in Section 5.

Algorithm 1: LoRA-TSD (conceptual form) 1: for k = 0, 1, . . . do

GkA ← ∇A L(Wk ), GkB ← ∇B L(Wk ) 3: X0 ← PT (GkW ) ▷ Eq. (14) 4: for t = 0, . . . , τ − 1 do
5: Xt+1 ← PT msign(Xt ) ▷ Eq. (12) 6: end for 7: ∆Wk ← −ηXτ ⊤ −1 8: ∆Bk ← ∆Wk A⊤ k (Ak Ak ) 9: ∆Ak ← (Bk⊤ Bk )−1 Bk⊤ (∆Wk − ∆Bk Ak ) 10: Ak+1 ← Ak + ∆Ak , Bk+1 ← Bk + ∆Bk 11: end for 2:

Discussion. For τ = 1, Algorithm 1 recovers the tangent-direction part of Riemannion [Bogachev et al., 2025]. Table 4 shows that using τ from 3 to 5 improves average downstream accuracy over τ = 1. LoRA-TSD maps the resulting direction through a LoRA-native factor retraction rather than the truncated-SVD retraction, which keeps the update in the factor parametrization and avoids an expensive SVD. Appendix B compares the resulting iterates in detail. Transformation invariance. Since the update is defined in weight space, it should not depend on which LoRA factorization represents the same adapter. Following LoRA-Rite [Yen et al., 2024], we call a LoRA optimizer transformation invariant as follows. Let (Bk , Ak ) be a pair of LoRA factors and let Bk′ = Bk R−1 , A′k = RAk for some invertible R ∈ Rr×r . If one optimizer step gives updates (∆Bk , ∆Ak ) and (∆Bk′ , ∆A′k ), then the optimizer is transformation invariant if (Bk + ∆Bk )(Ak + ∆Ak ) = (Bk′ + ∆Bk′ )(A′k + ∆A′k ) := Bk Ak + ∆Wk . Proposition 1 LoRA-TSD (Algorithm 1) is transformation invariant. This property is induced by the manifold view. The tangent projector, the projected gradient PT (GkW ), and the spectral tangent step Xτ depend only on the point Wk ∈ W0 + Mr , not on the particular LoRA factors used to represent it. We next show how to implement the same update in practice without expensive operations on m × n matrices. Factorization trick. The closed form in Eq. (14) still appears to define an m × n matrix. In practice we never materialize it. Regrouping the tangent projector of Eq. (13) as PT (Z) = PB Z(I − PA ) + ZPA

(19)

exhibits every tangent matrix as a sum of two rank-r terms, so it has rank at most 2r. We therefore store each iterate of Eq. (12) in a factored form Xt = Lt Rt with Lt ∈ Rm×2r and Rt ∈ R2r×n . For the initialization X0 = PT (GW ), applying Eq. (19) with Z = GW gives X0 = L0 R0 ,

(20)

where 

⊤ −1

L0 = B, GB (AA )

 ,

 (B ⊤ B)−1 GA (I − PA ) R0 = . A 

6

(21)

Here GA (I − PA ) = GA − (GA A⊤ )(AA⊤ )−1 A stays r × n, so no m × n product is ever formed. The matrix sign step is also applied through small matrices. Given Xt = Lt Rt , compute thin QR decompositions bL and Rt⊤ = QR R bR . Then Lt = QL R ⊤ bL R bR msign(Xt ) = QL msign(R ) Q⊤ R,

(22)

bL R b⊤ ) with a Newton– so the matrix sign reduces to a single 2r × 2r problem. In practice we compute msign(R R bL R b⊤ ) = U V ⊤ , Schulz iteration rather than an explicit SVD, as in Muon [Jordan et al., 2024]. Writing msign(R R e t = QL U and R et = V ⊤ Q⊤ . we set L R Finally, we project the msign output back onto the tangent space, again without forming the m × n product et R et . Using the two-term projector (19) and the small matrices L e t ∈ Rr×2r , Ct = B ⊤ L

et A⊤ ∈ R2r×r , Dt = R

et R et , the two projector terms become, with Z = L et )(I − PA ), PB Z(I − PA ) = B(B ⊤ B)−1 (Ct R e t Dt )(AA⊤ )−1 A, ZPA = (L neither of which forms an m × n matrix. Collecting the shared factors gives the projection directly in factored et R et ) = Lt+1 Rt+1 with form, PT (L  ⊤ −1  et (I − PA )   (B B) Ct R ⊤ −1 e Lt+1 = B, Lt Dt (AA ) , Rt+1 = . (23) A This mirrors the block structure of the initialization L0 , R0 in Eqs. (20)–(21). Hence the alternating projection loop reuses the Gram inverses (B ⊤ B)−1 and (AA⊤ )−1 already formed for X0 , stores only the thin factors Lt and Rt of size m × 2r and 2r × n together with the small 2r × 2r matrix used for the msign step, and never allocates an m × n matrix. Algorithm 2. Algorithm 2 is the practical implementation of LoRA-TSD. It adds momentum to Algorithm 1 and replaces each explicit tangent matrix by its factored representation Xt = Lt Rt from Eqs. (20)– (23). This factorization is only an implementation device: the underlying tangent update is unchanged, but all storage and multiplications stay in the thin factors, avoiding expensive operations on m × n matrices.

4

Experiments

Setup. We evaluate two models, Llama3.2-1B-Instruct and Llama-3.1-8B, on BoolQ [Clark et al., 2019], PIQA [Bisk et al.,

Algorithm 2: LoRA-TSD with factorization trick 0 , M 0 ← 0, 0 1: MA B

2: for k = 1, 2, . . . do

GkA ← ∇A L(Wk ), GkB ← ∇B L(Wk ) 4: MAk ← µMAk−1 + (1 − µ)GkA 5: MBk ← µMBk−1 + (1 − µ)GkB Factorized  projection onto  TWk Mr −1 6: L0 ← Bk , MBk (Ak A⊤ ) k  7: R0 ← (Bk⊤ Bk )−1 MAk (I − PAk ); Ak 8: for t = 0, . . . , τ − 1 do Factorized msign bL , R⊤ = QR R b 9: Lt = QL R t
R ⊤ ⊤ b b 10: U V ← msign RL RR ▷ Newton–Schulz ⊤ ⊤ e e 11: Lt ← QL U , Rt ← V QR Factorized projection e , Dt ← R e t A⊤ 12: Ct ← Bk⊤ L k  t  e t Dt (Ak A⊤ )−1 13: Lt+1 ← Bk , L k   et (I − PA ); Ak 14: Rt+1 ← (Bk⊤ Bk )−1 Ct R k 15: end for Retraction 16: ∆Wk ← −ηLτ Rτ ⊤ −1 17: ∆Bk ← ∆Wk A⊤ k (Ak Ak ) ⊤ −1 ⊤ 18: ∆Ak ← (Bk Bk ) Bk (∆Wk − ∆Bk Ak ) 19: Ak+1 ← Ak + ∆Ak , Bk+1 ← Bk + ∆Bk 20: end for 3:

7

2020], SIQA [Sap et al., 2019], OBQA [Mihaylov et al., 2018], QNLI [Wang et al., 2019], and MultiNLI [Williams et al., 2018]. We also evaluate Qwen3-32B [Yang et al., 2025] on SIQA and OBQA. These benchmarks test natural-language understanding through classification and multiple-choice questions, covering reading comprehension, physical and social commonsense, elementary science, and textual entailment. Such classification-style benchmarks are standard in the LoRA literature [Yen et al., 2024; Tastan et al., 2025; Liu et al., 2024]. To isolate optimizer effects, all methods use Riemannion’s locally optimal initialization (LOI), which improves average accuracy over standard initialization [Bogachev et al., 2025]. Appendix C specifies the adapter placement, preprocessing, and hyperparameters. Dataset

SGD

AdamW

BoolQ 80.93±0.21 PIQA 76.23±0.29 SIQA 71.97±0.35 OBQA 70.87±0.50 QNLI 91.03±0.35 MultiNLI 53.07±27.31

80.90±0.80 75.77±0.35 71.03±0.32 72.20±0.20 90.80±0.44 83.33±0.50

AVG

Muon Riem-SGD LoRA-Rite Riemannion LoRA-Pro LoRA-Muon LoRA-TSD Muon FT 83.07±0.40 79.70±0.35 83.03±0.06 78.10±0.50 64.43±12.19 78.80±0.35 73.90±0.50 43.80±2.26 74.37±0.32 71.87±0.76 46.80±1.06 74.00±1.00 92.27±0.06 82.30±4.12 92.80±0.26 85.90±0.53 74.70±5.19 86.17±0.40

74.02±4.42 79.01±0.14 80.85±0.09

65.29±1.39

81.53±0.25

81.73±0.21 77.40±0.52 72.10±0.30 72.07±0.81 91.57±0.23 84.63±0.31

80.07±0.49 74.23±0.23 68.53±0.50 71.33±0.42 89.30±0.20 83.02±1.14

83.47±0.59 79.17±0.51 73.93±0.12 73.73±1.45 92.93±0.21 86.37±0.06

84.63±0.42 80.53±0.25 75.40±0.35 74.00±1.20 93.07±0.42 87.33±0.06

83.10±0.30 79.20±0.66 73.67±0.46 71.93±0.95 93.23±0.25 86.63±0.06

79.92±0.27 77.75±0.19

81.60±0.32

82.49±0.16 81.29±0.24

Table 1: Accuracy on commonsense and NLI benchmarks with Llama-3.2-1B-Instruct, reported as mean ± standard deviation over three runs. Bold indicates the best LoRA result per dataset.

Baselines. We compare against standard factor-space optimizers, SGD, AdamW, and Muon applied independently to the two LoRA factors. We include three manifold methods, LoRA-Muon, Riemannian SGD, and Riemannion. LoRA-Rite and LoRA-Pro cover LoRA-specific optimizer designs. Together, these baselines cover much of the LoRA optimization design space, including direct factor-space updates, manifold optimization, gradient preconditioning, and weight-space correction. Finally, Muon FT fine-tunes the same adapted projections with Muon and serves as a full-tuning reference. Dataset

SGD AdamW Muon Riem-SGD LoRA-Rite Riemannion LoRA-Pro LoRA-Muon LoRA-TSD Muon FT

BoolQ PIQA SIQA OBQA QNLI MultiNLI

88.6 89.1 82.2 86.0 94.8 89.9

89.6 89.2 81.8 86.8 94.3 89.5

89.1 87.9 81.2 86.0 94.8 89.7

88.8 88.3 34.1 51.4 93.7 89.9

88.9 88.4 81.4 85.6 95.0 90.3

88.5 86.9 79.4 84.0 93.5 88.1

88.3 86.3 77.9 86.8 90.2 88.2

89.8 89.2 81.8 86.6 95.1 90.8

90.0 90.1 82.7 88.6 96.1 90.9

89.3 89.1 80.8 86.6 95.1 90.5

AVG

88.5

88.5

88.1

74.4

88.3

86.8

86.3

88.9

89.7

88.5

Table 2: Accuracy on commonsense and NLI benchmarks with Llama-3.1-8B. Bold indicates the best LoRA result per dataset.

Results. Tables 1, 2, and 3 report accuracy on the three models. Across the two Llama models, LoRA-TSD is best or tied for best among LoRA optimizers in all 12 model-dataset settings and achieves the highest average at both model scales. Notably, it also outperforms Muon Full-FT in 11 of the 12 settings, despite updating only low-rank adapters. Full parameter access therefore does not consistently translate into better downstream accuracy than a spectral update constrained to the LoRA tangent space. LoRA-Muon is the strongest competing LoRA method on average at both scales, although its average accuracy varies about twice as much across seeds as LoRA-TSD at the 1B scale. Both SGD baselines are less stable. Standard SGD has high variance on MultiNLI at the 1B scale. Riemannian SGD fails to converge reliably on SIQA and OBQA at both scales and is unstable on several additional 1B tasks. We tuned both with the same protocol as every other method, described in Appendix C, and report the measured results without additional tuning. Table 3 8

reports Qwen3-32B results on SIQA and OBQA, the two lowest-accuracy Llama-3.1-8B benchmarks in Table 2. Qwen3-32B uses the transferred 8B hyperparameters with minimal tuning, gradient clipping reduced from 1 to 0.25, a frozen FP8-E4M3FN backbone, and FP32 LoRA parameters and optimizer states. LoRA-TSD and LoRA-Rite are the two strongest methods on both tasks. Dataset

AdamW

Muon

LoRA-Muon

LoRA-Rite

LoRA-TSD

SIQA OBQA

84.3 94.6

84.2 94.4

84.4 95.0

84.8 95.6

85.0 96.0

Number of inner iterations. Table 4 shows a trade-off between quality and training cost. With τ = 1, LoRA-TSD is 0.92 average accuracy points below τ = 3, but runs faster than Riemannion because its factor-induced retraction is cheaper than truncated SVD (Table 5). Increasing τ to 3 gives the best average accuracy at the cost of two additional inner projections per training step. Increasing τ further to 4 or 5 provides no noticeable performance gain.

SIQA accuracy

Table 3: Accuracy on SIQA and OBQA with Qwen3-32B. Bold indicates the best result per dataset.

84 83 82 81 80 79 78 77

LoRA-TSD LoRA-Rite Muon AdamW

1

2

4

8

16

32

64

LoRA rank r

Rank ablation. Figure 1 shows that LoRA-TSD maintains Figure 1: Rank ablation on SIQA (Llama-3.1the most stable high-accuracy performance across ranks on 8B). LoRA-TSD maintains high accuracy across SIQA. Appendix C provides a deeper comparison on BoolQ ranks. and SIQA. τ

BoolQ

PIQA

SIQA

AVG

1 2 3 4 5

84.16 84.40 84.89 84.68 85.08

79.05 79.65 80.20 80.14 79.87

75.59 74.51 76.46 75.90 76.46

79.60 79.52 80.52 80.24 80.47

Table 4: Accuracy with Llama-3.2-1B-Instruct for different numbers τ of alternating-projection steps. Bold marks the best result in each column.

5 5.1

Theory Convergence guarantees

To our knowledge, LoRA-TSD is the first LoRA-specific optimizer with a stationary-point convergence guarantee for the nonconvex fixed-rank LoRA problem, expressed through the tangent-projected gradient. Existing convergence analyses of LoRA study standard gradient methods on the factorized objective. Mu and Klabjan [2026] establish a convergence rate for deterministic LoRA gradient descent, while Wang et al. [2026] derive guarantees for deterministic and stochastic variants. LoRA-RITE [Yen et al., 2024] provides a different result: an online-optimization bound on an averaged first-order term, strengthened under an additional smooth-change assumption on its preconditioners. Our analysis instead directly bounds the tangent-projected gradient, the natural stationarity measure for the fixed-rank problem. We next state the assumptions and define this measure formally. 9

Assumption 1 (Spectral smoothness and bounded gradients) The gradient ∇L is Lipschitz continuous with respect to the spectral norm: ∥∇L(W ) − ∇L(W ′ )∥∗ ≤ M ∥W − W ′ ∥2 . Moreover, ∥∇L(W )∥2 ≤ Gmax along the trajectory. Assumption 2 (Lower-bounded objective) The objective is bounded from below: L ≥ L∗ . Assumption 3 (Uniform factor nondegeneracy ) The LoRA factors remain full rank along the trajectory: there exists δ > 0 such that σr (Ak ), σr (Bk ) ≥ δ > 0 ∀ k. Assumption 4 (Uniform descent alignment) For some c0 > 0,

⟨PTk (GkW ), Xτ ⟩ ≥ c0 ∥PTk (GkW )∥∗ ,

where Xτ is the alternating-projection output. Discussion of assumptions. Assumptions (A1) and (A2), together with the step-size bound, are standard in non-convex optimization [Ghadimi and Lan, 2013; Bottou et al., 2018]. Assumption (A3) keeps the LoRA factors away from rank deficiency. This condition must account for the gauge ambiguity because the same residual Bk Ak can be represented with one factor arbitrarily small and the other arbitrarily large. We remove this artificial scale imbalance by periodically applying a gauge transformation that preserves Bk Ak , and hence Wk , while rebalancing the factor norms: s ∥Bk ∥F ck = , Ak ← ck Ak , Bk ← c−1 k Bk . ∥Ak ∥F We rescale the momentum buffers as MA ← c−1 k MA and MB ← ck MB to preserve the induced update direction. Figure 2a shows that the smallest singular values σr (Ak ) and σr (Bk ) remain above a positive δ throughout training, as required by (A3). The last condition, (A4), requires the approximate tangent oracle to retain a uniform descent component. For τ = 1, it holds with c0 = 1 because PT = PT∗ and PT (g) = g, ⟨g, X1 ⟩ = ⟨g, PT (msign(g))⟩ = ⟨PT (g), msign(g)⟩ = ⟨g, msign(g)⟩ = ∥g∥∗ . For τ > 1, (A4) can be enforced by returning bτ ∈ arg max⟨g, Xt ⟩. X 1≤t≤τ

Since X1 is included, this safeguard gives c0 = 1. It does not noticeably improve downstream performance, so we use Xτ and verify (A4) empirically in Figure 2b.

10

1 ⟨gk , Xτ ⟩F /∥gk ∥∗

minℓ σr

0.12 0.09 0.06 σr (Ak ) σr (Bk ) δ

0.03 0 250 500

1,000

1,500

2,000

2,500

0.9 0.8 0.7 0.6 0.5

3,000

q_proj k_proj v_proj minimum

0

500

training step k

1,000

1,500

2,000

training step k

(a) Minimum factor singular values across adapted layers. Both remain above δ.

(b) Descent ratio for selected projection types with τ = 3, where gk denotes PTk (GkW ). The dashed curve is the minimum across all 96 adapted matrices.

Figure 2: Empirical checks of Assumptions (A3) and (A4) during LoRA-TSD training on BoolQ with Llama3.1-8B. Stationarity in the LoRA setting. Because Wk is confined to the affine shift of the manifold Mr , the relevant notion of a stationary point is not GkW = 0, which is generally unattainable under a rank constraint, but the vanishing of the gradient’s component tangent to Mr , i.e. PT (GkW ) = 0. This tangential stationarity condition is the criterion used in the convergence bounds below. Convergence rate. For LoRA-TSD, which operates in the spectral-norm geometry matching the Muon update, we have the following. Theorem 1 (Convergence of LoRA-TSD) Let {Wk } be the iterates produced by Algorithm 1 (the momentum-free variant, Algorithm 2 differs only √ in the addition of momentum). Under assumptions (A1)–(A4) and step sizes 0 < ηk ≤ min 1, δ 2 /(2 2) , for any N ≥ 1: 1. Constant step size. If ηk ≡ η, then min ∥PTk (GkW )∥∗ ≤

0≤k<N

L(W0 ) − L∗ C + η. c0 ηN c0

and in particular, for η =

p (L(W0 ) − L∗ )/(CN ), r
2 C(L(W0 ) − L∗ ) min ∥PTk (GkW )∥∗ ≤ = O N −1/2 . 0≤k<N c0 N

P P P 2. Diminishing step size. If k ηk = ∞ and k ηk2 < ∞, then k ηk ∥PTk (GkW )∥∗ < ∞, and consequently limk→∞ ∥PTk (GkW )∥∗ = 0. Remark 1 If the factor spectral norms are uniformly bounded, then the diminishing-step guarantee also implies ∥GkA ∥∗ , ∥GkB ∥∗ → 0. See Appendix A. Directional equivalence with LoRA-Pro. LoRA-Pro is a popular and intuitive weight-space correction, so recovering it places our surrogate within an established LoRA optimization framework. 11

Proposition 2 (Directional equivalence with LoRA-Pro) Under the Frobenius norm, let XF and XLoRA-Pro denote the weight-space directions induced by Eq. (10) and LoRA-Pro, respectively. Then η XLoRA-Pro = XF . λ The updates coincide when η = λ. This equivalence transfers the convergence analysis to LoRA-Pro, with ∥ · ∥F replacing ∥ · ∥∗ . Theorem 2 (Convergence of LoRA-Pro) Under the Frobenius-norm analogues of assumptions (A1)–(A3), for the canonical X = 0 LoRA-Pro representative defined in Appendix A, with step size η ≤ min{1, 1/(2C)} and any N ≥ 1,
min ∥PTk (GkW )∥F = O N −1/2 . 0≤k<N

Moreover, for the constant step size η = min{1, 1/(2C)}, the iterates satisfy limk→∞ ∥PTk (GkW )∥F = 0. Proofs and constant definitions are given in Appendix A.

6

Conclusion

We presented LoRA-TSD, which performs Muon-style spectral descent in the LoRA tangent space through a factor-induced retraction. We established convergence to projected stationarity and recovered LoRA-Pro as the Frobenius counterpart with an analogous guarantee. Across six benchmarks and three model scales from 1B to 32B, LoRA-TSD outperforms generic and LoRA-specific optimizers and remains stable across adapter ranks. The approximate inner oracle remains the main limitation because it lacks an optimality guarantee. Controlled bounds and generative evaluation remain future work.

References Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2403.14608. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2106.09685. Michael Nguyen, Dhawalkumar Patel, Qing Lan, and Vivek Gangasani. Efficient and cost-effective multi-tenant LoRA serving with Amazon SageMaker. AWS Machine Learning Blog, May 2024. Accessed September 2, 2026. Nicholas Scozzafava, Deepam Mishra, Dmitry Soldatkin, Sam Sudakoff, Vivek Gangasani, Tatia Tsmindashvili, Ana Kolkhidashvili, Guram Dentoshvili, and Dachi Choladze. Impel enhances automotive dealership customer experience with fine-tuned LLMs on Amazon SageMaker. AWS Machine Learning Blog, June 2025. Accessed September 2, 2026. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314.

12

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023. URL https://arxiv.org/abs/2303.10512. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024. URL https://arxiv.org/abs/2404.02948. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024. URL https://arxiv.org/abs/2402.12354. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024. URL https://arxiv.org/abs/ 2402.09353. Andrey Veprikov, Vladimir Solodkin, Alexander Zyl, Andrey V. Savchenko, and Aleksandr Beznosikov. Weightlora: Keep only necessary adapters. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 12419–12437. Association for Computational Linguistics, 2026. doi: 10.18653/V1/2026. ACL-LONG.566. URL https://doi.org/10.18653/v1/2026.acl-long.566. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. URL https://arxiv.org/abs/1412.6980. Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized?, 2025. URL https://arxiv.org/abs/2407.18242. Nurbek Tastan, Stefanos Laskaridis, Martin Takac, Karthik Nandakumar, and Samuel Horvath. Loft: Low-rank adaptation that behaves like full fine-tuning, 2025. URL https://arxiv.org/abs/2505.21289. Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S. Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. Lora done RITE: Robust invariant transformation equilibration for LoRA optimization, 2024. URL https://arxiv.org/abs/2410.20625. Silvère Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013. doi: 10.1109/TAC.2013.2254619. Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation models. In International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2402.02347. Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Denis Bobkov, Vera Soboleva, Aibek Alanov, and Maxim Rakhuba. Lora meets riemannion: Muon optimizer for parametrization-independent low-rank adapters, 2025. URL https://arxiv.org/abs/2507.12142. Franz Louis Cesista, Katherine Crowson, Cédric Simal, and Stella Biderman. Lora-muon: Spectral steepest descent on the low-rank manifold, 2026. URL https://arxiv.org/abs/2606.12921. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation, 2024. URL https://arxiv.org/abs/2407.05000. Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology, 2024. URL https://arxiv. org/abs/2409.20325. 13

P.-A. Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008. Bart Vandereycken. Low-rank matrix completion by riemannian optimization. SIAM Journal on Optimization, 23(2):1214–1236, 2013. doi: 10.1137/110845768. URL https://arxiv.org/abs/1209.3834. Nikhil Ghosh, Tetiana Parshakova, and Robert M. Gower. PoLoRA: A preconditioned orthogonalized LoRA optimizer, 2026. URL https://arxiv.org/abs/2607.17620. Heinz H. Bauschke and Jonathan M. Borwein. On projection algorithms for solving convex feasibility problems. SIAM Review, 38(3):367–426, 1996. doi: 10.1137/S0036144593251710. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. URL https://arxiv.org/abs/1905.10044. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020. URL https://arxiv.org/abs/1911.11641. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. URL https://arxiv.org/abs/1904.09728. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. URL https://arxiv.org/abs/1809.02789. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019. URL https://arxiv.org/abs/1804.07461. Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018. URL https://arxiv.org/abs/1704.05426. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Siqiao Mu and Diego Klabjan. On the convergence rate of lora gradient descent, 2026. URL https://arxiv. org/abs/2512.18248. Ru Wang, Chengchang Liu, and John C. S. Lui. On the convergence of stochastic low-rank adaptation, 2026. URL https://arxiv.org/abs/2607.21975. Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. doi: 10.1137/120880811. 14

Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223–311, 2018. doi: 10.1137/16M1080173. Per-Åke Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics, 12(1):99–111, 1972. doi: 10.1007/BF01932678. Åke Björck and Gene H. Golub. Numerical methods for computing angles between linear subspaces. Mathematics of Computation, 27(123):579–594, 1973. doi: 10.1090/S0025-5718-1973-0348991-3. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016. URL https://arxiv.org/abs/1606.05250.

15

Appendix Supplementary Materials for LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

A

Proofs

A.1

Preliminaries

We state two classical results used throughout the proofs. In both theorems M̂ = M + E. Definition 1 (Principal angles) Let U, V ⊆ Rn be subspaces of equal dimension r. The principal angles 0 ≤ θ1 ≤ · · · ≤ θr ≤ π/2 between U and V are defined recursively by cos θk =

max

u∈U , v∈V ∥u∥=∥v∥=1

u⊤ v

s.t. u ⊥ u1 , . . . , uk−1 , v ⊥ v1 , . . . , vk−1 . Equivalently, if U, V ∈ Rn×r are orthonormal bases for U and V, then cos θk = σk (U ⊤ V ). The largest principal angle θr satisfies sin θr = ∥(I − U U ⊤ )V ∥2 . Theorem 3 (sin Θ theorem [Wedin, 1972] ) Let M, M̂ ∈ Rm×n with M̂ = M + E. Let U0 , V0 (resp. Û0 , V̂0 ) be orthonormal bases for the leading r left and right singular subspaces of M (resp. M̂ ). Suppose σr (M ) − σr+1 (M̂ ) ≥ ∆ > 0. Then   max ∥EV0 ∥2 , ∥E ⊤ U0 ∥2 ∥E∥2 max dist(Û0 , U0 ), dist(V̂0 , V0 ) ≤ ≤ , ∆ ∆ where dist(X̂0 , X0 ) := ∥(I − X0 X0⊤ )X̂0 ∥2 is the sine of the largest principal angle between the subspaces. Theorem 4 (Weyl’s inequality ) Let M, M̂ ∈ Rm×n with M̂ = M + E. Then |σi (M ) − σi (M̂ )| ≤ ∥E∥2 ,

∀ i = 1, . . . , min(m, n).

Consequently, σr (M ) − σr+1 (M̂ ) ≥ σr (M ) − σr+1 (M ) − ∥E∥2 . Notation. At each outer iteration k the algorithm produces: • GkW := ∇L(Wk ) is the full-space gradient (never formed explicitly). • GkA := Bk⊤ GkW and GkB := GkW A⊤ k are the factor gradients, i.e. ∂L/∂Ak and ∂L/∂Bk . • gk := PTk (GkW ) is the projection of the gradient onto the tangent space Tk = TWk Mr , which serves as the stationarity measure. 16

• X0 := PTk (GkW ) = gk initializes the inner loop. • Xt := PTk (msign(Xt−1 )), t = 1, . . . , τ , are the alternating-projection iterates, and Xτ is the final inner-loop output used as the update direction. • ∆Wk := −ηk Xτ is the full-space weight increment. • ∆Bk , ∆Ak are the factor increments recovered from ∆Wk via the canonical split (Eqs. (16)–(18)). We also define the constants c1 :=

A.2

1 , δ2

c2 := 1 +

√

2 c1 ,

C := 2Gmax rc1 + M c22 .

Proof of Proposition 1

We track the state through one pass of Algorithm 1 and show that each step produces the same result for (B, A) and for the reparametrized factors (B ′ , A′ ) = (BR−1 , RA), which by construction satisfy B ′ A′ = BA = W − W0 . We drop the iteration index k and write η for the step size. The two factorizations describe the same weight W , so GW = ∇W L(W ) is identical for both. The factor gradients differ, GA′ = (B ′ )⊤ GW = R−⊤ GA , GB ′ = GW (A′ )⊤ = GB R⊤ , but they enter the algorithm only through GW , as we now use. Consider the tangent projectors of Eq. (13) for the primed factors. Using (A′ (A′ )⊤ )−1 = R−⊤ (AA⊤ )−1 R−1 and ((B ′ )⊤ B ′ )−1 = R(B ⊤ B)−1 R⊤ , PA′ = (A′ )⊤ (A′ (A′ )⊤ )−1 A′ = A⊤ R⊤ R−⊤ (AA⊤ )−1 R−1 RA = PA , PB ′ = B ′ ((B ′ )⊤ B ′ )−1 (B ′ )⊤ = BR−1 R(B ⊤ B)−1 R⊤ R−⊤ B ⊤ = PB . Hence PT ′ = PT : the projector, and therefore the tangent space itself, depends only on W . The inner loop is unaffected as well. The initialization is X0 = PT (GW ), which is unchanged since both PT and GW are. The iteration Xt+1 = PT (msign(Xt )) applies the same operator PT and the reparametrization-free map msign, so by induction every Xt , and in particular Xτ , is identical for both factorizations. Consequently the tangent update ∆W = −ηXτ is identical. The reconstruction step transforms covariantly. With A† = A⊤ (AA⊤ )−1 and B † = (B ⊤ B)−1 B ⊤ , the same manipulations as above give (A′ )† = A† R−1 and (B ′ )† = RB † . Hence ∆B ′ = ∆W (A′ )† = ∆B R−1 , and, since ∆B ′ A′ = ∆BR−1 RA = ∆BA, ∆A′ = (B ′ )† (∆W − ∆B ′ A′ ) = R B † (∆W − ∆BA) = R ∆A. Finally, the updated factors are B ′ + ∆B ′ = (B + ∆B)R−1 and A′ + ∆A′ = R(A + ∆A), so (B ′ + ∆B ′ )(A′ + ∆A′ ) = (B + ∆B)R−1 R(A + ∆A) = (B + ∆B)(A + ∆A). ′ Adding W0 gives Wk+1 = Wk+1 , which proves the claim.

17

A.3

Proof of Proposition 2

Consider the Frobenius-norm version of the tangent surrogate Eq. (9): n o 1 min ⟨GA , ∆A⟩ + ⟨GB , ∆B⟩ + ∥∆BA + B∆A∥2F . ∆A,∆B 2λ The linear terms rewrite as ⟨GA , ∆A⟩ + ⟨GB , ∆B⟩ = ⟨B ⊤ GW , ∆A⟩ + ⟨GW A⊤ , ∆B⟩ = ⟨GW , ∆BA + B∆A⟩. e := ∆BA + B∆A and completing the square, Setting G e + ⟨GW , G⟩

1 e 2 1 e λ ∥G∥F = ∥G + λGW ∥2F − ∥GW ∥2F . 2λ 2λ 2

Dropping the constant, the problem reduces to min ∥∆BA + B∆A + λGW ∥2F .

∆A,∆B

b A := −∆A/λ and G b B := −∆B/λ gives Substituting G bA + G b B A − G W ∥2 . ∥∆BA + B∆A + λGW ∥2F = λ2 ∥B G F Since λ2 > 0 does not affect the minimizers, the problem is equivalent to bA + G b B A − G W ∥2 , min ∥B G F

b A ,G bB G

which is precisely the auxiliary problem solved by LoRA-Pro. Its induced weight-space direction is the orthogonal projection b⋆ + G b ⋆ A = PT (GW ). BG A B It follows from the substitution above that the surrogate solution is X ⋆ = −λPT (GW ). By contrast, a LoRA-Pro step with learning rate η induces −ηPT (GW ) = (η/λ)X ⋆ . Thus the directions agree for all positive η and λ, while the increments are equal when η = λ.

A.4

Setup for Theorem 1

Assumptions. We work under (A1)–(A4) and the theorem’s step-size condition, as stated in Section 5.1. For convenience we recall the problem-specific ones. (A3) requires the LoRA factors to stay full-rank: σr (Ak ), σr (Bk ) ≥ δ > 0. (A4) is the descent-margin condition on the alternating-projection output: ⟨gk , Xτ ⟩ ≥ c0 ∥gk ∥∗ , √ The step-size condition requires 0 < ηk ≤ min{1, δ 2 /(2 2)}. The following lemmas are used in the proof of Theorem 1.

c0 > 0.

Lemma 1 (Row-space projector perturbation) Let A ∈ Rr×n have rank r with σr (A) > 0, and let A′ = A + E satisfy ∥E∥2 ≤ σr (A)/2. Denote by PA = A⊤ (AA⊤ )−1 A and PA′ = (A′ )⊤ (A′ (A′ )⊤ )−1 A′ the orthogonal projectors onto the row spaces of A and A′ , respectively. Then ∥E∥2 ∥PA′ − PA ∥2 ≤ . σr (A)

18

Proof. Let θmax denote the largest principal angle between the row spaces of A and A′ . Write PA′ − PA = (I − PA )PA′ − PA (I − PA′ ). For any unit vector x, (I − PA )PA′ x ∈ col(I − PA ) and PA (I − PA′ )x ∈ col(PA ) are orthogonal, so ∥(PA′ − PA )x∥2 = ∥(I − PA )PA′ x∥2 + ∥PA (I − PA′ )x∥2 . Let QA′ be any orthonormal basis for the row space of A′ . By Björck and Golub [1973] (eq. 13), ∥(I −PA )PA′ ∥2 = ∥(I − PA )QA′ ∥2 = sin θmax , and by symmetry ∥PA (I − PA′ )∥2 = sin θmax . Bounding each term by the corresponding spectral norm, ∥(I − PA )PA′ x∥2 ≤ ∥(I − PA )PA′ ∥22 ∥PA′ x∥2 = sin2 θmax ∥PA′ x∥2 , ∥PA (I − PA′ )x∥2 ≤ ∥PA (I − PA′ )∥22 ∥(I − PA′ )x∥2 = sin2 θmax ∥(I − PA′ )x∥2 , and using ∥PA′ x∥2 + ∥(I − PA′ )x∥2 = ∥x∥2 = 1,
∥(PA′ − PA )x∥2 ≤ sin2 θmax ∥PA′ x∥2 + ∥(I − PA′ )x∥2 = sin2 θmax , with equality at the corresponding principal vector, so ∥PA′ − PA ∥2 = sin θmax . Apply Theorem 3 with M = A, M̂ = A′ , taking V0 to be the orthonormal basis for the row space of A. Since dist(V̂0 , V0 ) = sin θmax and ∆ = σr (A) − σr+1 (A′ ) = σr (A), sin θmax ≤

∥E∥2 ∥E∥2 = . ∆ σr (A)

Lemma 2 (Spectral norm of the alternating-projection output) √ For every outer iteration k, σ1 (Xτ ) ≤ 2. Proof. For any unit vector x, the two components PB Zx and (I − PB )ZPA x of PT (Z)x are orthogonal. Hence ∥PT (Z)x∥22 = ∥PB Zx∥22 + ∥(I − PB )ZPA x∥22 ≤ ∥Z∥22 (1 + ∥PA x∥22 ) ≤ 2∥Z∥22 . Since Xτ = PTk (msign(Xτ −1 )) and ∥msign(Xτ −1 )∥2 = 1, we get σ1 (Xτ ) ≤

√

2.

Lemma 3 (Factor-remainder bound ) Under (A3), the factor increments satisfy ∥∆Bk ∥2 ≤ and consequently

σ1 (∆Wk ) , σr (Ak )

∥∆Ak ∥2 ≤

σ1 (∆Wk ) , σr (Bk )

∥∆Bk ∆Ak ∥2 ≤ c1 σ1 (∆Wk )2 ≤ 2c1 ηk2 ,

where c1 = 1/δ 2 . Proof. Recall the canonical factor-update split (Eqs. (16)–(18)): ∆Bk = ∆Wk A†k ,

∆Ak = Bk† (∆Wk − ∆Bk Ak ),

⊤ −1 and B † = (B ⊤ B )−1 B ⊤ are the right and left Moore–Penrose pseudoinverses, where A†k = A⊤ k (Ak Ak ) k k k k satisfying ∥A†k ∥2 = 1/σr (Ak ) and ∥Bk† ∥2 = 1/σr (Bk ).

19

By submultiplicativity, ∥∆Bk ∥2 ≤ ∥∆Wk ∥2 ∥A†k ∥2 =

σ1 (∆Wk ) . σr (Ak )

Since ∆Bk Ak = ∆Wk PAk , the second increment satisfies ∆Ak = Bk† ∆Wk (I − PAk ). Therefore, ∥∆Ak ∥2 ≤ ∥Bk† ∥2 ∥∆Wk ∥2 ∥I − PAk ∥2 ≤

σ1 (∆Wk ) . σr (Bk )

Multiplying the two bounds and using σr (Ak ), σr (Bk ) ≥ δ, ∥∆Bk ∆Ak ∥2 ≤ ∥∆Bk ∥2 ∥∆Ak ∥2 ≤ Finally, σ1 (∆Wk ) = ηk σ1 (Xτ ) ≤

σ1 (∆Wk )2 1 ≤ 2 σ1 (∆Wk )2 = c1 σ1 (∆Wk )2 . σr (Ak )σr (Bk ) δ

√ 2 ηk by Lemma 2, so c1 σ1 (∆Wk )2 ≤ 2c1 ηk2 .

Lemma 4 (Lipschitz continuity of ∥gk ∥∗ ) Under (A1)–(A4) and the step-size condition, ∥gk+1 ∥∗ − ∥gk ∥∗ ≤ L′ ηk , where

! √ 2 2G max L′ = 4r 2M c2 + . δ2

Proof. Decompose k+1 k k k gk+1 − gk = PTk+1 (Gk+1 W ) − PTk (GW ) = PTk+1 (GW − GW ) + (PTk+1 − PTk )(GW ).

By Lemma 2 and spectral Lipschitz continuity of ∇L, k PTk+1 (Gk+1 W − GW )

2

≤ ≤

√ √ √

k 2 ∥Gk+1 W − G W ∥2

2 M ∥Wk+1 − Wk ∥2

2 M ∥∆Bk Ak + Bk ∆Ak + ∆Bk ∆Ak ∥2
≤ 2 M ∥∆Wk ∥2 + ∥∆Bk ∆Ak ∥2 √ √
≤ 2 M 2 ηk + 2c1 ηk2 √ ≤ 2M ηk (1 + 2 c1 ) = 2M c2 ηk ,

=

√

√ where we used ∥∆Wk ∥2 = ηk σ1 (Xτ ) ≤ 2 ηk (Lemma 2), ∥∆Bk ∆Ak ∥2 ≤ c1 σ1 (∆Wk )2 ≤ 2c1 ηk2 (Lemma 3), and ηk ≤ 1 (the step-size condition). Recall PT (Z) = PB Z + ZPA − PB ZPA . Define ∆PA := PAk+1 − PAk and ∆PB := PBk+1 − PBk . Then (PTk+1 − PTk )(GkW ) = ∆PB GkW + GkW ∆PA − PBk+1 GkW PAk+1 − PBk GkW PAk

= ∆PB GkW (I − PAk+1 ) + (I − PBk )GkW ∆PA . Consequently, (PTk+1 − PTk )(GkW )

2

≤ ∆PB GkW (I − PAk+1 )

2

+ (I − PBk )GkW ∆PA

20

2

≤ ∥∆PA ∥2 + ∥∆PB ∥2 Gmax .

By Lemma 1 (whose assumption ∥∆Ak ∥2 ≤ σr (Ak )/2 is satisfied since ∥∆Ak ∥2 ≤ and the step-size condition), ∥∆Ak ∥2 ∥∆PA ∥2 = ∥PAk+1 − PAk ∥2 ≤ . σr (Ak ) √ Using the factor-update bound ∥∆Ak ∥2 ≤ 2ηk /δ, √ 2 ∥∆PA ∥2 ≤ 2 ηk . δ Similarly,

√

2ηk /δ ≤ δ/2 by Lemma 3

√ 2 ∥∆Bk ∥2 ∥∆PB ∥2 ≤ ≤ 2 ηk . σr (Bk ) δ

Therefore, (PTk+1 − PTk )(GkW )

√ 2 2Gmax ≤ ηk . δ2 2

Combining the two contributions, ∥gk+1 − gk ∥2 ≤

! √ 2 2Gmax ηk . 2M c2 + δ2

Since rank(gk+1 − gk ) ≤ 4r, ∥gk+1 − gk ∥∗ ≤ 4r ∥gk+1 − gk ∥2 . Finally, by the reverse triangle inequality, ! √ 2 2Gmax ∥gk+1 ∥∗ − ∥gk ∥∗ ≤ ∥gk+1 − gk ∥∗ ≤ 4r 2M c2 + ηk = L′ ηk . δ2

A.5

Proof of Theorem 1

Theorem (Convergence of LoRA-TSD, restated). Let {Wk } be the iterates produced by Algorithm 1 (the momentum-free variant, Algorithm 2 differs only in the addition of momentum) and let gk = PTk (GkW ). Under assumptions (A1)–(A4) and the step-size condition, for any N ≥ 1: 1. Constant step size. If ηk ≡ η, then min ∥gk ∥∗ ≤

0≤k<N

and for η =

L(W0 ) − L∗ C + η, c0 ηN c0

p (L(W0 ) − L∗ )/(CN ), 2 min ∥gk ∥∗ ≤ 0≤k<N c0

2. Diminishing step size. If

P

k ηk = ∞ and

r

C(L(W0 ) − L∗ ) = O(N −1/2 ). N

2 k ηk < ∞, then limk→∞ ∥gk ∥∗ = 0.

P

Since Xτ ∈ Tk and PTk = PT∗k , ⟨GkW , ∆Wk ⟩ = −ηk ⟨GkW , Xτ ⟩ = −ηk ⟨PTk (GkW ), Xτ ⟩ = −ηk ⟨gk , Xτ ⟩ ≤ −ηk c0 ∥gk ∥∗ , where the last step uses (A4). 21

By Lemma 3, ∥∆Bk ∆Ak ∥2 ≤ 2c1 ηk2 . Combining with (A1) applied to the increment Wk+1 − Wk = ∆Wk + ∆Bk ∆Ak , where ∆Wk = ∆Bk Ak + Bk ∆Ak = −ηk Xτ is the first-order factor increment, L(Wk+1 ) − L(Wk ) ≤ ⟨GkW , ∆Wk + ∆Bk ∆Ak ⟩ +

M ∥∆Wk + ∆Bk ∆Ak ∥22 . 2

The three terms are bounded as follows. ⟨GkW , ∆Wk ⟩ ≤ −ηk c0 ∥gk ∥∗ , |⟨GkW , ∆Bk ∆Ak ⟩| ≤ Gmax rσ1 (∆Bk ∆Ak ) ≤ 2Gmax rc1 ηk2 , √ √ ∥∆Wk + ∆Bk ∆Ak ∥2 ≤ 2ηk + 2c1 ηk2 ≤ 2 c2 ηk , giving

L(Wk+1 ) − L(Wk ) ≤ −ηk c0 ∥gk ∥∗ + Cηk2 .

Part (i): constant step size. Summing over k = 0, . . . , N − 1 and telescoping, c0 η

N −1 X

∥gk ∥∗ ≤ L(W0 ) − L(WN ) + Cη 2 N.

k=0

Since L(WN ) ≥ L∗ by (A2), c0 η

N −1 X

∥gk ∥∗ ≤ L(W0 ) − L∗ + Cη 2 N.

k=0

P Dividing by c0 ηN and using min ≤ N1 , N −1

min ∥gk ∥∗ ≤

0≤k<N

1 X L(W0 ) − L∗ Cη ∥gk ∥∗ ≤ + . N c0 ηN c0 k=0

p The right-hand side is minimized over η > 0 at η = (L(W0 ) − L∗ )/(CN ), giving r 2 C(L(W0 ) − L∗ ) min ∥gk ∥∗ ≤ = O(N −1/2 ). 0≤k<N c0 N Part (ii): diminishing step size. Summing the descent inequality over all k, c0

∞ X

ηk ∥gk ∥∗ ≤ L(W0 ) − L∗ + C

k=0

∞ X

ηk2 < ∞,

k=0

P P so k ηk ∥gk ∥∗ < ∞. If lim inf k ∥gk ∥∗ = 2ε > 0, then ∥gk ∥∗ > ε for all large k, and k ηk ∥gk ∥∗ ≥ ε k ηk = ∞, a contradiction. Hence lim inf k ∥gk ∥∗ = 0. To upgrade to limk ∥gk ∥∗ = 0, suppose for contradiction that lim supk ∥gk ∥∗ = s > 0 and fix 0 < ε < s/3. Since lim inf k ∥gk ∥∗ = 0, we can extract an infinite sequence of disjoint intervals [mj , ℓj ] defined as follows: let mj be the last index before the j-th exceedance of 2ε at which ∥gmj ∥∗ ≤ ε, and let ℓj be the first subsequent index with ∥gℓj ∥∗ ≥ 2ε. By construction, ∥gk ∥∗ ≥ ε for all k ∈ (mj , ℓj ]. Applying the triangle inequality and Lemma 4, ℓj −1 ℓj −1 X X ′ ε ≤ ∥gℓj ∥∗ − ∥gmj ∥∗ ≤ ∥gk+1 ∥∗ − ∥gk ∥∗ ≤ L ηk . P

k=mj

k=mj

22

Hence

Pℓj −1

′ k=mj ηk ≥ ε/L , and since ∥gk ∥∗ ≥ ε on (mj , ℓj ], ℓj −1

X

ℓj −1

ηk ∥gk ∥∗ ≥ ε

k=mj

X

ηk ≥

k=mj +1

ε2 − ε ηmj . L′

2 ′ Since ηk → 0, the P right-hand side exceeds ε /(2L ) for all large j. Summing over the infinitely many disjoint intervals gives k ηk ∥gk ∥∗ = ∞, contradicting the finiteness established above. Therefore limk ∥gk ∥∗ = 0.

Factor-gradient consequence. The chain rule gives GkA = Bk⊤ GkW . Left-multiplying PTk (GkW ) = PBk GkW + GkW PAk − PBk GkW PAk by Bk⊤ and using Bk⊤ PBk = Bk⊤ yields Bk⊤ PTk (GkW ) = Bk⊤ GkW + GkA PAk − GkA PAk = GkA . Analogously, GkB = PTk (GkW )A⊤ k . If σ1 (Ak ), σ1 (Bk ) ≤ Γ, then ∥GkA ∥∗ , ∥GkB ∥∗ ≤ Γ∥PTk (GkW )∥∗ , which proves Remark 1.

A.6

Setup for Theorem 2 Algorithm 3: LoRA-Pro

1: for k = 0, 1, . . . do

GkA ← ∇A L(Wk ), GkB ← ∇B L(Wk ) 3: G∗Ak ← Bk† GkW , G∗Bk ← (I − PBk )GkW A†k 4: Ak+1 ← Ak − ηk G∗Ak , Bk+1 ← Bk − ηk G∗Bk 5: end for 2:

▷ canonical X = 0 minimizer

Assumptions. The proof uses (A1)–(A3) and the step-size condition. The descent-margin condition (A4) is not needed because LoRA-Pro solves the inner problem exactly. For convenience we recall the relevant ones. (A1) requires Frobenius smoothness and gradient boundedness: ∥∇L(W )∥F ≤ Gmax . (A2) requires L to be bounded below. (A3) requires  the factors to stay full-rank: σr (Ak ), σr (Bk ) ≥ δ > 0. Here the step-size condition specialises to ηk ≤ min 1, 1/(2C) with C as defined below. Explicit solution of the inner problem. The following result from Wang et al. [2025] gives the closed-form minimizer of the LoRA-Pro inner problem. Lemma 5 (Wang et al. 2025, Theorem 2.1 ) Let B ∈ Rm×r and A ∈ Rr×n be full-rank. The optimal solution of minGA ,GB ∥BGA + GB A − GW ∥2F is G∗A = B † GW + XA,

G∗B = (I − PB )GW A† − BX,

where X ∈ Rr×r is an arbitrary matrix, B † = (B ⊤ B)−1 B ⊤ , A† = A⊤ (AA⊤ )−1 , and PB = B(B ⊤ B)−1 B ⊤ . The free matrix X reflects a gauge ambiguity of the factor updates. Indeed, adding XA to G∗A and subtracting BX from G∗B leaves the induced tangent direction unchanged because the two terms BXA cancel. We fix this 23

ambiguity by choosing X = 0, which gives G∗Ak = Bk† GkW and G∗Bk = (I − PBk )GkW A†k . This choice yields the canonical orthogonal split Bk G∗Ak = PBk GkW ,

G∗Bk Ak = (I − PBk )GkW PAk ,

whose two components have orthogonal column spaces. Thus X = 0 removes only a cancelling factor-space component and preserves the LoRA-Pro update in weight space. Since the second-order product of the factor increments depends on the chosen representative, the convergence result below concerns this explicit canonical choice. We also define 1 2 C1 := 2 , C3 := 1 + C1 Gmax , C := C1 Gmax + M 2 C3 . δ

A.7

Proof of Theorem 2

Let GkW := ∇L(Wk ) and gk := PTk (GkW ). By the LoRA-Pro update rule, Ak+1 = Ak − ηG∗Ak and Bk+1 = Bk − ηG∗Bk , where (G∗Ak , G∗Bk ) is the minimizer of ∥GkW − (Bk GA + GB Ak )∥2F . Since the feasible set is exactly the tangent space Tk , the minimizer satisfies Bk G∗Ak + G∗Bk Ak = PTk (GkW ) = gk . Therefore the first-order inner product evaluates to ⟨GkW , −η(Bk G∗Ak + G∗Bk Ak )⟩ = −η⟨GkW , gk ⟩ = −η⟨PTk (GkW ), gk ⟩ = −η∥gk ∥2F . By Lemma 5 with X = 0, G∗Ak = Bk† GkW and G∗Bk = (I − PBk )GkW A†k . We now show these can be expressed in terms of gk . For G∗Ak : using PTk (GkW ) = PBk GkW + (I − PBk )GkW PAk and Bk† (I − PBk ) = 0, Bk† gk = Bk† PBk GkW + Bk† (I − PBk )GkW PAk = Bk† GkW = G∗Ak . Hence G∗Ak = Bk† gk , and ∥G∗Ak ∥F = ∥Bk† gk ∥F ≤ ∥Bk† ∥2 ∥gk ∥F =

∥gk ∥F ∥gk ∥F ≤ . σr (Bk ) δ

Next, we estimate G∗Bk . GkW − gk = GkW − PBk GkW − (I − PBk )GkW PAk = (I − PBk )GkW − (I − PBk )GkW PAk = (I − PBk )GkW (I − PAk ). Multiplying from the left by I − PBk and from the right by A†k , (I − PBk )(GkW − gk )A†k = (I − PBk )2 GkW (I − PAk )A†k . Since (I − PAk )A†k = 0, it follows that (I − PBk )(GkW − gk )A†k = 0, i.e. (I − PBk )GkW A†k = (I − PBk )gk A†k . Hence G∗Bk = (I − PBk )GkW A†k = (I − PBk )gk A†k and ∥G∗Bk ∥F = ∥(I − PBk )GkW A†k ∥F = ∥(I − PBk )gk A†k ∥F ∥gk ∥F ∥gk ∥F ≤ . ≤ ∥I − PBk ∥2 ∥gk ∥F ∥A†k ∥2 = σr (Ak ) δ 24

It follows that ∥η 2 G∗Bk G∗Ak ∥F ≤ η 2 ∥G∗Bk ∥F ∥G∗Ak ∥F ≤

η2 ∥gk ∥2F = C1 η 2 ∥gk ∥2F . δ2

Wk+1 − Wk = (Bk − ηG∗Bk )(Ak − ηG∗Ak ) − Bk Ak = −η(G∗Bk Ak + Bk G∗Ak ) + η 2 G∗Bk G∗Ak = −ηgk + η 2 G∗Bk G∗Ak . Since ∥gk ∥F ≤ ∥GkW ∥F ≤ Gmax and η ≤ 1, ∥Wk+1 − Wk ∥F ≤ η∥gk ∥F + C1 η 2 ∥gk ∥2F ≤ η∥gk ∥F (1 + C1 Gmax ) = C3 η∥gk ∥F . By Frobenius smoothness (A1), M ∥Wk+1 − Wk ∥2F 2 M 2 2 = −η∥gk ∥2F + η 2 ⟨GkW , G∗Bk G∗Ak ⟩ + C η ∥gk ∥2F . 2 3

L(Wk+1 ) − L(Wk ) ≤ ⟨GkW , Wk+1 − Wk ⟩ +

Since

|⟨GkW , G∗Bk G∗Ak ⟩| ≤ ∥GkW ∥F ∥G∗Bk G∗Ak ∥F ≤ C1 Gmax ∥gk ∥2F , L(Wk+1 ) − L(Wk ) ≤ −η∥gk ∥2F + Cη 2 ∥gk ∥2F = −η(1 − Cη)∥gk ∥2F .

For η ≤ min{1, 1/(2C)}, 1 − Cη ≥ 1/2, so η L(Wk+1 ) − L(Wk ) ≤ − ∥gk ∥2F . 2 Summing over k = 0, . . . , N − 1 and using L(WN ) ≥ L∗ , N −1

η X ∥gk ∥2F ≤ L(W0 ) − L∗ . 2 k=0

Since min ≤ N1

P

and min ∥gk ∥F ≤ (min ∥gk ∥2F )1/2 , s 2(L(W0 ) − L∗ ) min ∥gk ∥F ≤ = O(N −1/2 ). 0≤k<N ηN

Letting N → ∞ gives

B

P∞

2 k=0 ∥gk ∥F < ∞, hence limk→∞ ∥gk ∥F = 0.

Retraction Analysis

This appendix proves that the factor-induced map used by LoRA-TSD is a retraction on the fixed-rank manifold, and compares it with the truncated-SVD retraction of Riemannion. We work on the fixed-rank manifold Mr = {W ∈ Rm×n : rank W = r}. The constant affine shift W0 does not affect the tangent geometry, so we omit it. Fix a point W = BA ∈ Mr with full-rank factors B ∈ Rm×r and A ∈ Rr×n , and recall the tangent space TW Mr = {∆B A + B ∆A : ∆B ∈ Rm×r , ∆A ∈ Rr×n }. We use the standard notion of a retraction [Absil et al., 2008]. A map RW : TW Mr → Mr that is smooth on a neighborhood of the origin is a retraction at W when (i) RW (0) = W

(centering), and

d (ii) dt RW (tξ) t=0 = ξ for every ξ ∈ TW Mr

(local rigidity). 25

m

n Factor-induced (LoRA-TSD) Truncated SVD [Bogachev et al., 2025] Speedup Layer

4096 4096 1024 4096 14336 4096

0.434 0.434 0.431

1.025 0.911 1.215

2.36× 2.10× 2.82×

q_proj k/v_proj gate/up_proj

Table 5: Wall-clock time (ms) of a single retraction step, factor-induced (LoRA-TSD) versus truncated-SVD (Riemannion), at the weight shapes of Llama-3.1-8B, measured on an NVIDIA H200.

The factor-induced map. Given a tangent direction ξ ∈ TW Mr , LoRA-TSD reconstructs the factor increments by the canonical split of Eqs. (16)–(18),
∆B = ξA† , ∆A = B † ξ − ∆B A , with A† = A⊤ (AA⊤ )−1 and B † = (B ⊤ B)−1 B ⊤ . Both increments depend linearly on ξ. LoRA-TSD then applies the update

RW (ξ) := B + ∆B A + ∆A . Proposition 3 (The factor-induced map is a retraction) If B and A have full rank, then RW is a retraction at W = BA in the sense of Absil et al. [2008]. Proof. For smoothness, note that the increments ∆B and ∆A depend linearly, hence smoothly, on ξ. The full-rank matrices form an open set, so for small ∥ξ∥2 both B + ∆B and A + ∆A remain full rank. Their product then has rank exactly r and lies in Mr . Hence RW is smooth from a neighborhood of 0 ∈ TW Mr into Mr . Centering holds because at ξ = 0 both increments vanish, so RW (0) = BA = W . For local rigidity, expand the product,
RW (ξ) = BA + ∆B A + B ∆A + ∆B ∆A. The reconstructed increments satisfy ∆B A + B ∆A = ξPA + PB ξ(I − PA ) = PT (ξ) = ξ, using A† A = PA , BB † = PB , the projector formula (13), and PT (ξ) = ξ for ξ ∈ TW Mr . Hence RW (ξ) = W + ξ + ∆B ∆A. Because the increments are linear in ξ, the scaled direction tξ reconstructs to t ∆B and t ∆A, so

RW (tξ) = B + t ∆B A + t ∆A = W + t ξ + t2 ∆B ∆A. The second-order coefficient is finite. Since ∆B A = ξPA , we have ∆A = B † ξ(I − PA ). Therefore, ∥ξ∥2 , σr (A)

∥∆A∥2 ≤

∥∆B ∆A∥2 ≤

∥ξ∥22 , σr (A)σr (B)

∥∆B∥2 ≤ so that

∥ξ∥2 , σr (B)

d the same estimate as Lemma 3. Hence RW (tξ) = W + t ξ + O(t2 ), and dt RW (tξ) t=0 = ξ for every ξ ∈ TW Mr , which is condition (ii).

26

Relation to the truncated-SVD retraction. The connection to Riemannion has two parts: the tangent direction and the retraction. For τ = 1, LoRA-TSD recovers the tangent-direction part of Riemannion, since both methods apply a matrix-sign orthogonalization to the tangent-projected gradient. For τ > 1, LoRA-TSD repeats the tangent projection and matrix-sign steps as an alternating-projection refinement toward the constrained spectral oracle of Eq. (11). In our hyperparameter tuning, τ = 3 to 5 performs markedly better than τ = 1, so this refinement accounts for part of the empirical gain. SVD (ξ) = SVD (W + ξ), the best rank-r approxRiemannion instead uses the metric-projection retraction RW r imation of W + ξ. That this map is a smooth retraction on Mr is standard [Vandereycken, 2013]. Being SVD satisfy R (ξ) = W + ξ + O(∥ξ∥2 ), so they agree to first order in ξ and can retractions, both RW and RW W 2 differ only at second order. For the factor-induced map the second-order term is exactly ∆B ∆A, whereas for the truncated-SVD map it is the curvature correction of Mr . The two therefore produce different iterates whenever ξ = ̸ 0, even though their tangent directions coincide. In particular, equal tangent directions at τ = 1 do not make LoRA-TSD and Riemannion identical after retraction. The factor-induced retraction costs only the single product ∆B ∆A and follows directly from the LoRA parameterization, while the truncated-SVD retraction requires an SVD at every step. Table 5 reports the wall-clock cost of a single retraction step at the attention and MLP weight shapes of Llama-3.1-8B. The factor-induced retraction is essentially independent of the layer size, whereas the truncated-SVD cost grows with it, so the gap widens for the larger projections.

C

Experimental Details and Additional Results

Datasets. We evaluate Llama-3.2-1B-Instruct and Llama-3.1-8B on six English benchmarks covering commonsense reasoning and natural language inference. We also evaluate Qwen3-32B on SIQA and OBQA. We report accuracy for every benchmark because each test example has a discrete target, making correctness unambiguous. The label distributions are sufficiently balanced for accuracy to provide an appropriate summary of performance. • BoolQ [Clark et al., 2019] is a reading-comprehension dataset of naturally occurring yes/no questions, each paired with a Wikipedia passage from which the model predicts a yes or no answer. • PIQA [Bisk et al., 2020] tests physical commonsense: given a goal, the model chooses which of two candidate solutions is physically plausible. • SIQA [Sap et al., 2019] (Social IQa) probes social and emotional commonsense, asking the model to reason about the motivations, reactions, and consequences of everyday social interactions. • OBQA [Mihaylov et al., 2018] (OpenBookQA) is a multiple-choice science question-answering dataset modeled on open-book exams, requiring elementary science facts to be combined with broad commonsense. • QNLI [Wang et al., 2019] is the question-answering natural-language-inference task of the GLUE benchmark, derived from SQuAD [Rajpurkar et al., 2016]. Each example pairs a question with a sentence, and the model decides whether the sentence contains the answer. • MultiNLI [Williams et al., 2018] is a multi-genre natural-language-inference corpus in which the model classifies the relation between a premise and a hypothesis as entailment, contradiction, or neutral. Hyperparameter tuning protocol. For each optimizer and model size, we first evaluated three or four learning rates on BoolQ. We then ran three additional trials around the best value from this coarse sweep. We selected the learning rate by the highest validation score reached at any point during training. The selected configuration was frozen and reused without further tuning on all remaining benchmarks. This protocol avoids per-dataset tuning and keeps the comparison focused on the optimizer. The selected values are reported in Tables 7 and 8. For Qwen3-32B, we transfer the Llama-3.1-8B settings with minimal tuning and reduce gradient clipping from 1 to 0.25 for training stability. 27

2,500

1.4 mean

1,500

1.2 1,000

training step

2,000

1.3

σ1 /σk

Rank ablation. Table 6 gives the full rank sweep on BoolQ and SIQA with Llama-3.18B, extending the summary in Section 4. All hyperparameters other than the rank r are held fixed. LoRA-TSD achieves the best accuracy in 11 of 14 settings and stays within a narrow accuracy band as r varies. LoRA-Rite is competitive at small ranks but degrades at larger ranks, including a sharp drop at r = 64 on SIQA.

1.1 500

1

1

4

8

12

16

20

24

28

32

singular-value index k

Spectrum of the update. Does the tangent-space update produce the balanced spectrum that motivates the spectral-norm Figure 3: Per-step singular-value ratios σ1 /σk of the LoRA-TSD geometry? Figure 3 plots the ratio σ1 /σk of update, colored by training step, with the mean highlighted. the LoRA-TSD update against the singular- The spectrum stays close to flat across all steps, reflecting the value index k for every recorded training step. spectral-norm geometry of the tangent-space update. The LoRA rank here is r = 16, so σ32 is the last nonzero singular value of the tangent update ∆Bk Ak + Bk ∆Ak , which has rank at most 2r = 32. The ratio stays close to one along the whole spectrum, reaching only about 1.2 on average at the smallest singular value, and the per-step curves remain tightly clustered throughout training. The update is therefore almost equalized rather than dominated by a few leading directions, which is what the spectral-norm geometry inherited from Muon is meant to produce. BoolQ

SIQA

r

LoRA-TSD

Muon

LoRA-Rite

AdamW

LoRA-TSD

Muon

LoRA-Rite

AdamW

1 2 4 8 16 32 64

89.82 89.17 89.69 90.49 90.46 89.79 90.06

65.63 64.07 89.57 89.82 89.14 89.14 89.11

88.81 89.66 89.66 63.70 88.93 87.86 71.93

88.65 88.96 88.87 89.48 89.60 89.02 89.97

82.14 82.60 82.50 83.47 83.06 82.45 82.29

80.55 80.71 80.86 81.53 81.20 81.01 81.06

81.78 82.65 82.55 81.73 81.40 81.06 77.84

80.96 81.78 80.86 81.73 81.80 81.83 81.83

Table 6: Rank ablation on BoolQ and SIQA (Llama-3.1-8B). Accuracy versus LoRA rank r with all other hyperparameters fixed. Bold marks the best method per rank within each dataset.

28

Method

η

Method-specific hyperparameters

LoRA-SGD AdamW Muon Riem-SGD LoRA-Rite LoRA-TSD Riemannion LoRA-Pro

−3

10 10−3 10−2 2·10−3 8·10−5 5·10−3 3·10−5 5·10−4

momentum 0.9 β = (0.9, 0.999), ϵ = 10−8 momentum 0.95, Nesterov, NS steps 5 momentum 0.95 β = (0.9, 0.999), ϵ = 10−6 τ = 5, NS steps 5 momentum 0.95, NS steps 6 ϵinv = 10−6

Muon Full-FT

3·10−2

momentum 0.95, Nesterov, NS steps 5

Table 7: Selected hyperparameters for Llama-3.2-1B-Instruct, tuned on BoolQ. Common to all methods: rank r = 16, α = 16, dropout 0.05, effective batch size 32, 3000 steps, linear schedule, adapters on q_proj/k_proj/v_proj.

Method LoRA-SGD AdamW Muon Riem-SGD LoRA-Rite LoRA-TSD Riemannion LoRA-Pro

η 5·10−3 10−4 10−4 2·10−3 10−4 4·10−3 3·10−5 5·10−5

Method-specific hyperparameters momentum 0.9 β = (0.9, 0.999), ϵ = 10−8 momentum 0.95, Nesterov, NS steps 5 momentum 0.95 β = (0.9, 0.999), ϵ = 10−6 τ = 5, NS steps 5, rebalance every 250 momentum 0.95, NS steps 6 ϵinv = 10−6

Table 8: Selected hyperparameters for Llama-3.1-8B, tuned on BoolQ. Common to all methods: rank r = 16, α = 16, dropout 0.05, effective batch size 32, 3000 steps, linear schedule, adapters on q_proj/k_proj/v_proj.

29

Related documents

• Musec: MomentUm SpEctral Clipping for Stable Muon-type Training#673520
• Erratum to: Computation of pseudospectral abscissa for large-scale nonlinear eigenvalue problems#955431
• Musec: MomentUm SpEctral Clipping for Stable Muon-type Training#1056379
• AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery#175284
• Graph Domain Adaptation via Theory-Grounded Spectral Regularization.#351318