Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR Ruijia Zhang∗ JHU
Jiacheng Zhu† Meta
Hanqing Zhu‡ UT Austin
Laixi Shi§ JHU
arXiv:2606.31813v1 [cs.LG] 30 Jun 2026
July 1, 2026
Abstract Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learning with verifiable rewards (RLVR) are less well understood. In particular, two structurally initialized LoRA variants, PiSSA and MiLoRA, which outperform standard LoRA under SFT, can underperform standard LoRA under RLVR and may even exhibit training instability. These observations suggest that how to initialize the low-rank matrices in RLVR remains unclear. In this work, we develop a theoretical analysis of LoRA in RLVR, showing that orthonormal initialization achieves the minimal gap between LoRA’s outcome and that of full fine-tuning. Guided by this insight, we propose geometry-preserving orthonormal initialization for low-rank adaptation in RLVR, leading to two new variants, LoRA-RLPO and LoRA-RLMO. Experiments on mathematical reasoning benchmarks show that the proposed orthonormal initialization stabilizes RLVR training and outperforms standard LoRA, contrasting with PiSSA and MiLoRA. Finally, our unified analysis for LoRA initialization also explains why PiSSA and MiLoRA can underperform in RLVR, which may be of independent interest. Code and checkpoints are publicly available at the repository.
Contents 1 Introduction
2
2 Related Works
3
3 Background 3.1 Low-rank adaptation (LoRA) and variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Finetuning paradigms: SFT vs. RLVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 4 5
4 Instability of SVD-Based LoRA Initializations in RLVR 4.1 Instability sources of LoRA family in RLVR. . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 7
5 Geometry-Preserving Orthonormal Initialization for LoRA 10 5.1 Optimization Dynamics of LoRA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.2 Geometry-Preserving Orthonormal Initialization for LoRA . . . . . . . . . . . . . . . . . . . . 11 6 Experiments and Analysis
11
7 Conclusion
14
∗ Department of Applied Mathematics and Statistics, Johns Hopkins University, MD, USA. † Meta Superintelligence Labs. ‡ Department of Electrical and Computer Engineering, The University of Texas at Austin, TX, USA. § Department of Electrical and Computer Engineering, Johns Hopkins University, MD, USA.
1
A Proof of LoRA Optimization Dynamics 19 A.1 Proof of Theorem 5.2 and Proposition 5.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B Proof of Gradient Amplification of PiSSA over OLoRA
22
C Experimental Details 24 C.1 Training details of RLVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.2 Evaluation setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 D Additional Ablation Study 27 D.1 PiSSA and MiLoRA failure analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 D.2 Further ablations for RLVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 D.3 Generalization to supervised fine-tuning (SFT) . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1
Introduction
Large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023) are typically pretrained on large-scale dataset via next-token prediction (Brown et al., 2020) and then fine-tuned on relatively smaller datasets to specialize for downstream applications. This paradigm has achieved remarkable success across diverse domains, including mathematical reasoning (Luo et al., 2025; Azerbayev et al., 2024), code generation (Rozière et al., 2024; Luo et al., 2024), healthcare (Singhal et al., 2023; Chen et al., 2023), and finance (Wu et al., 2023; Yang et al., 2025). Because fine-tuning is far more accessible than pretraining a new LLM, it has attracted substantial interest in the community. While fine-tuning all parameters in an LLM (“full fine-tuning”) is natural, it is practically highly memory-intensive: fully fine-tuning even a 7B model can require over 100GB of GPU memory (Dettmers et al., 2023). This high resource demand limits accessibility for practitioners and motivates parameter-efficient fine-tuning (PEFT) methods, which update only a small subset of parameters while keeping the base model frozen (Houlsby et al., 2019; Li and Liang, 2021). Among them, Low-Rank Adaptation (LoRA) (Hu et al., 2022) is widely used due to its efficiency and ease of implementation. For any weight matrix W0 ∈ Rm×n in a pretrained model, LoRA parameterizes the update as ∆W lora = BA with B ∈ Rm×r and A ∈ Rr×n , where r is much smaller than min(m, n). This significantly reduces the number of parameters under training, while still yielding a dense matrix W0 + ∆W lora at inference time. Beyond supervised fine-tuning (SFT), a widely used LLM fine-tuning paradigm that trains on high-quality question–response pairs, reinforcement learning with verifiable rewards (RLVR) has recently emerged as a pivotal paradigm, proving effective across tasks such as mathematical reasoning and coding (Guo et al., 2025; Shao et al., 2024). RLVR uses rule-based feedback (e.g., answer correctness) instead of learned reward models. However, RLVR incurs substantially higher memory costs than SFT, as it requires keeping a reference model in memory to compute KL divergence (Ziegler et al., 2020; Zhou et al., 2024) and storing multiple responses per prompt for group-based advantage estimation (Shao et al., 2024). This makes LoRA and its memory-efficient variants particularly attractive for RLVR, especially given that LoRA has already shown strong potential in this setting, matching full fine-tuning in certain cases (Schulman and Lab, 2025). Despite this progress, the behavior of LoRA and its structural variants under RLVR remains less understood than under SFT, limiting further advances in low-rank fine-tuning for RL. In particular, how to initialize the low-rank matrices B and A is increasingly unclear in light of recent observations. PiSSA (Meng et al., 2024) and MiLoRA (Wang et al., 2025), two LoRA variants that improve performance and accelerate convergence in SFT, can underperform standard LoRA under RLVR and may even exhibit training instability (Yin et al., 2025). Both methods initialize B and A via the singular value decomposition (SVD) of pretrained weights, but in opposite directions: PiSSA uses the top-r principal singular directions, while MiLoRA targets the bottom-r tail directions. In addition, prior work suggests that, due to the KL constraint, RLVR updates are encouraged to stay close to the reference policy (Wu et al., 2026; Shenfeld et al., 2025) and may favor off-policy subspaces that differ from those preferred by SFT (Zhu et al., 2025). This discrepancy between RLVR and SFT likely reflects their distinct optimization dynamics. Consequently, LoRA design principles developed for SFT are no longer guaranteed to transfer to RLVR, leaving the appropriate initialization and subspace choice in RLVR an open question. In this work, we focus on:
2
Figure 1: Comparison of LoRA initialization strategies. LoRA uses random Gaussian A0 with B0 = 0. PiSSA and MiLoRA initialize both adapters from the principal and minor singular components of W0 , respectively, with B0 ̸= 0 and singular value scaling. Our proposed methods, LoRA-RLPO and LoRA-RLMO, initialize orthonormal A0 from the principal and minor right singular vectors with B0 = 0. What initialization is effective for low-rank adaptation in RLVR fine-tuning? To this end, we provide a rigorous analysis demonstrating that by initializing B = 0 in accordance with standard LoRA, orthonormal initialization for A is potentially optimal and yields superior performance in practice. Our primary contributions are as follows: • Orthonormal initialization towards optimal. To understand the behavior of LoRA, we provide a theoretical analysis of LoRA’s optimization dynamics, showing that orthonormal initialization of A with B = 0 minimizes the gap between LoRA and full fine-tuning (Theorem 5.2). • Geometry-preserving orthonormal initialization. Motivated by this result, we propose LoRARLPO and LoRA-RLMO, SVD-based initializations for A that remain orthonormal while preserving geometric information from pretrained weights. Both methods outperform standard LoRA in RLVR in the conducted experiments. • Insights into LoRA variants’ failures in RLVR. Our analysis framework offers a unified explanation for the instability of SVD-based methods such as PiSSA and MiLoRA in RLVR. Their failures stem from two coupled factors: subspace geometry, which accelerates updates along specific directions, and singular value scaling, which amplifies update magnitudes. Together, these induce aggressive optimization trajectories that rapidly violate the implicit KL constraint, destabilizing training regardless of whether principal or minor singular directions are targeted.
2
Related Works
Low-rank adaptation (LoRA) and its variants. Among parameter-efficient fine-tuning (PEFT) methods, LoRA (Hu et al., 2022) and its variants have become a popular class, parameterizing weight updates as a product of two low-rank matrices while keeping the base model frozen. Numerous variants have been proposed to improve upon standard LoRA. One line modifies the optimization process: AdaLoRA (Zhang et al., 2023) adaptively allocates rank across layers based on importance scores; LoRA+ (Hayou et al., 2024) uses different learning rates for A and B; DoRA (Liu et al., 2024a) decomposes updates into magnitude and direction components; rsLoRA (Kalajdzievski, 2023) adjusts the scaling factor to stabilize training at higher ranks; and VeRA (Kopiczko et al., 2024) shares frozen random matrices across layers to further reduce parameters. Another line improves LoRA initialization beyond the default random scheme (Hu et al., 2022; Hayou et al., 2024). SVD-based methods have drawn particular attention: PiSSA (Meng et al., 2024) 3
initializes adapters using principal singular components, while MiLoRA (Wang et al., 2025) uses minor components. These methods achieve faster convergence and improved performance in supervised fine-tuning, but recent evaluations show that they underperform standard LoRA and exhibit instability in RLVR (Yin et al., 2025). Our work follows this line and focuses on understanding and addressing this gap through geometry-preserving orthonormal initialization. Orthonormality in LoRA. Several prior works have explored the role of orthonormality in LoRA, primarily in the context of supervised fine-tuning. Zhu et al. (2024) investigate the asymmetry between the two LoRA matrices, showing that A extracts features from inputs while B maps these features to outputs; they further demonstrate that fixing A as a random orthonormal matrix and training only B outperforms standard LoRA. OLoRA (Büyükakyüz, 2024) uses Qrthogonal-Right triangular(QR) decomposition to initialize both LoRA matrices with orthonormal bases derived from the pretrained weights, achieving faster convergence on SFT tasks. From a complementary perspective, OFT (Qiu et al., 2023) and BOFT (Liu et al., 2024b) enforce orthogonality of weight updates throughout training, rather than only at initialization. However, these studies are largely empirical and confined to the SFT regime, whose learning dynamics differ substantially from those of RLVR. Our work provides the first theoretical explanation for why orthonormal initialization improves LoRA in RLVR, showing that it enables LoRA to more closely track the trajectory of full fine-tuning. LoRA for Reinforcement learning with verifiable rewards. While LoRA has been extensively studied in supervised fine-tuning (Hu et al., 2022; Liu et al., 2024a; Kalajdzievski, 2023; Hayou et al., 2024), its behavior under RL-based fine-tuning remains far less understood, despite its widespread adoption for memoryefficient PPO and GRPO training on consumer hardware (Santacroce et al., 2023; Guo et al., 2025; Shao et al., 2024). Zhu et al. (2025) provides theoretical analysis showing that RLVR updates favor off-principal directions, in contrast to SFT which targets principal components, suggesting that methods designed for SFT may not transfer directly to RLVR. Yin et al. (2025) systematically evaluate PEFT methods under RLVR and find that SVD-based initializations such as PiSSA and MiLoRA underperform standard LoRA and exhibit training instability. Despite this progress, the appropriate initialization and subspace choice for LoRA in RLVR remains unsettled. Our work addresses this gap by providing a unified theoretical framework that explains both failure modes and showing that, when paired with the standard LoRA choice of B = 0, geometry-preserving orthonormal initialization of A offers a principled practical solution.
3
Background
In this section, we formalize the fine-tuning problem for LLMs. Consider an LLM parameterized by θ = {W (ℓ) }L ℓ=1 , the collection of weight matrices across its L layers (e.g., fully connected and attention layers). Without loss of generality and a slight abuse of notation, we focus on a single weight matrix W in the following discussion. Full fine-tuning optimizes the model parameters θ by updating all weight matrices. Specifically, for any pretrained weight matrix W0 ∈ Rm×n , full fine-tuning learns W = W0 + ∆W full , where the update ∆W full ∈ Rm×n is unconstrained, so as to minimize a task-specific loss L(θ) = L({W (ℓ) }L ℓ=1 ). This approach requires storing full gradients and optimizer states for all weight matrices, which can become prohibitive for large-scale models.
3.1
Low-rank adaptation (LoRA) and variants
We first review the LoRA algorithm (Hu et al., 2022). Consider a pretrained weight matrix W0 ∈ Rm×n , LoRA parameterizes the weight update as with
W = W0 + ∆W lora ,
∆W lora = BA,
(1)
where B ∈ Rm×r and A ∈ Rr×n are low-rank matrices with rank r much smaller than m and n (r ≪ min{m, n}). Consequently, the optimization of ∆W lora is restricted to a low-rank subspace of Rm×n . The initialization is set as follows: r×n . (2) B0 = 0m×r , A0 ∼ N 0, n1
4
SVD-based initialization variants. Beyond the initialization in (2), many prior works propose alternative initializations for low-rank fine-tuning. We describe two representative SVD-based variants below. Let W0 = U ΣV ⊤ be the singular value decomposition, where U ∈ Rm×k , Σ = diag(σ1 , . . . , σk ) with σ1 ≥ σ2 ≥ · · · ≥ σk > 0, and V ∈ Rn×k . Here, k = rank(W0 ), i.e., the number of positive singular values of W0 . PiSSA (Meng et al., 2024) initializes with top-r principal components as follows: ⊤ A0 = Σ1/2 r Vr
B0 = Ur Σ1/2 r ,
(3)
where Ur and Vr denote the first r columns of U and V , respectively, and Σr is the r × r diagonal matrix of the top r singular values. MiLoRA (Wang et al., 2025) initializes using the bottom-r minor components: 1/2
1/2
⊤ A0 = Σ−r V−r ,
B0 = U−r Σ−r ,
(4)
where U−r and V−r denote the last r columns of U and V , respectively, and Σ−r is the r × r diagonal matrix of the bottom r singular values. Both methods then replace each pretrained matrix W0 with the residual W0 − B0 A0 , which is kept frozen, and optimize only BA thereafter. Equivalently, the effective weight is parameterized as W = (W0 −B0 A0 )+BA, ensuring W = W0 at initialization regardless of which singular components are used.
3.2
Finetuning paradigms: SFT vs. RLVR
Besides the parameter-update setting, we now introduce two widely used fine-tuning frameworks and their corresponding objective functions. Supervised fine-tuning (SFT). Consider a supervised dataset D = {(qi , a⋆i )}N i=1 consisting of many input-output pairs, where qi denotes a prompt or instruction and a⋆i is the corresponding ground-truth response. The SFT process minimizes cross-entropy against ground-truth labels as follows: LSFT (θ) := −E(q,a∗ )∼D [log πθ (a∗ |q)] . SFT imposes no explicit constraint on the weight movement, allowing the parameters to drift arbitrarily far from W0 . Reinforcement learning with verifiable rewards (RLVR). RLVR fine-tunes large language models with RL using automatically verifiable, rule-based rewards R (e.g., exact-match correctness on math or code), thereby eliminating the need for a learned reward model. In this work, we focus on DAPO (Yu et al., 2026) because it is a representative state-of-the-art RLVR algorithm for long-CoT reasoning and, unlike GRPO, removes the explicit KL penalty to the reference policy, making it a cleaner setting for isolating and understanding the factors that affect RLVR training stability. Moreover, DAPO and related clipped-policy RLVR algorithms are widely used for evaluating LoRA-style adaptation, making them a natural testbed for analyzing the effect of low-rank initialization (Yin et al., 2025). Specifically, DAPO samples a group of outputs {oi }G i=1 for each question q paired with answer a, and updates the policy by optimizing the following clipped importance-ratio objective: " |oi | G X X 1 DAPO + L (θ ) = E(q,a)∼D, {oi }∼πθ (·|q) PG i=1 |oi | i=1 t=1 # (5) + + min ri,t (θ )Âi,t , clip ri,t (θ ), 1 − ϵlow , 1 + ϵhigh Âi,t , s.t. π
(o
|q,o
)
0 < {oi | is_equivalent(a, oi )} < G,
i,<t where ri,t (θ+ ) = πθθ+(oi,ti,t|q,oi,<t ) , Âi,t =
Ri −mean({Ri }G i=1 ) . A key ingredient of DAPO is the clipped importance std({Ri }G i=1 )
ratio, which constrains ri,t (θ+ ) to [1 − ϵlow , 1 + ϵhigh ] and thereby implicitly limits the policy drift between πθ+ and πθ . 5
LoRA
Training reward
PiSSA
MiLoRA
Gradient norm (log scale)
KL Divergence (log scale)
0.0
10 1 100
1.0
10 2
KL Divergence
Gradient norm
Reward
0.5
10 1
1.5 2.0 200
300
Training step
400
500
10 4 10 5 10 6
10 2 100
10 3
100
200
300
Training step
400
500
10 7
100
200
300
Training step
400
500
Figure 2: Training dynamics of RLVR via DAPO on benchmark DAPO-MATH. Left: Training reward comparison. Middle: The aggregate Frobenius norm of gradients over trainable parameters. Right: KL divergence during training. Both PiSSA and MiLoRA exhibit training reward collapse, higher gradient norm and KL divergence than standard LoRA. Training stability demands constrained KL divergence. The conservative-update principle implemented by DAPO’s clipped importance ratio is not unique to DAPO: it underlies many popular policy-gradient algorithms (e.g., TRPO Schulman et al. (2015) and PPO Schulman et al. (2017)) as well as RLVR popular variants such as GRPO (Shao et al., 2024). By preventing the updated policy from deviating too far from the current one in a single step, the clipping mechanism can be viewed as a surrogate method to constrain the following KL divergence between πθ+ and πθ (Kakade and Langford, 2002) in a safe region: # " X πθ+ (yt | q, o<t ) . (6) DKL (πθ+ ∥ πθ ) = Eq∼D, y∼πθ+ (·|q) log πθ (yt | q, o<t ) t Consequently, an excessively large DKL (πθ+ ∥ πθ ) may violate this implicit trust-region constraint and can cause performance degradation or training collapse, since the surrogate objective is no longer guaranteed to be a lower bound of the true reward objective (Kakade and Langford, 2002). We therefore adopt DKL (πθ+ ∥ πθ ) as a key diagnostic for the training stability of RLVR, and report its sample estimate over response tokens in the empirical analyses that follow.
4
Instability of SVD-Based LoRA Initializations in RLVR
Noting that the behavior of LoRA variants in RLVR remains underexplored, limiting further advances in lowrank fine-tuning for many tasks such as reasoning. In this work, we focus on studying the initialization module of LoRA for RLVR, focusing on two prominent variants (PiSSA and MiLoRA) that have demonstrated strong performance in supervised fine-tuning (SFT). While both PiSSA and MiLoRA outperform standard LoRA in SFT, they instead underperform standard LoRA under RLVR and even exhibit clear training collapse, as shown in Figure 2 (left). To understand the source of this failure, we monitor the introduced KL divergence in (6) and the Frobenius norm of the gradient for training stability, as shown in Figure 2 (middle and right). Both PiSSA and MiLoRA incur substantially larger gradient norms and higher cumulative KL divergence than standard LoRA throughout training, indicating markedly unstable optimization processes. These observations raise a natural question: Why do LoRA initialization principles that succeed in SFT, such as those of PiSSA and MiLoRA, break down under RLVR? Initialization of LoRA largely governs final optimization outcome. To answer the question, we begin by analyzing how the initialization step shapes the subsequent optimization trajectory and the resulting fine-tuned model. To this end, we visualize the post-training update distribution and cumulative energy after RLVR fine-tuning in Figure 3. Recall that PiSSA (cf. (3)) and MiLoRA (cf. (4)) initialize LoRA in the top and 1/2 1/2 bottom singular-vector subspaces, respectively, with singular value scaling through Σr or Σ−r . As shown in 6
PiSSA
MiLoRA
Cumulative energy proportion
Normalized update magnitude
Change distribution 0.10 0.08 0.06 0.04 0.02 0.00
0.0
0.2
0.4
0.6
0.8
Relative singular index (head tail)
1.0
LoRA
Cumulative Energy
1.0 0.8 0.6 0.4 0.2 0.0
0.0
0.2
0.4
0.6
0.8
Relative singular index (head tail)
1.0
Figure 3: SVD-aligned update distribution and cumulative energy after RLVR training. For each method, we analyze the trained LoRA update ∆W = αr BA on the query projection and attention output projection matrices from Transformer layers 0, 14, and 27 of the 28-layer model, where each projection weight has size W ∈ R1536×1536 . Given the singular value decomposition of the frozen pretrained weight W = U ΣV ⊤ , we measure theP update magnitude along the pretrained singular directions as ci = |u⊤ i ∆W vi |, and normalize it by pi = ci / j cj . The left panel plots pi over the relative singular-mode index, ordered from the to smallest singular values of W , while the right panel plots the corresponding cumulative Plargest P energy j≤i c2j / j c2j . Curves are averaged over the six analyzed projection matrices. The distinct patterns indicate that initialization affects not only the starting point of optimization, but also the spectral structure of the updates learned during RLVR. Figure 3, the update magnitudes and final energy distributions of the fine-tuned outputs closely mirror these initialized subspaces: PiSSA produces substantially larger updates along the top singular-vector directions, MiLoRA concentrates its updates along the bottom singular-vector directions, and in both cases the fine-tuned outcomes retain high cumulative energy in the corresponding spectral regions. This demonstrates that LoRA initialization does not merely set the optimization starting point, but rather it intrinsically governs the entire optimization trajectory and, consequently, the final outcome.
4.1
Instability sources of LoRA family in RLVR.
Another key observation from Figure 3 is that, after geometry-informed initialization, the final learned updates remain highly concentrated along the predefined singular directions, producing aggressively amplified weight changes. Such directionally intensified updates likely drive the large gradient norms and rapid KL growth observed under RLVR in Figure 2, which in turn contribute to the performance collapse of PiSSA and MiLoRA. To pinpoint the source of this instability, we disentangle three potential factors: (1) the learning rate during optimization, and two arising from initialization, namely, (2) the selected singular subspace, which determines the update direction, and (3) the singular value scaling, which amplifies updates along that direction. Learning rate partially controls update magnitude. To assess the role of the learning rate in driving instability, we conduct an ablation comparing constant and decaying schedules, shown in Figure 4. Slowing optimization through learning rate decay offers partial mitigation for both PiSSA and MiLoRA: as Figure 4 reveals, MiLoRA with a decaying learning rate approaches the stable behavior of standard LoRA. PiSSA, however, still suffers severe collapse in the later phases of training, indicating that learning rate decay alone does not resolve the underlying instability. This points to an inherently vulnerable optimization landscape for PiSSA: once updates are initialized and accelerated along its geometry-informed directions, the optimization trajectory tends to exit the stable optimization regime.
7
LoRA constant
PiSSA constant
PiSSA decaying
LoRA constant
0.5 1.0 1.5 2.0
MiLoRA decaying
200
400
0.0
Training reward
Training reward
0.0
MiLoRA constant
0.5 1.0 1.5
100
200
300
Training step
400
2.0
500
100
300
Training step
500
Figure 4: Ablation on learning-rate decay for PiSSA and MiLoRA under RLVR training. The left panel compares PiSSA with a constant learning rate and a cosine-decaying schedule, together with the LoRA constant-learning-rate baseline; the right panel shows the analogous comparison for MiLoRA. Both methods benefit from slower optimization, confirming that enlarged effective updates are a primary source of instability in RLVR training. Nevertheless, while MiLoRA with learning-rate decay approaches the stable behavior of standard LoRA, PiSSA still exhibits significant late-stage collapse, pointing to an inherently more hazardous optimization direction. Singular value scaling destabilizes training across singular subspaces. Beyond the choice of subspace, singular value scaling is another potentially critical factor for optimization stability. To isolate its effect from subspace selection, we use OLoRA (Büyükakyüz, 2024) as a controlled baseline. OLoRA initializes with top-r principal components as follows: A0 = Vr⊤ .
B0 = Ur ,
(7)
Since OLoRA and PiSSA target the same principal subspace of W0 , any difference in stability can be attributed to the singular value scaling inherent to PiSSA. We investigate this effect from both theoretical and empirical perspectives. Theoretical analysis. We first present a theorem that quantifies how PiSSA’s initialization potentially induces excessive update magnitudes compared to OLoRA. Theorem 4.1 (PiSSA Gradient Amplification). The first-step weight updates ∆W1PiSSA of PiSSA and ∆W1OLoRA of OLoRA satisfy: ∥∆W1PiSSA ∥F ≥ σr , (8) ∥∆W1OLoRA ∥F where σr is the r-th largest singular value of W0 . The proof is deferred to Appendix B. Although PiSSA and OLoRA share the same principal subspace, PiSSA scales each retained singular mode by the corresponding singular value, amplifying the update norm by at least σr to leading order. For pretrained LLMs, singular values typically follow a heavy-tailed distribution with σr ≫ 1 at moderate rank r (see Figure 9 in Appendix). Theorem 4.1 therefore implies that PiSSA inflates weight updates by a substantial factor relative to OLoRA, increasing the risk of exceeding the implicit KL budget in RLVR. To proceed, we invoke a result from Zhu et al. (2025), which shows that enforcing a KL constraint DKL (πθ+ ∥ πθ ) keeps the updated policy close to the current policy and, in turn, bounds the magnitude of the corresponding weight change. Theorem 4.2 (KL constraint implies weight bound (Zhu et al., 2025, Gate I)). Assume log πθ is C 3 , where F (θ) denotes the Fisher information matrix 1 . Consider a single-step update of the model parameters from θ 1 Here
C 3 means having continuous derivatives up to order 3. The Fisher information matrix is defined as F (θ) = Ex∼D, y∼πθ (·|x) [∇θ log πθ (y|x)∇θ log πθ (y|x)⊤ ].
8
Top Singular Subspace LoRA constant
PiSSA constant
Bottom Singular Subspace OLoRA constant
LoRA constant
Training reward
0.0
OLoRA-tail constant
0.0
0.5
0.2
1.0
0.4
1.5
0.6 0.8
2.0
KL Divergence
MiLoRA constant
0.2
Training step
Training step
100 10 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8
10 2 10 3 10 4 10 5 10 6 10 7 10 8 100
200
300
Training step
400
500
100
200
300
Training step
400
500
Figure 5: Singular value scaling exacerbates instability beyond subspace selection. We compare constant-learning-rate DAPO training dynamics for LoRA, PiSSA, OLoRA, MiLoRA, and OLoRA-tail. The first column compares methods associated with the top singular subspace (LoRA, PiSSA, and OLoRA), while the second column compares methods associated with the bottom singular subspace (LoRA, MiLoRA, and OLoRA-tail). The first row reports training reward, and the second row reports KL divergence between the rollout policy and the current policy. Within each subspace, removing singular value scaling, from PiSSA to OLoRA for the top subspace and from MiLoRA to OLoRA-tail for the bottom subspace, mitigates reward collapse and reduces policy drift. These controlled comparisons show that singular value scaling is a distinct source of RLVR instability, beyond the choice of singular subspace. to θ+ , so that each weight matrix W ⊂ θ is updated to W + ∆W . Suppose that the KL divergence term in (6) DKL (πθ+ ∥ πθ ) ≤ K and that, on the update subspace,q F (θ) ⪰ µI for some µ > 0. Then, for K sufficiently 2K small, every weight block W ⊂ θ satisfies ∥∆W ∥F ≤ 1 + o(1) . µ Theorem 4.2 shows that whenp the KL divergence is constrained by K, a corresponding bound is imposed on every weight update, ∥∆W ∥F ≲ 2K/µ. Consequently, an excessively large ∥∆W ∥F will drive DKL (πθ+ ∥ πθ ) beyond the budget K, potentially violating the conservative-update requirement and destabilizing training (see Section 3.2). As Theorem 4.1 establishes, PiSSA’s singular-value scaling amplifies the weight update ∥∆W1PiSSA ∥F along the principal spectral directions relative to OLoRA’s ∥∆W1OLoRA ∥F , making it more prone to such violation and leaving the safe KL region and therefore more vulnerable to training instability in RLVR. Empirical analysis. Figure 5 confirms this analysis in the principal singular subspace. In the first column, the top panel reports training reward and the bottom panel reports KL divergence between the rollout policy and the current policy for LoRA, PiSSA, and OLoRA. Compared with PiSSA, OLoRA incurs substantially smaller KL divergence, which by Theorem 4.2 implies a correspondingl smaller weight update magnitude. This reduced policy drift partially mitigates the reward collapse observed in PiSSA. To verify that this destabilizing effect is not unique to the principal subspace, we extend the ablation to the minor singular directions via a recent method OLoRA-tail (Lab et al., 2026). As a counterpart to MiLoRA, OLoRA-tail targets the same tail singular subspace but removes singular value scaling: ⊤ A0 = V−r ,
B0 = U−r ,
where U−r and V−r denote the last r columns of U and V , respectively. As shown in the second column of 9
Figure 5, removing singular value scaling in the tail subspace yields markedly more stable dynamics and closely mirrors standard LoRA. Taken together, the comparisons in Figure 5 reveal a nuanced interaction between subspace geometry and singular value scaling. In the principal singular directions, OLoRA delays and weakens the reward collapse observed in PiSSA. In the minor singular directions, OLoRA-tail nearly eliminates the instability observed in MiLoRA, yielding reward and KL dynamics close to standard LoRA. Thus, although certain singular subspaces are more prone to exceeding the KL leash, removing singular value scaling and enforcing orthonormality provides a viable path to more stable RLVR training (Lab et al., 2026). This motivates a deeper study of LoRA optimization dynamics and a principled initialization strategy that utilizes the geometry of singular subspace while avoiding destabilizing singular value scaling.
5
Geometry-Preserving Orthonormal Initialization for LoRA
In this section, we formally analyze the optimization dynamics of LoRA to investigate initialization strategy, studying how initialization can bridge the gap to full fine-tuning while maintaining training stability. The proof has been deferred to the Appendix A. Based on this analysis, we propose orthonormal initialization strategy that preserves the geometry of the pretrained weight space while avoiding singular value scaling.
5.1
Optimization Dynamics of LoRA
Since LoRA constrains updates to a rank-r subspace with r < n, it often cannot exactly match full fine-tuning. We quantify this approximation gap with respect to the initialization strategy in Theorem 5.2, aiming to reduce the gap as much as possible. Orthonormal initialization in LoRA leads closer to full fine-tuning. Without loss of generality, we consider a general RLVR objective: LRLVR (θ) := Ex∼D, y∼πθ (·|x) [R(x, y)] − β · KL(πθ ∥πref ),
(9)
where the reward R(x, y) ∈ {0, 1} equals 1 if y is a valid solution to x and 0 otherwise, πref is some reference policy, and β > 0 denotes a KL penalty coefficient Following standard LoRA, we let B0 = 0 and consider a more general initialization for A0 ∈ Rr×n . Let T denote the total number of training iterations and t ∈ {0, 1, . . . , T } the current iteration. Consider a input x and a single linear layer with weight matrix Wt = W0 + Bt At . The forward pass computes logits zt = Wt x, and the policy πθ (y | x) = softmax(zt ) gives the probability of generating output y, where θ represent the model parameters, namely Wt at time t. At ∂L ∂L time t, the gradients with respect to the two LoRA matrices are ∂A = Bt⊤ Gt , ∂B = Gt A⊤ t , where Gt = ∇Wt L denotes the gradient of the loss with respect to Wt . Assumption 5.1. The RLVR loss LRLVR from (9) is L-smooth with respect to logits z = W x. The gradient is bounded: ∥Gt ∥F ≤ M for all t. The assumption on the loss holds for many RLVR objectives with binary rewards, with a detailed clarification provided in Appendix A. The assumption on the gradient is typically enforced via the clipped importance-ratio mechanism, which is widely used in DAPO, GRPO, and other RL algorithms. Theorem 5.2 (LoRA Approximation Error). Let WTfull and WTLoRA be the weights after T steps of full s PT −1 fine-tuning and LoRA under RLVR. Under Assumption 5.1, define ΓT (η) := T1 s=0 1 + Lη∥x∥22 , then for a fixed training horizon T and sufficiently small η, 1 2 WTLoRA − WTfull F ≤ M η ΓT (η) In − A⊤ 0 A0 2 + O(η ). T
(10)
Furthermore, for A0 ∈ Rr×n with r < n, In − A⊤ 0 A0 2 ≥ 1 holds with equality if A0 has orthonormal rows, i.e., A0 A⊤ = I . r 0
10
Theorem 5.2 shows that when the learning rate η is sufficiently small, the approximation error between LoRA and full fine-tuning is controlled by ΓT (η)∥In − A⊤ 0 A0 ∥2 . The factor ΓT (η) is independent of the initialization. Thus, orthonormal initialization of A0 minimizes the initialization-dependent term to 1. With standard LoRA initialization B0 = 0, Theorem 5.2 implies that orthonormal initialization A0 minimizes the approximation gap between LoRA and full fine-tuning in RLVR, making LoRA’s performance closer to that of full fine-tuning. Orthonormal initialization stabilizes RLVR training. Beyond minimizing the gap between LoRA and full fine-tuning, orthonormal initialization also controls the update magnitude at each step, which is critical for stable training under RLVR. Proposition 5.3 (Bounded Weight Updates for Orthonormal Initialization). For the LoRA parameterization ∆W LoRA = BA with B0 = 0 and row-orthonormal A0 , the first-step LoRA update satisfies ∆W1LoRA F = η G0 A⊤ 0 A0 F ≤ η ∥G0 ∥F ,
(11)
where G0 = ∇W0 L. This result shows that, at the first iteration, orthonormal initialization guarantees that the LoRA weight update ∆W1LoRA F does not exceed that of full fine-tuning in Frobenius norm. This controlled first-step behavior mitigates abrupt policy shifts and supports stable RLVR fine-tuning.
5.2
Geometry-Preserving Orthonormal Initialization for LoRA
The theoretical insights above indicate that orthonormal initialization of A0 is beneficial for low-rank finetuning in RLVR, both in minimizing the gap to full fine-tuning and in ensuring bounded weight updates. Motivated by this, we propose two initialization schemes that enforce orthonormality of A0 while setting B0 = 0m×r , illustrated in Figure 1 alongside comparisons to prior works. To preserve the geometric structure of the pretrained weight matrix W0 , both schemes are derived from its SVD. • Principal orthonormal initialization (LoRA-RLPO). We initialize the adapter A0 using the principal singular vectors: B0 = 0m×r , A0 = Vr⊤ , where Vr ∈ Rn×r contains the top-r right singular vectors of W0 . This preserves the geometric information of the pretrained model by aligning the adapter with the principal directions of W0 . The design is similar in spirit to PiSSA, as both retain the top-r singular directions of W0 ; however, PiSSA additionally incorporates the singular value scaling. • Minor orthonormal initialization (LoRA-RLMO). Analogously, we define an initialization targeting the minor subspace: ⊤ B0 = 0m×r , A0 = V−r , where V−r ∈ Rn×r consists of the bottom-r right singular vectors of W0 . While MiLoRA also targets the minor subspace, it incorporates singular-value scaling and a nonzero B0 ; in contrast, LoRA-RLMO adopts an orthonormal A0 with B0 = 0.
6
Experiments and Analysis
In this section, we conduct experiments to validate our theoretical findings across various benchmarks. Experimental setup. We fine-tune DeepSeek-R1-Distill-Qwen-1.5B using DAPO (Yu et al., 2026) on DAPO-Math-17k (Yu et al., 2026) with rank r = 16 LoRA applied to all linear layers. We compare standard LoRA, PiSSA, MiLoRA, LoRA-RLPO, and LoRA-RLMO across five mathematical reasoning benchmarks: GSM8K (Cobbe et al., 2021) (1,319 samples), MATH500 (Hendrycks et al., 2021) (500 samples, mean@4), and AIME 2022/2023/2024 (30 samples each, mean@32) (Zhang and Math-AI, 2024). Full experimental details are provided in Appendix C. 11
LoRA
MiLoRA
LoRA-RLMO (Ours)
PiSSA
LoRA-RLPO (Ours)
B0 A0
0 N (0, n1 )
1/2 U−r Σ−r 1/2 ⊤ Σ−r V−r
0 ⊤ V−r
1/2 Ur Σr 1/2 Σr Vr⊤
0 Vr⊤
GSM8K@1 MATH500@4 AIME22@32 AIME23@32 AIME24@32 Avg
75.64±1.25 81.93±7.56 43.33±6.67 38.89±5.09 72.22±3.85 62.40±2.96
75.41±1.35 76.47±6.79 28.89±1.92 30.00±3.33 47.78±6.94 51.71±0.98
76.42±0.76 86.80±2.31 42.22±1.92 41.11±5.09 72.22±1.92 63.76±1.64
9.65±8.20 13.20±9.85 0.00±0.00 0.00±0.00 0.00±0.00 4.57±3.26
75.59±1.14 87.33±1.81 46.67±3.33 42.22±6.94 73.33±0.00 65.03±0.55
Table 1: Comparisons of SVD-based LoRA initialization methods with cosine learning rate decay. Results are reported as mean±std .
Figure 6: KL divergence during training for different initialization methods. PiSSA shows the highest KL divergence, MiLoRA exhibits intermediate KL growth, and LoRA remains relatively stable. Our proposed methods (LoRA-RLMO and LoRA-RLPO) maintain the lowest KL trajectories overall, indicating improved training stability. Performance. Table 1 compares SVD-based LoRA initialization methods at step 500 across five mathematical reasoning benchmarks, reporting the mean and standard deviation over multiple seeds. Overall, LoRA-RLPO achieves the highest average accuracy (65.03±0.55 %), followed by LoRA-RLMO (63.76±1.64 %) and LoRA (62.40±2.96 %). LoRA-RLPO obtains the best performance on MATH500, AIME22 , AIME23, and AIME24 , while LoRA-RLMO achieves the strongest GSM8K result. MiLoRA consistently trails behind LoRA on average, and PiSSA performs substantially worse than the other methods in this setting. These results show that our geometry-preserving orthonormal initializations improve the stability and effectiveness of SVD-informed LoRA variants for RLVR. Training stability. Figure 6 monitors and compares the KL divergence term DKL (πθ+ ∥ πθ ) in (6) during training for different initialization methods. PiSSA shows the largest KL divergence by a wide margin, while MiLoRA exhibits intermediate KL growth. LoRA, LoRA-RLMO, and LoRA-RLPO remain in a low-KL regime throughout training. Among them, our proposed methods are particularly stable, with KL trajectories that are consistently comparable to, and even lower than that of standard LoRA. Together with their stronger final evaluation results, these observations indicate that geometry-preserving initialization supports both stable optimization and improved downstream performance in RLVR. Ablation: orthonormality versus SVD geometry. The theoretical analysis implies that an orthonormal A0 minimizes the approximation error to full fine-tuning. To disentangle the two ingredients of the proposed LoRA-RLPO and LoRA-RLMO: orthonormality and SVD-based geometry information, and determine whether orthonormality alone improves RLVR training or the geometry information also contributes, we introduce two
12
Bottom-r Subspace Similarity
Top-r Subspace Similarity
LoRA 1.0
LoRA-RLMO (Ours)
LoRA-RLPO (Ours)
Top-r Singular Vectors - Layer-wise
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0 0
1.0
4
8
12
16
20
Layer Number
0.0
24
Bottom-r Singular Vectors - Layer-wise
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0 0
4
8
12
16
20
Layer Number
0.0
24
Wavelet-LoRA
DCT-LoRA
Top-r Singular Vectors - All Linear Layers
A oRA oRA urs) urs) LoR CT-L O (O avelet-L O (O P D M L L W A-R A-R LoR LoR
Bottom-r Singular Vectors - All Linear Layers
A oRA oRA urs) urs) LoR CT-L O (O avelet-L O (O P D M L L R W AA-R LoR LoR
Figure 7: Subspace similarity between learned adapters and singular vectors of pretrained weights W0 . Top row: similarity with principal (top-r) right singular vectors. Bottom row: similarity with minor (bottom-r) right singular vectors. Left column shows per-layer similarity; right columns show averages over all linear layers. model-agnostic baselines: DCT-LoRA and Wavelet-LoRA. Both of them satisfy the orthonormality condition A0 A⊤ 0 = Ir but use no information from the pretrained weight matrix W0 . DCT-LoRA initializes B0 = 0 and A0 = Dr , where D ∈ Rn×n is the orthonormal Discrete Cosine Transform (DCT) matrix (Ahmed et al., 2006). Its entries are (p 1/n, i = 0, π(2j + 1)i Dij = αi cos , αi = p 2n 2/n, i = 1, . . . , n − 1, for i, j = 0, . . . , n − 1. Here, Dr ∈ Rr×n denotes the first r rows of D. Wavelet-LoRA initializes B0 = 0 and constructs A0 as a row-orthonormal matrix from a Haar-wavelet-transformed random basis. Let G ∈ Rr×n be Gaussian and let H denote the row-wise Haar wavelet transform. With QR = H(G)⊤ , we set A0 = Q⊤ , so that A0 A⊤ 0 = Ir . We first verify that LoRA-RLPO and LoRA-RLMO indeed leverage and preserve the geometric structure of the pretrained weights by steering the optimization toward a specific singular subspace, whereas the model-agnostic variants DCT-LoRA and Wavelet-LoRA do not. To this end, we measure the subspace similarity between the learned model parameter A and the singular vectors of pretrained matrix W0 . For each layer, we compute the similarity as ∥AV ∥F /∥A∥F and report the average across all layers. As shown in Figure 7, LoRA-RLPO maintains high similarity with the principal singular vectors throughout training, consistent with its initialization from Vr , and LoRA-RLMO likewise maintains high similarity with the minor singular vectors. In contrast, DCT-LoRA and Wavelet-LoRA exhibit uniformly low similarity across all singular subspaces, confirming that these model-agnostic bases do not exploit the pretrained weight geometry. 13
Performance comparison
Average accuracy LoRA Wavelet-LoRA DCT-LoRA
Accuracy (%)
80 70 60 50 40
Geometry-Preserving
Average accuracy (%)
90
Model-Agnostic
65.03
65.0 64.5 64.0
63.76
63.62
63.5 63.09 63.0 62.5
30
K M8
GS
00 TH5 MA
AIM
E22
AIM
E23
AIM
E24
e rag
Ave
O
LP
A-R
LoR
O LM
A-R
LoR
A
A
LoR
let
ve Wa
oR T-L
DC
Figure 8: Left: Performance comparison of orthonormal LoRA variants with standard LoRA. Right: Average accuracy comparison between geometry-preserving and model-agnostic methods. Furthermore, Figure 8 (left) shows that both DCT-LoRA and Wavelet-LoRA outperform standard LoRA on most benchmarks, confirming that orthonormality alone improves RLVR training. Figure 8 (right) shows that LoRA-RLMO achieves performance comparable to the model-agnostic methods, while LoRA-RLPO significantly outperforms all others, indicating that principal-subspace alignment provides substantial benefit beyond orthonormality alone. Thus, both orthonormality and SVD geometry contribute to the strong performance of our methods, with the latter yielding a particularly large gain for LoRA-RLPO. These results also suggest that learning in the principal subspace is not inherently harmful in RLVR, but it is considerably more fragile. Although both LoRA-RLPO and PiSSA are initialized in the principal singular subspace, only LoRA-RLPO remains stable and achieves the best overall performance (Table 1). Subspace choice alone therefore does not determine the outcome; rather, our results indicate that geometry-informed orthonormal initialization combined with cosine learning-rate decay is sufficient to make principal-subspace learning both stable and effective in RLVR. Further ablation study. We provide additional ablation study regarding the proposed initialization strategy in Appendix D. First, regarding the recent finding in Yin et al. (2025), we revisits the failure modes of PiSSA and MiLoRA in Appendix D.1: we show that MiLoRA’s initial update is not negligible in norm (see Table 4), and that the tail singular spectrum remains nonzero (see Figure 9), suggesting that its instability is not induced by near-zero tail initialization, differ from the finding in Yin et al. (2025) that may be of independent interest to readers. Second, we evaluates the generalization of the proposed geometry-preserving initializations methods beyond the main 1.5B model on mathematical reasoning tasks in Appendix D.2, including additional task domain code-generation, addtional model families Llama 3.2-3B-Instruct model (Table 5), larger-size Qwen2.5-7B-Instruct model (Table 6), learning-rate sensitivity (Figure 10), and one-time SVD preprocessing cost (Table 7). Finally, we demonstrate that the proposed methods also improve beyond RL fine-tuning frameworks to supervised fine-tuning in Appendix D.3, where we evaluate on GLUE and GSM8K and show with stronger final performance and faster convergence (Table 8 and Figure 11).
7
Conclusion
In this work, we studied why geometry-informed LoRA variants that are effective in supervised fine-tuning can become unstable under RLVR. We identify two primary factors governing this instability: (1) subspace geometry, which fundamentally shapes the optimization trajectory and concentrates the energy of parameter updates in certain subspaces; and (2) singular-value scaling, a distinct destabilizing factor that amplifies gradient magnitudes and drives rapid violations of the KL-divergence constraint. We provide theoretical results showing that orthonormal initialization, paired with the standard LoRA choice of zero-initializing B0 , minimizes the approximation gap to full fine-tuning and helps control update magnitudes. Building on 14
these insights, we propose LoRA-RLPO and LoRA-RLMO, two geometry-preserving orthonormal initialization schemes that retain useful spectral information from the pretrained weights without singular-value scaling. Empirical results on mathematical reasoning benchmarks show that these methods stabilize RLVR training and improve downstream performance. Overall, our findings suggest that orthonormal, geometry-aware initialization offers a principled and effective foundation for low-rank adaptation in RLVR.
Acknowledgments Laixi Shi acknowledges funding support from MERL. Ruijia Zhang thanks Chenliang Li, Di Zhang, Wenbin Wang, Qihan Liu, Pony Ma, Andrew Chen, Qingyu Yin, and the anonymous reviewers for their insightful discussions and constructive feedback, which helped improve this paper. Ruijia Zhang also thanks Mind Lab for broader and larger-scale experimental validation of the theoretical insights in this work.
References Ahmed, N., Natarajan, T., and Rao, K. R. (2006). Discrete cosine transform. IEEE transactions on Computers, 100(1):90–93. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. (2024). Llemma: An open language model for mathematics. Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J. Z., Foster, C., Hsu, J., Jaiswal, M., Lee, W. Y., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang, J., Skowron, A., Tan, S., Tang, X., Wang, K. A., Winata, G. I., Yvon, F., and Zou, A. (2026). Lessons from the trenches on reproducible evaluation of language models. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. Büyükakyüz, K. (2024). Olora: Orthonormal low-rank adaptation of large language models. Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., Sallinen, A., Sakhaeirad, A., Swamy, V., Krawczuk, I., Bayazit, D., Marmet, A., Montariol, S., Hartley, M.-A., Jaggi, M., and Bosselut, A. (2023). Meditron-70b: Scaling medical pretraining for large language models. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Dao, T. (2024). Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, volume 2024, pages 35549–35562. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.
15
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., et al. (2025). Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. Hayou, S., Ghosh, N., and Yu, B. (2024). LoRA+: Efficient low rank adaptation of large models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F., editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 17783–17806. PMLR. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2022). Lora: Low-rank adaptation of large language models. Iclr, 1(2):3. Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML ’02, page 267–274, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. Kalajdzievski, D. (2023). A rank stabilization scaling factor for fine-tuning with lora. Kopiczko, D., Blankevoort, T., and Asano, Y. (2024). Vera: Vector-based random matrix adaptation. In International Conference on Learning Representations, volume 2024, pages 6815–6835. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. (2023). Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626. Lab, M., :, Bo, V., Cao, S., Cao, V., Chen, A., Chen, K., Cheng, C., Chiang, S., Fan, K., Feng, H., Feng, H., Fu, A., Gao, J., Gu, H., Guan, A., Ho, N., Hong, M., Hou, H., Hua, P., Huang, C., Jiang, M., Jiang, N., Jiang, Y., Jin, Q., Kong, F., Lei, A., Lei, K., Li, A., Li, L., Li, R., Li, T., Li, W., Li, Z., Lin, A., Lin, J., Liu, K., Liu, K., Liu, L., Liu, X., Lu, I., Luo, M., Lv, R., Ma, P., Niu, V., Qiu, A., Wang, V., Yang, R., Yao, M., Ye, C., Ye, R., Ye, W., Ying, J., Zeng, D., Zhan, Y., Zhang, A., Zhang, D., Zhang, R., Zhang, S., Zhang, S., Zhang, Y., Zhao, W., Zhou, A., Zhou, A., Zhou, Y., Zhu, X., and Zhuang, M. (2026). On the scaling of peft: Towards million personal models of trillion parameters. Li, X. L. and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597. Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. (2024a). Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. Liu, W., Qiu, Z., Feng, Y., Xiu, Y., Xue, Y., Yu, L., Feng, H., Liu, Z., Heo, J., Peng, S., et al. (2024b). Parameter-efficient orthogonal finetuning via butterfly factorization. In International Conference on Learning Representations, volume 2024, pages 38317–38350. Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., Tang, Y., and Zhang, D. (2025). Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. (2024). Wizardcoder: Empowering code large language models with evol-instruct. In International Conference on Learning Representations, volume 2024, pages 27168–27188.
16
Meng, F., Wang, Z., and Zhang, M. (2024). Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038–121072. Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf, B. (2023). Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320–79362. Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., and He, Y. (2021). {Zerooffload}: Democratizing {billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G. (2024). Code llama: Open foundation models for code. Santacroce, M., Lu, Y., Yu, H., Li, Y., and Shen, Y. (2023). Efficient rlhf: Reducing the memory usage of ppo. Schulman, J. and Lab, T. M. (2025). Lora without regret. Thinking Machines Lab: Connectionism. https://thinkingmachines.ai/blog/lora/. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In Bach, F. and Blei, D., editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889–1897, Lille, France. PMLR. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Shenfeld, I., Pari, J., and Agrawal, P. (2025). Rl’s razor: Why online reinforcement learning forgets less. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. (2025). Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023). Large language models encode clinical knowledge. Nature, 620(7972):172–180. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023). Llama: Open and efficient foundation language models. Wang, H., Li, Y., Wang, S., Chen, G., and Chen, Y. (2025). Milora: Harnessing minor singular components for parameter-efficient llm finetuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4823–4836. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-ofthought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. Wu, F., Xuan, W., Lu, X., Liu, M., Dong, Y., Harchaoui, Z., and Choi, Y. (2026). The invisible leash: Why rlvr may or may not escape its origin.
17
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. (2023). Bloomberggpt: A large language model for finance. Yang, H., Liu, X.-Y., and Wang, C. D. (2025). Fingpt: Open-source financial large language models. Yin, Q., Wu, Y., Shen, Z., Li, S., Wang, Z., Li, Y., Leong, C. T., Kang, J., and Gu, J. (2025). Evaluating parameter efficient methods for rlvr. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al. (2026). DAPO: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244. Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. (2023). Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. Zhang, Y. and Math-AI, T. (2024). American invitational mathematics examination (aime) 2024. Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. (2023). Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277. Zhou, J., Yang, H., Steven, Tang, Xiang, M., Guan, H., and Liu, T. (2024). Understanding and alleviating memory consumption in rlhf for llms. Zhu, H., Zhang, Z., Huang, H., Su, D., Liu, Z., Zhao, J., Fedorov, I., Pirsiavash, H., Sha, Z., Lee, J., Pan, D. Z., Wang, Z., Tian, Y., and Tai, K. S. (2025). The path not taken: RLVR provably learns off the principals. Zhu, J., Greenewald, K., Nadjahi, K., de Ocáriz Borde, H. S., Gabrielsson, R. B., Choshen, L., Ghassemi, M., Yurochkin, M., and Solomon, J. (2024). Asymmetry in low-rank adapters of foundation models. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2020). Fine-tuning language models from human preferences.
18
A
Proof of LoRA Optimization Dynamics
In this section, we provide detailed proofs for the results in Section 5.1. We begin with some preliminary results that will be used in the subsequent proofs. Proof of Assumption 5.1: smoothness of the loss function. We verify that Assumption 5.1 holds for two widely used fine-tuning formulations: supervised fine-tuning (SFT) with the cross-entropy loss, and reinforcement learning with verifiable rewards (RLVR) with the policy-gradient surrogate loss. Consider a linear layer z = W x, and let p(z) := πθ (· | x) = softmax(z) ∈ Rd denote the token-probability vector over a Pd vocabulary of size d, with i-th entry pi (z) = exp(zi ) j=1 exp(zj ). SFT with cross-entropy loss. For a one-hot target token a, the cross-entropy loss is LCE (z) = −
d X
1{i = a} log pi (z) = − log pa (z),
(12)
i=1
where 1{·} is the indicator function. Its gradient and Hessian with respect to the logits are ∇z LCE (z) = p(z) − ea ,
∇2z LCE (z) = H(z) := diag(p(z)) − p(z) p(z)⊤ ,
where ea is the standard basis vector corresponding to token a. Since H(z) is a symmetric positive semidefinite matrix, for any v ∈ Rd we have ⊤
0 ≤ v H(z) v =
d X i=1
pi (z) vi2 −
d X
pi (z) vi
2
≤
i=1
d X
pi (z) vi2 ≤ ∥v∥22 .
(13)
i=1
Hence ∥H(z)∥op = sup∥v∥2 =1 v ⊤ H(z) v ≤ 1, so LCE is 1-smooth with respect to the logits z, i.e., L = 1. RLVR with policy gradient. We consider a variance-reduced policy-gradient objective, a standard surrogate of (5) that is widely used in RL: b y) log πθ (y | x) = Ex∼D, y∼π (· | x) A(x, b y) log py (z) , LPG (θ) = Ex∼D, y∼πθ (· | x) A(x, θ
(14)
b is an estimate of the advantage function. As in standard policy-gradient updates, both A b and the where A sampling distribution are treated as fixed when differentiating with respect to the current logits, (14) thus b Consequently, the Hessian with respect to z is takes the same form as (12), but reweighted by A. b y) diag(p(z)) − p(z) p(z)⊤ , ∇2z LPG = − Ex∼D, y∼πθ (· | x) A(x, and invoking (13) yields b y)| . ∇2z LPG op ≤ Ex∼D, y∼πθ (· | x) |A(x,
(15)
b y)| ≤ 1, giving L = 1 in this case as well. For binary verifiable rewards R ∈ {0, 1}, we have |A(x, First-step LoRA update. We first introduce the following lemma about one-step LoRA update that will be used to compare the LoRA trajectory with the full fine-tuning trajectory. Lemma A.1 (First-step LoRA update). Let W0 ∈ Rm×n be the pretrained weight and consider the LoRA parameterization W = W0 + BA with B0 = 0m×r and A0 ∈ Rr×n . Under one gradient-descent step with step size η, the induced LoRA update satisfies ∆W1LoRA = ∆W1full A⊤ 0 A0 , where ∆W1full = −η ∇W L(W )|W =W0 and ∆W1LoRA = B1 A1 .
19
(16)
Proof. Define the full-weight gradient at initialization as G0 := ∇W L(W )|W =W0 ∈ Rm×n . By the chain rule applied to W = W0 + BA, the gradients of the LoRA factors at (B, A) = (B0 , A0 ) are ∇A L(W0 + BA)|(B,A)=(B0 ,A0 ) = B0⊤ G0 = 0, Therefore, A1 = A0 ,
∇B L(W0 + BA)|(B,A)=(B0 ,A0 ) = G0 A⊤ 0.
(17)
B1 = −ηG0 A⊤ 0.
The first LoRA weight update is consequently ∆W1LoRA = B1 A1 = −ηG0 A⊤ 0 A0 . Since ∆W1full = −ηG0 , the claimed identity follows.
A.1
Proof of Theorem 5.2 and Proposition 5.3
We prove the main approximation bound in Theorem 5.2 first, then establish the additional lower-bound statement in the theorem, and finally prove Proposition 5.3. Step 1: introducing useful notations. Let WtLoRA and Wtfull denote the LoRA and fulfl fine-tuning weights after t gradient-descent steps, initialized from the same pretrained weight: W0LoRA = W0full = W0 . For LoRA, we express the weights as WtLoRA = W0 + Bt At ,
B0 = 0,
A0 ∈ Rr×n .
Define the approximation error at step t between LoRA and full fine-tuning as Et := WtLoRA − Wtfull F . We will prove a recursion for Et and then optimize the initialization-dependent factor ∥In − A⊤ 0 A0 ∥2 . Fixing an input x, we denote the logits as ztLoRA := WtLoRA x,
ztfull := Wtfull x.
Throughout this section, we use the following notations for brevity: := ∇W L(W )|W =W LoRA = gtLoRA x⊤ , GLoRA t t
:= ∇W L(W )|W =W full = gtfull x⊤ , Gfull t t
where gtLoRA and gtfull are the corresponding gradients with respect to logits. Step 2: Proof of the main results in (10). We next compare the LoRA and full fine-tuning dynamics step by step and then unroll the resulting recursion. The full fine-tuning update is full Wt+1 = Wtfull − ηGfull t . For LoRA, gradient descent on the factors gives At+1 = At − ηBt⊤ GLoRA , t
Bt+1 = Bt − ηGLoRA A⊤ t t .
With the above results in hand, combined with the fact that B0 = 0, we have for any fixed horizon T and sufficiently small η, Bt = O(η), At = A0 + O(η 2 ), 0 ≤ t ≤ T. 20
Thus the LoRA weight increment satisfies LoRA Wt+1 − WtLoRA = Bt+1 At+1 − Bt At 3 = −ηGLoRA A⊤ t 0 A0 + O(η ).
(18)
Comparing the two dynamics, we obtain LoRA full full Wt+1 − Wt+1 = WtLoRA − Wtfull − ηGLoRA (A⊤ − GLoRA + O(η 3 ). t 0 A0 − In ) + η Gt t
(19)
Taking Frobenius norms and applying the triangle inequality gives LoRA + O(η 3 ) Et+1 ≤ Et + η GLoRA (In − A⊤ − Gfull t 0 A0 ) F + η Gt t F LoRA + O(η 3 ) − Gfull In − A ⊤ ≤ Et + η GLoRA t 0 A0 2 + η G t t F F LoRA + O(η 3 ), ≤ Et + M η In − A⊤ − Gfull 0 A0 2 + η G t t F
(20)
where the second inequality uses ∥GLoRA ∥F ≤ M in Assumption 5.1. Then it remains to bound the gradient t discrepancy. Invoking GLoRA − Gfull = gtLoRA − gtfull x⊤ , t t we have
GLoRA − Gfull = gtLoRA − gtfull 2 ∥x∥2 . t t F
Applying the L-smoothness of the loss with respect to logits in Assumption 5.1, one has gtLoRA − gtfull 2 ≤ L ztLoRA − ztfull 2 = L (WtLoRA − Wtfull )x 2 ≤ LEt ∥x∥2 . Substituting the above results into (20) yields 3 Et+1 ≤ 1 + Lη∥x∥22 Et + M η In − A⊤ 0 A0 2 + O(η ).
(21)
To continue, as E0 = ∥W0 − W0 ∥F = 0, recursively applying (21) gives ET ≤ M η
−1 TX (1 + Lη∥x∥22 )s .
3 In − A⊤ 0 A0 2 + O(η )
s=0
Dividing by T and using
T −1
ΓT (η) := gives
s 1 X 1 + Lη∥x∥22 T s=0
ET 1 2 WTLoRA − WTfull F ≤ M η ΓT (η) In − A⊤ = 0 A0 2 + O(η ). T T
Step 3: lower bound of In − A⊤ 0 A0 2 . It remains to justify the lower bound on the initializationdependent factor and the case of equality for row-orthonormal initialization. Since rank r < n, rank(A⊤ 0 A0 ) ≤ rank(A0 ) ≤ r < n. ⊤ Therefore, A⊤ 0 A0 has at least n − r zero eigenvalues, so In − A0 A0 has at least n − r eigenvalues equal to 1. ⊤ Therefore, as In − A0 A0 is a symmetric matrix, its spectral norm is equal to the largest absolute value of its eigenvalues, which is at least 1, namely,
In − A⊤ 0 A0 2 ≥ 1.
(22)
⊤ If A0 has orthonormal rows, then A0 A⊤ 0 = Ir and A0 A0 is the orthogonal projector onto row(A0 ). Hence ⊤ ⊥ In − A0 A0 is the orthogonal projector onto row(A0 ) , and
In − A⊤ 0 A0 2 = 1. Combining this equality with (22) shows that row-orthonormal initialization attains the minimum possible value of the initialization-dependent factor In − A⊤ 0 A0 2 for the gap between the LoRA and full-fined models. 21
Step 4: Proof of Proposition 5.3. The proposition follows by applying Lemma A.1 and using the fact that A⊤ 0 A0 is a projection under row-orthonormal initialization. Define G0 := ∇W L(W )|W =W0 . ⊤ If A0 has orthonormal rows, then A⊤ 0 A0 is an orthogonal projection and ∥A0 A0 ∥2 = 1. By Lemma A.1,
∆W1LoRA = −ηG0 A⊤ 0 A0 . Using the mixed Frobenius–spectral norm inequality, ∥AB∥F ≤ ∥A∥F ∥B∥2 , we obtain
B
⊤ ∆W1LoRA F = η∥G0 A⊤ 0 A0 ∥F ≤ η∥G0 ∥F ∥A0 A0 ∥2 = η∥G0 ∥F .
Proof of Gradient Amplification of PiSSA over OLoRA
In this section, we prove Theorem 4.1, which explains why PiSSA is more aggressive than OLoRA even when both are initialized on the same principal singular subspace. We prove the first-order comparison between PiSSA and OLoRA under the residual parameterization W = (W0 − B0 A0 ) + BA, which ensures that both methods start from the same effective weight W0 . For both PiSSA and OLoRA, for a fixed input x, define G0 := ∇W L(W )|W =W0 = gx⊤ . Then we can represent G0 by expanding g and x in the singular-vector bases of W0 . Let W0 = U ΣV ⊤ , with the left singular vectors {ui } and right singular vectors {vj } form orthonormal bases. Denoting αi := u⊤ i g, we can express g and x in the SVD basis as X X g= (u⊤ α i ui , i g)ui = i
(23)
βj := vj⊤ x,
x=
X
i
(vj⊤ x)vj =
X
j
βj vj ,
(24)
j
Therefore, G0 = gx⊤ =
X
α i ui
X
i
j
βj vj
⊤
=
X
αi βj ui vj⊤ ,
(25)
i,j
where matrices {ui vj⊤ }i,j are orthonormal under the Frobenius inner product. To continue, recalling the gradient for the first-step in (17) gives B1 = B0 − ηG0 A⊤ 0,
A1 = A0 − ηB0⊤ G0 ,
leading to the update 2 ⊤ ⊤ ∆W1 = B1 A1 − B0 A0 = −η B0 B0⊤ G0 + G0 A⊤ 0 A0 + η G0 A0 B0 G0 .
(26)
Thus, up to second-order terms in η, the update is governed by the two projection factors B0 B0⊤ and A⊤ 0 A0 . Before proceeding to the main proofs, we introduce two lemmas: the first gives the one-step updates for PiSSA and LoRA, while the second compares their coefficients.
22
First-step updates for PiSSA and OLoRA. Let R = {1, . . . , r} denote the retained principal components. 1/2
1/2
Lemma B.1 (PiSSA and OLoRA first-step updates). For PiSSA with B0 = Ur Σr and A0 = Σr Vr⊤ , and for OLoRA with B0 = Ur and A0 = Vr⊤ , the first-step updates satisfy X ∆W1s = −η csij αi βj ui vj⊤ + O(η 2 ), s ∈ {PiSSA, OLoRA}, i,j
where cPiSSA = σi 1{i∈R} + σj 1{j∈R} , ij 1/2
cOLoRA = 1{i∈R} + 1{j∈R} . ij 1/2
Proof. For PiSSA, applying B0 = Ur Σr
and A0 = Σr Vr⊤ gives
B0 B0⊤ = Ur Σr Ur⊤ ,
⊤ A⊤ 0 A0 = Vr Σr Vr ,
σi 1{i∈R} αi βj ui vj⊤ ,
G0 Vr Σr Vr⊤ =
leading to the two factors Ur Σr Ur⊤ G0 =
X
X
i,j
σj 1{j∈R} αi βj ui vj⊤ .
i,j
Substitution into (26) yields ∆W1PiSSA = −η
X
σi 1{i∈R} + σj 1{j∈R} αi βj ui vj⊤ + O(η 2 ).
i,j
Analogously, for OLoRA,
B0 B0⊤ = Ur Ur⊤ ,
⊤ A⊤ 0 A0 = Vr Vr ,
leading to Ur Ur⊤ G0 =
X
1{i∈R} αi βj ui vj⊤ ,
G0 Vr Vr⊤ =
i,j
X
1{j∈R} αi βj ui vj⊤ .
i,j
Substitution into (26) yields ∆W1OLoRA = −η
X
1{i∈R} + 1{j∈R} αi βj ui vj⊤ + O(η 2 ).
i,j
Lemma B.2 (Coefficient comparison). For all (i, j), cPiSSA ≥ σr cOLoRA . ij ij Proof. If i ∈ R, then σi ≥ σr ; if j ∈ R, then σj ≥ σr . Therefore, cPiSSA = σi 1{i∈R} + σj 1{j∈R} ≥ σr 1{i∈R} + 1{j∈R} = σr cOLoRA . ij ij
Now we are positioned to prove the main result.
23
Proof of Theorem 4.1. ApplyingLemma B.1 gives X X ∆W1PiSSA = −η cPiSSA αi βj ui vj⊤ + O(η 2 ), ∆W1OLoRA = −η cOLoRA αi βj ui vj⊤ + O(η 2 ). ij ij i,j
i,j
As the basis {ui vj⊤ }i,j is Frobenius-orthonormal, one has X i,j
cPiSSA αi βj ui vj⊤ ij
2
= F
X
cPiSSA ij
2
X
αi2 βj2 ,
i,j
cOLoRA αi βj ui vj⊤ ij
i,j
2
= F
X
cOLoRA ij
2
αi2 βj2 .
i,j
Applying Lemma B.2 to all (i, j) gives X
cPiSSA αi βj ui vj⊤ ij
i,j
and then
F
≥ σr
X
cOLoRA αi βj ui vj⊤ ij
i,j
, F
∆W1PiSSA F ≥ σr ∆W1OLoRA F + O(η 2 ).
Furthermore, whenever the leading OLoRA update is nonzero, we have lim inf η→0
C
Experimental Details
C.1
Training details of RLVR
∆W1PiSSA F ∆W1OLoRA F
≥ σr .
Model and dataset for training. We conduct our main DAPO experiments on DeepSeek-R1-DistillQwen-1.5B. All methods are trained on the DAPO-Math-17k dataset, which contains 17,000 mathematical reasoning problems with verifiable answers. We apply LoRA adapters to all linear layers of the policy model, including the attention and MLP projections. Unless otherwise specified, all methods share the same base model, training data, reward function, and optimization setup. Implementation setup. Training prompts are taken from the prompt field of the DAPO-Math-17k parquet file. We apply left truncation with a maximum prompt length of 512 tokens. During rollout generation, the maximum response length is set to 16,384 tokens, matching the long-reasoning regime used by DAPO. For each prompt, we sample 8 candidate responses by drawing directly from the model’s output distribution without any modification. To clarify the sampling hyperparameters, temperature τ controls the randomness of the generation distribution. Top-p (nucleus sampling) restricts sampling to the smallest token set whose cumulative probability exceeds p, while top-k limits the pool to the k most probable tokens (with k = −1 indicating this filter is disabled). Specifically, here we use τ = 1.0, top-p = 1.0, top-k = −1, i.e., no temperature scaling and no vocabulary truncation. All experiments are implemented in the verl (Sheng et al., 2025) framework. We use vLLM (Kwon et al., 2023) for rollout generation and Flash Attention (Dao, 2024) for efficient attention computation. Training is run on 8 NVIDIA A100-SXM4-80GB GPUs with FSDP (Zhao et al., 2023). The rollout engine uses tensor parallelism of size 2 during training (Shoeybi et al., 2019). We enable gradient checkpointing, remove-padding optimization, chunked prefill, dynamic batch sizing, actor parameter offload, actor optimizer offload, and reference parameter offload (Chen et al., 2016; Ren et al., 2021). Hyperparameter settings. We use two training configurations for our DAPO experiments. The main configuration uses a constant learning-rate schedule and serves as the primary setting for comparing LoRA, PiSSA, and MiLoRA. We additionally run cosine-decay variants to study the effect of the learning-rate schedule, keeping all remaining hyperparameters matched to the corresponding constant learning-rate runs. Unless otherwise stated, all methods use the DAPO training objective with group-relative advantage estimation,
24
LoRA-style adapters on all linear layers, 8 responses per prompt, and the same DAPO-style objective without an explicit KL reward penalty or actor KL loss. In the constant learning-rate setting, we train for 500 optimization steps using AdamW with no warmup, weight decay 0.1, and gradient clipping at 1.0. For the standard LoRA baseline and the rank-16 variants, we use learning rate 1 × 10−5 , rank r = 16, and scaling parameter α = 32, corresponding to an effective scaling factor α/r = 2. For PiSSA and MiLoRA, we follow the commonly used configuration with learning rate 1 × 10−5 , rank r = 16, and α = 32. LoRA dropout is set to 0.0 for all methods. The clipping range is asymmetric, with a lower clip ratio of 0.2 and an upper clip ratio of 0.28. The prompt batch size is 128, and the PPO mini-batch size is 32. In the cosine-decay setting, we keep the optimizer, warmup, weight decay, batch size, rollout configuration, and adapter configuration matched to the corresponding constant-LR run, replacing only the constant schedule with cosine decay. For our 1.5B cosine-decay runs, the initial learning rate is 1 × 10−5 , warmup is 0, and weight decay is 0.1. Table 2: Hyperparameters for the DAPO 1.5B experiments. Hyperparameter
Constant-LR setting
Cosine-decay setting
Model and Software Base model Training framework Inference engine Flash Attention PyTorch Hardware
DeepSeek-R1-Distill-Qwen-1.5B verl 0.7.0.dev vLLM 0.11.0 2.8.1 2.8.0+cu126 8 × NVIDIA A100-SXM4-80GB
DeepSeek-R1-Distill-Qwen-1.5B verl 0.7.0.dev vLLM 0.11.0 2.8.1 2.8.0+cu126 8 × NVIDIA A100-SXM4-80GB
Optimization Optimizer Learning rate Learning rate schedule Warmup steps Weight decay Gradient clipping Training steps
AdamW 1 × 10−5 Constant 0 0.1 1.0 500
AdamW 1 × 10−5 Cosine decay 0 0.1 1.0 500
Batch Size Prompt batch size PPO mini-batch size Responses per prompt
128 32 8
128 32 8
GRPO / DAPO Advantage estimator KL reward coefficient Actor KL loss coefficient Clip ratio lower / upper Clip ratio c Loss aggregation Entropy coefficient Overlong buffer length Overlong penalty factor
GRPO 0.0 0.0 0.2/0.28 10.0 Token mean 0 4096 1.0
GRPO 0.0 0.0 0.2/0.28 10.0 Token mean 0 4096 1.0
Adapter Configuration Adapter type Rank (r) Alpha (α) Target modules
LoRA-style 16 32 All linear layers
LoRA-style 16 32 All linear layers
25
Table 2 Hyperparameter
Constant-LR setting
Cosine-decay setting
Dropout Bias
0.0 None
0.0 None
Training Rollout Generation Max prompt length Max response length Temperature Top-p Top-k Rollout tensor parallel size Rollout GPU memory utilization Chunked prefill
512 16384 1.0 1.0 −1 2 0.75 Enabled
512 16384 1.0 1.0 −1 2 0.75 Enabled
C.2
Evaluation setup
Evaluation benchmarks.
We evaluate all methods on five mathematical reasoning benchmarks:
• GSM8K (Cobbe et al., 2021): Grade-school math word problems requiring multi-step arithmetic reasoning. We use the full test set of 1,319 examples and report pass@1 with greedy decoding. • MATH500 (Hendrycks et al., 2021): A 500-problem subset of the MATH benchmark covering algebra, geometry, counting, probability, number theory, and precalculus. We report pass@4. • AIME 2022/2023/2024: Competition-level mathematical reasoning problems from the American Invitational Mathematics Examination. Each year contains 30 problems. We report pass@32. Table 3: Evaluation settings for the DAPO 1.5B experiments. Benchmark
Metric
Samples
Temperature
Top-p
Max response length
GSM8K MATH500 AIME 2022–2024
pass@1 pass@4 pass@32
1 4 32
0 0.6 0.6
1.0 0.95 0.95
4096 8192 16384
Evaluation setup and metrics. We report pass@1 with greedy decoding for GSM8K, pass@4 with temperature sampling for MATH500, and pass@32 with temperature sampling for AIME. Specifically, for the greedy decoding in GSM8K, we set τ = 0 and top-p = 1.0. For the temperature sampling in MATH500 and AIME, we utilize τ = 0.6 and top-p = 0.95. Across all benchmarks, top-k is disabled (k = −1) and a maximum prompt length of 1024 tokens is used. To accommodate the increasing reasoning complexity and expected output lengths, the maximum response length is set to 4096 for GSM8K, 8192 for MATH500, and 16,384 for AIME. For answer scoring, we use the DAPO math reward implementation provided in verl. GSM8K is scored with flexible numerical extraction to avoid undercounting correct answers that do not follow the strict #### answer prefix. For MATH500 and AIME, we use the default mathematical answer normalization and exact-match scoring implemented by the DAPO reward function.
26
Layer 1
Top 16 singular values
Layer 14
Layer 28
Tail 16 singular values
1.75
10
Mean singular value
Mean singular value
1.50 8 6 4 2 0
1.25 1.00 0.75 0.50 0.25
up_proj
gate_proj
Module
0.00
down_proj
up_proj
gate_proj
Module
down_proj
Figure 9: Mean singular values of MLP projection weights in DeepSeek-R1-Distill-Qwen-1.5B (28 layers total) across representative layers 1, 14, and 28, corresponding to the first, middle, and last transformer layers, respectively.
D
Additional Ablation Study
D.1
PiSSA and MiLoRA failure analysis
Table 4: Initialization norms (Frobenius) for LoRA and MiLoRA on DeepSeek-R1-Distill-Qwen-1.5B, averaged across all target linear layers at rank r = 16.
∥B0 A0 ∥F
LoRA
MiLoRA
0
5.11
Recent work (Yin et al., 2025) attributes MiLoRA’s failure to its near-zero initialization: the minor singular values are so small that once RL begins, the gradient flow pulls the update toward the principal components, causing a spectral collapse similar to PiSSA. We revisit this explanation on DeepSeek-R1-Distill-Qwen-1.5B and report two observations that differ from this account and may be of independent interest to readers. First, the initial MiLoRA adapter is not negligible in magnitude. As reported in Table 4, MiLoRA has ∥B0 A0 ∥F = 5.11, which differs from the near-zero initialization hypothesis. Moreover, standard LoRA satisfies ∥B0 A0 ∥F = 0 yet remains stable during RL training, suggesting that initial magnitude alone does not account for MiLoRA’s behavior. Second, the trained MiLoRA model retains non-negligible mass in the off-principal subspace. Figure 9 shows that although the top singular components dominate the tail across MLP projections, the tail values remain clearly non-zero. For DeepSeek-R1-Distill-Qwen-1.5B (28 layers total), we examine three representative layers (1, 14, 28) and find that the mean of the bottom-16 singular values ranges from 0.46 to 1.73, depending on projection the module. Together, these observations suggest that the tail spectrum does not appear to vanish, either at initialization or throughout training.
D.2
Further ablations for RLVR
Additional task domains and model families. We further evaluate on Llama 3.2-3B-Instruct and Qwen2.5-1.5B-Instruct (different model families). To assess domain generality, we additionally consider code
27
generation: Llama 3.2-3B-Instruct trained on MBPP-style program synthesis (Austin et al., 2021) with a test-case-based reward. LoRA-RLPO remains stable and effective under this different reward structure. Table 5: Code generation on Llama 3.2-3B-Instruct (MBPP-style, test-case reward). LoRA
LoRA-RLPO(Ours)
LoRA-RLMO(Ours)
43.44 ± 3.10
45.67 ± 1.41
46.11 ± 1.26
Larger model size. We fine-tune Qwen2.5-7B-Instruct using GRPO (Shao et al., 2024) on DAPO-Math17k (Yu et al., 2026), with rank r = 32 LoRA applied to all linear layers. For the 7B experiments, we train for 150 optimization steps using AdamW with a constant learning-rate schedule and no warmup. For fair comparisons, we use a learning rate of 1 × 10−5 and scaling factor α = 64 for standard LoRA, LoRA-RLPO, and LoRA-RLMO. For PiSSA and MiLoRA, we adopt the same learning rate of 1 × 10−5 and α = 64, in line with their standard tuning practices, ensuring a fair comparison across all methods. The effective batch size is 32 (4 prompts per batch with 8 responses sampled per prompt), and we use a KL penalty coefficient of β = 0.001 in the GRPO objective. As shown in Table 6, our proposed geometry-preserving initializations maintain their empirical advantages at this larger scale. Consistent with our observations on the 1.5B model, the SVD-based variants PiSSA and MiLoRA struggle under the strict KL constraints of RLVR, yielding average scores (24.74 and 26.72) substantially below standard LoRA (30.39). In contrast, both LoRA-RLMO and LoRA-RLPO maintain stable optimization dynamics and achieve consistent improvements. Notably, LoRA-RLPO attains the highest average score of 35.96 across all mathematical reasoning benchmarks, with substantial gains on GSM8K and AIME. These results demonstrate that our theoretically motivated initializations scale robustly to 7B-parameter models without additional hyperparameter tuning. LoRA
MiLoRA
LoRA-RLMO (Ours)
PiSSA
LoRA-RLPO (Ours)
B0 A0
0 N (0, n1 )
1/2 U−r Σ−r 1/2 ⊤ Σ−r V−r
0 ⊤ V−r
1/2 Ur Σr 1/2 Σr Vr⊤
0 Vr⊤
GSM8K@1 MATH500@4 AIME22@16 AIME23@16 AIME24@16 Avg
79.13±4.6 52.80±2.3 3.33±2.7 11.11±4.2 5.56±1.6 30.39
57.27±12.9 54.13±1.5 2.22±1.9 11.11±1.9 8.89±1.9 26.72
74.30±12.3 56.73±1.3 7.78±1.6 10.00±0.0 13.33±2.7 32.42
55.60±44.0 50.33±1.1 4.44±1.9 6.67±3.3 6.67±3.3 24.74
85.29±2.4 55.60±2.6 7.78±1.6 13.33±2.7 17.78±6.8 35.96
Table 6: Evaluation results of Qwen2.5-7B-Instruct fine-tuned with GRPO on DAPO-Math-17k. Our proposed LoRA-RLPO and LoRA-RLMO initializations outperform standard LoRA, whereas PiSSA and MiLoRA exhibit performance degradation. Learning rate sensitivity. We conduct a learning-rate sweep on Qwen2.5-7B-Instruct to evaluate the robustness of different initialization methods. Figure 10 reports the average accuracy across learning rates {10−6 , 10−5 , 10−4 }. LoRA-RLPO and LoRA-RLMO consistently outperform standard LoRA at every learning rate. All methods achieve peak performance at 10−5 , which we adopt as the default learning rate for all experiments reported in this paper.
28
Average Accuracy (%)
40 35 30 25 20 15 10 5 0
LoRA LoRA-RLMO LoRA-RLPO
1e-6
1e-5
1e-4
Learning Rate
Figure 10: Learning rate sensitivity comparison. LoRA-RLPO and LoRA-RLMO consistently outperform standard LoRA across learning rates, with peak performance at 10−5 . SVD initialization preprocessing cost. Table 7 reports the SVD preprocessing cost across model sizes. Measurements are conducted with rank r = 32 in bfloat16 precision on a single NVIDIA A100 GPU. Note that LoRA-RLPO and LoRA-RLMO have identical SVD costs. This preprocessing is a one-time cost incurred before training begins. Table 7: SVD preprocessing cost evaluated with r = 32, bfloat16 precision, on a single A100 GPU. Model
Parameters
Wall-Clock Time
Peak GPU Memory
4B 7B 14B
1.8 min 3.5 min 12.3 min
8.3 GiB 16.0 GiB 29.8 GiB
Qwen3-4B-Instruct Qwen2.5-7B-Instruct Qwen2.5-14B-Instruct
D.3
Generalization to supervised fine-tuning (SFT)
To investigate whether the proposed geometry-preserving initializations generalize beyond the RL fine-tuning framework, we conduct supervised fine-tuning (SFT) experiments on Qwen2.5-7B-Instruct. We evaluate across two benchmark categories: • GLUE (CoLA and MRPC): classification tasks evaluating linguistic acceptability and paraphrase detection. • GSM8K: grade-school math reasoning, evaluated via strict exact match using the Language Model Evaluation Harness (Biderman et al., 2026) under the standard Chain-of-Thought (CoT) prompting setup (Cobbe et al., 2021; Wei et al., 2022). As shown in Table 8, both LoRA-RLPO and LoRA-RLMO generalize effectively to the SFT paradigm, consistently outperforming standard LoRA across all three benchmarks. Figure 11 shows the training loss curves across the three SFT tasks. Both LoRA-RLPO and LoRA-RLMO consistently converge faster and reach a lower final loss than standard LoRA. The improvement is most pronounced on the GSM8K reasoning task. Table 8: SFT evaluation results on Qwen2.5-7B-Instruct. Results are reported as mean ± standard deviation across 3 random seeds. Task CoLA (acc.) MRPC (acc.) GSM8K (strict)
LoRA
LoRA-RLPO
LoRA-RLMO
85.46 ± 0.22 86.52 ± 0.25 23.96 ± 8.89
86.42 ± 0.24 88.48 ± 0.25 29.74 ± 0.54
86.48 ± 0.50 87.91 ± 0.51 33.61 ± 2.99
29
Figure 11: Train loss curves across three SFT tasks (3 seeds, shaded region denotes ±1 standard deviation). Both LoRA-RLPO and LoRA-RLMO demonstrate faster convergence and lower final loss than standard LoRA. Experimental setup. For all SFT experiments, we use rank r = 32, α = 64, and a constant learning rate of 1 × 10−5 . Models are trained for 3 epochs with a global batch size of 32 (4 per device × 8 gradient accumulation steps). All evaluations are averaged across three random seeds ({1, 42, 123}).
30