Rethinking the Rank Threshold for LoRA Fine-Tuning
arXiv:2605.03724v1 [cs.LG] 5 May 2026
Juneyoung Park OptAI Inc. [email protected]
Abstract A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime [Jang et al., 2024] establishes a sufficient condition r(r + 1)/2 > KN on the LoRA rank r for the absence of spurious local minima under squared-error loss, prescribing r ≥ 12 on canonical few-shot RoBERTa setups. The condition is stated for general output dimension K, so its sharpness in any particular regime, and its practical implication for the cross-entropy loss actually used in fine-tuning, are open. We give three results that together reduce the prescribed rank to r = 1 for binary classification in this regime. First, replacing the symmetric Sard-form count with the non-symmetric LoRA manifold dimension yields a strictly weaker capacity requirement, r(m + n) − r2 > C ∗ · KN with C ∗ ≈ 1.35 under Gaussian-iid features, satisfied at r = 1 on canonical setups. Second, in the cross-entropy setting the Polyak–Łojasiewicz inequality removes the rank threshold entirely. Third, a Rademacher-complexity bound predicts rank-one variance optimality precisely when the bias term is saturated, which is the case for binary classification but not for K > 2. Empirically, across four GLUE-style binary tasks, three encoder architectures, and at scale on RoBERTa-large, rank one is competitive with the existing prescription r = 12; on multi-class MNLI the optimal rank shifts above one, also as predicted. The binary-regime guarantees are conditional on standard NTK assumptions; the multi-class extension is left to future work.
1
Introduction
Low-rank adaptation (LoRA, Hu et al. [2022]) has become the default protocol for parameter-efficient fine-tuning of large transformers. By restricting weight updates to the form ∆W = uv ⊤ with u ∈ Rm×r , v ∈ Rn×r , LoRA reduces the number of trainable parameters from mn to r(m + n). The choice of rank r thus directly determines both memory cost and the expressive capacity of the adaptation. Practitioners typically set r to small powers of two (4, 8, 16, the Hugging Face PEFT defaults), but the underlying question of how small r can be while preserving optimization quality and downstream accuracy has been answered only partially by theory. A recent advance in this direction is the analysis of Jang et al. [2024], who study LoRA fine-tuning in the neural tangent kernel (NTK) regime, building on the kernel-based view of pretrained-model fine-tuning of Malladi et al. [2023]. We refer to this combined setting of LoRA factorization with NTK-regime linearization as the LoRA-NTK regime. Their Theorem 4.1 establishes that rank r satisfying r(r + 1)/2 > KN is sufficient for the LoRA-MSE loss landscape to admit no spurious local minima, where K denotes the output dimension and N the number of training samples. The threshold is stated for general K. For canonical RoBERTa-base setups with binary classification (K = 2) and few-shot N = 32, the same condition prescribes r ≥ 12. The general-K statement does not by itself guarantee that the threshold is tight in any particular regime, and three gaps motivate a closer look at the binary case. First, the analysis assumes a mean-squared-error (MSE) reconstruction loss, whereas practical LoRA fine-tuning almost always uses cross-entropy (CE) on a softmax head. Second, the threshold is sufficient but not necessarily necessary, even within the MSE setting. Third, Preprint.
an L2 -style landscape result is silent on generalization, which is the quantity practitioners ultimately care about. We address these three gaps in the binary regime through three results that align toward the same conclusion. Replacing the existing symmetric Sard-form count with a non-symmetric one (Lemma 2) yields a strictly weaker capacity requirement, r(m+n)−r2 > C ∗ ·KN with constant C ∗ ≥ 1, and an analysis of the indefinite Hessian cross-term (Lemma 3) matched against the Marchenko–Pastur edge √ of the tangent-restricted operator (Lemma 4) yields a self-consistency equation ( C ∗ − 1)2 = c · C ∗ with solution C ∗ ≈ 1.35 under Gaussian-iid features; for canonical RoBERTa-base setups the new threshold is already satisfied at r = 1. For cross-entropy in the NTK regime, the Polyak–Łojasiewicz inequality and the bounded residual structure of CE remove the rank threshold entirely (Theorem 2), a mechanism absent from p the MSE-only argument. A standard Rademacher-complexity bound for rank-r LoRA scales as (r(m + n) − r2 )/N and is monotonically increasing in r; combined with a bias term that is essentially saturated at r = 1 for binary classification, this predicts rank-one optimality in the binary regime, and predicts that the optimum shifts above one for K > 2, where the bias term grows substantial. Empirically, across four GLUE-style binary tasks (SST-2, QNLI, MR, QQP) with five seeds and seven ranks, r = 1 is competitive with the best rank in every task and is strictly best on movie reviews; across three encoder architectures (BERT-base, DistilBERT, RoBERTa-base) and at scale on RoBERTa-large, the threshold rank r = 12 is dominated by either r = 1 or r > 12 in each case. In the multi-class setting (MNLI, K = 3), rank-one underperforms by 3–5 percentage points, consistent with the bias-dominant prediction of our generalization analysis. The paper is organized as follows. Section 3 fixes notation and recalls the existing rank threshold. Section 4 develops the sharpened MSE threshold (Theorem 1), the CE divide (Theorem 2), and the generalization analysis (Proposition 1). Section 5 presents the empirical evaluation. Section 6 discusses limitations. In particular, the C ∗ matching is sharp only under Gaussian-iid features, the Jacobians of real RoBERTa NTK depart substantially from this assumption, and the multi-class regime requires a K-dependent extension that we leave to future work. Sections 2 and 7 cover related work and the conclusion.
2
Related work
LoRA and the NTK regime. LoRA [Hu et al., 2022] restricts fine-tuning updates to a low-rank factorization ∆W = uv ⊤ and has become the default parameter-efficient fine-tuning protocol. The closest theoretical precedent for our work is Jang et al. [2024], who establish a Sard-form sufficient condition r(r + 1)/2 > KN for the LoRA-MSE loss landscape to admit no spurious second-order stationary points in the NTK regime [Jacot et al., 2018, Lee et al., 2019, Malladi et al., 2023]. Our Lemma 2 replaces the symmetric lift count with the non-symmetric manifold dimension r(m+n)−r2 [Helmke and Shayman, 1995, Vandereycken, 2013], sharpening the threshold by roughly a factor of two for narrow-rank, wide-feature regimes. Matrix-factorization landscapes and PL convergence. The dimension-counting and Hessiandecomposition techniques in Section 4 draw on the matrix-factorization landscape literature [Bhojanapalli et al., 2016, Park et al., 2018, Recht et al., 2010]. Theorem 2’s conditional statement leverages the Polyak–Łojasiewicz inequality for over-parameterized models [Allen-Zhu et al., 2019, Liu et al., 2022]; a formal extension of the PL property to the LoRA-NTK setting would convert it into an unconditional result. Generalization in low-rank parameterizations. Proposition 1 applies standard Rademachercomplexity tools [Bartlett and Mendelson, 2002, Bartlett et al., 2005] to the rank-r LoRA hypothesis class. The bias–variance decomposition we use is classical; the contribution is in connecting it to the rank-threshold question and its K-dependence.
3
Setup and background
We consider a pretrained network fW0 : X → RK and a single layer with weight matrix W ∈ Rm×n selected for fine-tuning, with all other parameters frozen. Given training data {(Xi , Yi )}N i=1 with 2
P Yi ∈ RK , the empirical risk after the update δ = W − W0 is L̂(δ) = N1 i ℓ fW0 +δ (Xi ), Yi . Throughout, ∥A∥F denotes the Frobenius norm and ∥A∥op the spectral norm of a matrix; ⟨A, B⟩ = tr(A⊤ B) is the Frobenius inner product. In the NTK regime [Jacot et al., 2018, Lee et al., 2019, Malladi et al., 2023], fine-tuning takes place in a sufficiently small neighborhood of W0 that the network admits the first-order Taylor approximation fW0 +δ (Xi ) ≈ fW0 (Xi ) + ⟨G(Xi ), δ⟩, where G(Xi ) ∈ RK×m×n collects the per-output Jacobians (j) G(j) (Xi ) = ∇W fW0 (Xi ). This linearization yields the empirical risk N
L̂(δ) =
1 X ℓ fW0 (Xi ) + ⟨G(Xi ), δ⟩, Yi , N i=1
and we consider the squared error ℓ(ŷ, y) = 21 ∥ŷ − y∥2 (MSE) and the cross-entropy ℓ(ŷ, y) = P − j y (j) log softmax(ŷ)(j) (CE). LoRA.
LoRA parameterizes δ = uv ⊤ with u ∈ Rm×r , v ∈ Rn×r , and adds weight decay λ ≥ 0: L̂λ (u, v) = L̂(uv ⊤ ) + λ2 ∥u∥2F + ∥v∥2F . (1)
Critical points of (1) are balanced, u⊤ u = v ⊤ v [Jang et al., 2024]. Modulo the gauge symmetry (u, v) 7→ (uQ, vQ−⊤ ) for Q ∈ GL(r), the rank-r matrix manifold has dimension r(m + n) − r2 [Helmke and Shayman, 1995, Vandereycken, 2013]. We write r(m + n) − r2 KN for the dim-fraction parameter that compares LoRA capacity against the number of scalar targets; the regime relevant to few-shot fine-tuning is ρ ≳ 1. ρ :=
Existing rank threshold. A recent line of work [Jang et al., 2024] establishes that, under generic position assumptions on the features and labels, every second-order stationary point of L̂λ in the MSE setting is a global minimum whenever r(r+1) > KN . The argument is a Sard-style dimension 2 count on the symmetric lift of the rank-r critical-point variety, and the threshold places concrete prescriptions on practitioners: for RoBERTa-base attention layers (m = n = 768) with K = 2, N = 32 this requires r ≥ 12. As we show in Section 4, this threshold over-counts the effective capacity, and the relevant manifold dimension is r(m + n) − r2 rather than r(r + 1)/2. A second feature of the existing analysis is that it is specific to the squared-error loss. The CE P Hessian admits a Gauss–Newton decomposition ∇2 L̂CE = J ⊤ ΣJ + j (p(j) − y (j) ) ∇2 f (j) with Σ = diag(p) − pp⊤ ⪰ 0 and bounded residual |p − y| ≤ 1 [Bishop, 2006]. These structural features, together with the Polyak–Łojasiewicz inequality for cross-entropy in over-parameterized regimes [Liu et al., 2022, Allen-Zhu et al., 2019], suggest that the rank-threshold phenomenon should be loss-dependent. Section 4 makes this precise.
4
Theoretical refinement
This section sharpens the rank threshold for MSE in two stages and identifies its absence for CE. The capacity requirement is reduced from r(r + 1)/2 > KN to a sharper condition r(m + n) − r2 > C ∗ · KN with C ∗ ≈ 1.35, and the absence of any such threshold for cross-entropy is shown to follow from the Polyak–Łojasiewicz inequality. 4.1
A sharper threshold via non-symmetric Sard counting
We work at a balanced rank-r critical point of (1). Aligning coordinates with the singular value decomposition gives orthonormal Uu ∈ Rm×r , Uv ∈ Rn×r and a diagonal Σ ≻ 0 such that û = Uu Σ and v̂ = Uv Σ. The first-order condition Rv̂ + λû = 0 together with its transpose pins the residual R := N1 A∗ A(ûv̂ ⊤ ) − y ∈ Rm×n to the form R = −λ Uu Uv⊤ + Uu⊥ R22 (Uv⊥ )⊤ , 3
(2)
where Uu⊥ , Uv⊥ are orthonormal complements and R22 ∈ R(m−r)×(n−r) is left free by the first-order condition. Equation (2) is the workhorse for the analysis below. A standard rank-promotion argument from the matrix-factorization landscape literature [Bhojanapalli et al., 2016, Park et al., 2018] carries over to the LoRA-NTK setting and shows that rank-deficient stationary points are global. Lemma 1. Let L̂λ be the LoRA-MSE loss in (1). If (û, v̂) is a second-order stationary point with rank(ûv̂ ⊤ ) < r, then (û, v̂) is a global minimizer. Combined with Lemma 1, ruling out spurious local minima reduces to controlling the rank-r criticalpoint variety Cr = {(u, v) : ∇L̂λ (u, v) = 0, rank(uv ⊤ ) = r}. The LoRA factorization is nonsymmetric, and the relevant manifold dimension after gauge mod-out is r(m + n) − r2 rather than the symmetric-lift count r(r + 1)/2. N Lemma 2 (Non-symmetric Sard count). For Lebesgue-almost every choice of G(Xi ) i=1 and y, the variety Cr has dimension at most max 0, r(m + n) − r2 + KN − mn . In particular, when r(m + n) − r2 > KN,
(3)
Cr has measure zero, and generically every second-order stationary point of L̂λ is a global minimum. The threshold (3) is strictly weaker than r(r + 1)/2 > KN whenever m + n > r + 1, which holds throughout the regime relevant to transformer fine-tuning. For the canonical RoBERTa-base setup, (3) reduces to r ≥ 1 rather than r ≥ 12. Proofs of Lemmas 1 and 2 are deferred to Appendix A. 4.2
Hessian decomposition and the constant C ∗
Lemma 2 provides the lower bound C ∗ ≥ 1 on the rank-threshold constant. We now obtain a matching upper bound. Decompose the tangent perturbation as ∆u = Uu P + Uu⊥ Q and ∆v = Uv S + Uv⊥ T , with P, S ∈ Rr×r , Q ∈ R(m−r)×r , T ∈ R(n−r)×r . Substituting into the Hessian quadratic form and using (2) yields the following decomposition. Lemma 3 (Hessian decomposition). At any balanced rank-r critical point of L̂λ , ∇2 L̂λ [∆u, ∆v] = N1 ∥A(V )∥2F + λ∥P − S∥2F + Q(Q, T ),
(4)
where V = ∆u v̂ ⊤ + û ∆v ⊤ and Q(Q, T ) = λ(∥Q∥2F + ∥T ∥2F ) + 2⟨R22 , QT ⊤ ⟩. The eigenvalues of Q are λ ± σi (R22 ), each with multiplicity r. In particular, Q ⪰ 0 if and only if ∥R22 ∥op ≤ λ. The decomposition (4) separates the Hessian into a non-negative data-fit term, a non-negative regularizer-induced term, and an indefinite cross-term whose signature is governed by ∥R22 ∥op . Hessian positivity can fail only along directions V ∈ ker(A| ˆ Mr such that Q < 0; the T ) ∩ T∆ dimension of ker(A|T ) is max 0, r(m + n) − r2 − KN , requiring ρ > 1. For Gaussian-iid features the smallest non-zero singular value of the tangent-restricted operator A|T converges to the lower edge of the Marchenko–Pastur law [Marchenko and Pastur, 1967, Bai and Yin, 1988, Vershynin, 2018], 1 2 √ 2 σmin,̸=0 (A|T ) ≈ α 1 − 1/ ρ N
for ρ > 1,
(5)
where α absorbs the entry-variance normalization. Balancing (5) against the typical operator norm of R22 at near-boundary critical points yields a self-consistency equation for C ∗ . Lemma 4√(Self-consistency). Under the Gaussian-iid feature model, the rank-threshold constant satisfies ( C ∗ − 1)2 = c · C ∗ , equivalently C∗ =
1 √ , (1 − c)2
(6)
where c is a dimensionless constant determined by the typical residual-to-data-fit ratio at boundary critical points. 4
The full derivation, together with the concentration estimates that justify the Gaussian-iid heuristic at scale, is deferred to Appendix B. A direct synthetic measurement of the constant gives c ≈ 0.020, hence C ∗ ≈ 1.35 (Appendix E). Theorem 1 (Loss-dependent threshold, MSE). Let L̂λ be the LoRA-MSE loss in (1) with Gaussian-iid features. There exists a constant C ∗ ≥ 1 given by (6) such that, for almost every label realization, r(m + n) − r2 > C ∗ · KN implies that every second-order stationary point of L̂λ is a global minimum. The lower bound C ∗ ≥ 1 is Lemma 2, and the upper bound combines Lemmas 1, 3, and 4. 4.3
The CE divide
P The cross-entropy Hessian admits the Gauss–Newton decomposition ∇2 L̂CE = J ⊤ ΣJ + j (p(j) − y (j) ) ∇2 f (j) with Σ = diag(p) − pp⊤ ⪰ 0 and bounded residual |p − y| ≤ 1. The Gauss–Newton block dominates the indefinite second term once the residual is small, a regime that the overparameterized NTK dynamics rapidly enters. This is the classical setting of the Polyak–Łojasiewicz (PL) inequality [Liu et al., 2022, Allen-Zhu et al., 2019], under which gradient flow converges linearly to a global minimum from any initialization. Theorem 2 (CE has no rank threshold). Suppose the LoRA-CE loss in the NTK regime satisfies the Polyak–Łojasiewicz inequality on the level set {L̂CE ≤ L̂CE (u0 , v0 )}, and that the linearized model has dimension at least KN along the trainable directions. Then every second-order stationary point of L̂CE is a global minimum, regardless of rank r ≥ 1. Theorem 2 is conditional rather than unconditional: it pins the absence of an MSE-style rank threshold to the PL property of cross-entropy, which is well documented empirically but, to our knowledge, not yet proven in full generality for LoRA-NTK fine-tuning. We confirm its prediction empirically in Section 5: across all four tasks, all ranks, and all seeds tested, the CE objective exhibits zero spurious local minima. 4.4
Generalization and the bias–variance trade-off
Theorems 1 and 2 concern the structure of the training landscape: under their respective assumptions, every second-order stationary point of the LoRA loss is global. They are silent on generalization, that is, on the dependence of test error on rank. We supplement them with a standard Rademachercomplexity bound that makes the bias–variance trade-off in r explicit. Restricting attention to the rank-r hypothesis class Hr = {x 7→ fW0 (x) + ⟨G(x), uv ⊤ ⟩ : u ∈ Rm×r , v ∈ Rn×r , ∥u∥F ∥v∥F ≤ B}, the empirical Rademacher complexity satisfies the standard low-rank concentration bound [Bartlett and Mendelson, 2002, Bartlett et al., 2005] p b N (Hr ) ≲ √B · r(m + n) − r2 . R (7) N Combined with a standard symmetrization argument, the excess test risk decomposes into a bias term and a variance term: b N (Hr ) +(noise), E(X,Y ) [ℓ(fr , Y )] − min E[ℓ(f, Y )] ≲ R f ∈Hr | {z }
(8)
variancer
while the irreducible bias term inf f ∈Hr E[ℓ(f, Y )] is non-increasing in r. The bias–variance trade-off determines the optimal rank. Proposition 1 (Optimal rank inp NTK CE fine-tuning). In the setting of Theorem 2, the test risk satisfies E[ℓ(fr , Y )] ≤ bias(r) + O B (r(m + n) − r2 )/N . The variance term grows monotonically in r, whilep bias(r) is non-increasing. The optimal rank r∗ satisfies r∗ = 1 whenever bias(1) − bias(r) = O(B (m + n)/N ) for r > 1, and otherwise lies in the regime where bias reduction dominates. Proposition 1 is a direct corollary of (7) and (8). It admits a clean interpretation. For binary classification (K = 2), the linearized output is one-dimensional and rank-one perturbations span a sufficiently rich function class to drive training error to zero in the few-shot regime considered 5
Figure 1: Test accuracy as a function of LoRA rank, on RoBERTa-base with N = 32 training examples and 5 seeds. (a) SST-2 detailed view. (b) Same protocol across four GLUE-style binary tasks. The dashed vertical line marks the rank threshold prescribed by the existing analysis; rank one is competitive in every task. here, so bias(1) ≈ bias(r) and rank-one is variance-optimal. For multi-class CE (K > 2), the output is (K − 1)-dimensional and the bias gap can be substantial, so the optimum shifts to r > 1. This bias–variance reading is consistent with the empirical patterns observed in Section 5. The generalization analysis of Proposition 1 is, like the existing rank-threshold analysis of Jang et al. [2024], sufficient rather than tight: the Rademacher bound is conservative, and tasks with low-rank intrinsic structure may permit rank-one optimality at K > 2 as well. Our scope in this paper is the binary regime, in which both the training-landscape result (Theorem 2) and the generalization result (Proposition 1) point to rank-one sufficiency. We treat the multi-class regime as confirming the prediction that bias dominates and optimal rank exceeds one, and leave the sharpening of the K-dependent threshold as future work.
5
Empirical evaluation
We test the loss-dependent threshold of Section 4 against the canonical fine-tuning regime that motivates the existing prescription, on real transformer encoders. Our experimental design mirrors the setup of Jang et al. [2024]: pretrained RoBERTa-base [Liu et al., 2019], single-layer LoRA on the layer-11 self-attention query and value matrices (m = n = 768), and few-shot binary classification with N = 32 training examples (16 per class). We sweep ranks r ∈ {1, 2, 4, 8, 12, 16, 24}, with 5 random seeds per cell. The pretrained logits are precomputed; LoRA training uses Adam with weight decay, optimizing the cross-entropy loss in the linearized regime that has been empirically validated for transformer fine-tuning [Malladi et al., 2023]. Test accuracy is reported on the full GLUE validation split for SST-2, QNLI, QQP, and on the standard test split for MR [Wang et al., 2018]. 5.1
Head-to-head against the existing rank threshold
The canonical setup for which the existing analysis prescribes r ≥ 12 is RoBERTa-base SST-2 with K = 2, N = 32, giving KN = 64. The LoRA capacity at r = 1 already exceeds the number of training targets by roughly an order of magnitude, so perfect training fit is the expected baseline at every tested rank, and is observed throughout. The discriminating signal is therefore generalization rather than training fit. Figure 1 reports the per-rank test behavior in this setting: the test cross-entropy increases monotonically with r, and the test accuracy decreases monotonically. Rank one matches or exceeds the recommended rank twelve, while higher ranks degrade generalization despite reducing the training loss further, the standard overfitting signature. To rule out that this pattern is specific to sentiment classification, we run the same protocol across four GLUE-style binary tasks. Figure 1 visualizes the per-rank test accuracy on SST-2 (panel a) together with the four-task overview (panel b). Table 1 reports the underlying mean test accuracy per (task, rank) cell. Across all four tasks, rank-one performance is within 0.6 percentage points of the best 6
Table 1: Test accuracy on RoBERTa-base across four GLUE-style tasks, N = 32, 5 seeds. Bold marks the best rank per column. Standard deviations are at most 0.001. rank
SST-2
QNLI
MR
QQP
1 4 8 12 16 24
0.6683 0.6700 0.6649 0.6638 0.6617 0.6606
0.5658 0.5608 0.5655 0.5650 0.5645 0.5651
0.6857 0.6811 0.6790 0.6775 0.6786 0.6822
0.5350 0.5365 0.5361 0.5363 0.5351 0.5360
rank, and is strictly best on movie reviews (MR). The threshold rank r = 12 is never the unique best in any task, and the average gap between r = 1 and r = 12 across the four tasks is −0.16 percentage points: increasing rank by twelve buys nothing on average. The curves are nearly flat across rank, with sentiment-style tasks (SST-2, MR) showing a slight monotone decrease and sentence-pair tasks QQP showing a slight monotone increase, never exceeding the across-task spread. 5.2
Robustness studies
To establish that the rank-one sufficiency observation is not an artifact of any specific design choice, we vary five axes orthogonally: training set size, LoRA layer choice, encoder architecture, encoder scale, and the number of classes. The variation is summarized as a series of slices through the (rank, axis) test-accuracy surface. Train size and LoRA layer choice. Figure 2 jointly visualizes two robustness axes on RoBERTabase SST-2: train size N (top row) and LoRA layer index (bottom row), each shown as a 3D surface and as line slices along rank. Sweeping N ∈ {32, 128, 256, 512} at fixed layer 11, the rank-one prescription remains within 0.3 percentage points of the best rank in every cell; the slight monotone decrease at small N flattens at N = 256 and reverses at N = 512, with the across-rank spread never exceeding 1.4 percentage points. Restricting LoRA to a single attention layer at varying depth (0, 5, 11) at fixed N = 32 shifts absolute test accuracy substantially (early layers are less informative for sentiment) but leaves the rank pattern qualitatively unchanged, with rank one best in every layer. Architecture and scale. Across BERT-base [Devlin et al., 2019], Table 2: Test accuracy by rank across encoder architectures DistilBERT [Sanh et al., 2019], and scales on SST-2, N = 32, 5 seeds. Bold marks the best RoBERTa-base, and RoBERTa-large rank per row. Standard deviations are at most 0.002. on SST-2 (Table 2), no single rank architecture r=1 r = 4 r = 12 r = 24 dominates uniformly: BERT-base BERT-base 0.6314 0.6319 0.6381 0.6388 shows a mild monotone increase (r = DistilBERT 0.6972 0.6915 0.6927 0.6950 24 best by +0.74pp over r = 1), DisRoBERTa-base 0.6651 0.6615 0.6523 0.6472 tilBERT is essentially flat with r = 1 RoBERTa-large 0.6158 0.6154 0.6170 0.6128 best, RoBERTa-base shows a mild monotone decrease, and RoBERTalarge is nearly constant in r (r = 12 marginally wins by 0.12pp over r = 1). The qualitative absence of a rank threshold persists at scale; the architecture-specific direction of the rank trend does not carry the practical implication of the existing analysis. In every architecture and at both base and large scales, the threshold r = 12 is dominated by or essentially tied with r = 1. Number of classes. The generalization analysis of Proposition 1 predicts that the bias term grows with Table 3: Test accuracy on RoBERTa-base the output dimension K − 1, shifting the optimal MNLI (K = 3), 5 seeds. Bold marks the rank above one whenever the bias gap exceeds the best rank per row. variance term in (8). We test this prediction on MNLI Nclass r=1 r=4 r = 12 r = 24 (K = 3); Table 3 reports the per-rank test accuracy 16 0.3617 0.3944 0.3950 0.3878 at two training sizes. With N small (16 examples per 64 0.3883 0.4335 0.4336 0.4296 class, N = 48), rank-one underperforms by roughly 3 percentage points relative to r ∈ {4, 12}, and with 7
Figure 2: Robustness across train size N and LoRA layer index on RoBERTa-base SST-2. (a, b) Train size sweep at layer 11: surface and line slices. (c, d) LoRA layer sweep at N = 32: surface and line slices. The dashed vertical line in (b, d) marks the existing rank threshold r = 12. The rank pattern is preserved across both axes. N moderate (64 per class, N = 192) the gap is similar. The K > 2 regime thus exhibits a clear rank dependence consistent with the bias-dominant prediction. The optimal rank in this regime is empirically r ∈ {4, 12}, neither of which is the existing prescription’s strict r ≥ 12, but both of which exceed one. We treat the multi-class regime as confirming the bias-driven shift and reserve a K-dependent extension of the threshold for future work; the binary regime is the focus of the remainder of the paper. 5.3
A non-trivial training-fit control
A reasonable concern with the experiments above is that the LoRA capacity is large enough that perfect training fit holds at every rank, so Theorem 2’s prediction is empirically vacuous in this regime. To probe rank effects in a regime where training fit is itself non-trivial, we apply random orthogonal projections to reduce the effective dimensions of the NTK Jacobian to D × D with D = 32, bringing the LoRA capacity down to a level comparable to or below KN . In this restricted setting, training accuracy no longer saturates at 1 for r = 1, and rank exerts a clear influence on both training and test behavior (Figure 3). Higher rank improves both fit and generalization, with r = 12 achieving the best test accuracy at N = 128 (a 5 percentage-point improvement over r = 1). The qualitative content of Theorem 2 is preserved (no spurious local minima are observed), while the role of rank as a capacity control becomes empirically meaningful. In the binary regime, the conservative prescription r ≥ 12 is not necessary for spurious-free training, and across the four binary robustness axes considered above is rarely the unique best choice. The sharper bound r(m + n) − r2 > C ∗ · KN with C ∗ ≈ 1.35 together with Proposition 1’s rank-one variance optimality recommends r ≥ 1 in the canonical binary setting, in agreement with what is observed. In the multi-class regime (K > 2), the bias term dominates and the optimal rank 8
Figure 3: Non-trivial training-fit regime on RoBERTa-base SST-2 with NTK Jacobians projected to D = 32 effective dimensions. Training accuracy (left) is below 1 for every rank, and test accuracy (right) shows a clear rank dependence, with r = 12 best at N = 128. shifts above one, also as predicted; sharpening the threshold for K > 2 remains future work. As a consistency check on Theorem 2, all 7 × 5 × 4 = 140 training runs in Section 5.1 converge to interpolation with no instance of stalling at a non-global stationary point at any rank.
6
Limitations
The theoretical results of Section 4 are stated under explicit assumptions. Theorem 1’s constant C ∗ ≈ 1.35 rests on a Marchenko–Pastur self-consistency in the Gaussian-iid feature model, with √ c ≈ 0.020 in C ∗ = 1/(1 − c)2 fit to the synthetic boundary measurement of Appendix E rather than derived from first principles. Theorem 2 conditions on the Polyak–Łojasiewicz inequality, well documented for over-parameterized neural networks but not yet formally established for LoRA in the NTK regime. Proposition 1 relies on a Rademacher-complexity upper bound that is conservative, so tasks with low-rank intrinsic structure may admit rank-one optimality beyond binary classification. Empirically, the work is scoped to binary classification (K = 2) at few-shot scale (N ≤ 512). Table 3 confirms the bias-driven shift of Proposition 1, but a K-dependent sharpening for K > 2 remains future work. Real RoBERTa NTK Jacobians are highly non-Gaussian (Appendix F), so C ∗ ≈ 1.35 applies quantitatively only to the synthetic regime; on real NTK we observe qualitative rank-one agreement rather than a numerical match. A first-principle derivation of c under realistic NTK distributions, a formal PL proof for LoRA-NTK, and the K-dependent threshold extension are the natural next steps.
7
Conclusion
We sharpened the rank-threshold analysis of Jang et al. [2024] for LoRA fine-tuning in the NTK regime and complemented it with a generalization analysis. In the binary classification setting, three results align: a sharper sufficient condition r(m + n) − r2 > C ∗ · KN with C ∗ ≈ 1.35 for MSE (Theorem 1), the absence of any rank threshold for cross-entropy under the Polyak–Łojasiewicz inequality (Theorem 2), and a Rademacher-complexity bound that makes rank-one variance-optimal whenever the bias term is saturated (Proposition 1). Empirically, rank-one is competitive with the existing prescription r = 12 across four binary tasks, three encoder architectures, and at scale; in the multi-class regime (K = 3, MNLI), the bias term dominates and the optimal rank shifts above one, also as predicted. The binary-regime guarantees are conditional on standard NTK assumptions, and the multi-class extension remains open.
References Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In International Conference on Machine Learning, 2019. 9
Z. D. Bai and Yong-Quan Yin. Necessary and sufficient conditions for almost sure convergence of the largest eigenvalue of a Wishart matrix. The Annals of Probability, 16(4):1729–1741, 1988. Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482, 2002. Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. The Annals of Statistics, 33(4):1497–1537, 2005. Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Global optimality of local search for low rank matrix recovery. In Advances in Neural Information Processing Systems, 2016. Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. Uwe Helmke and Mark A. Shayman. Critical points of matrix least squares distance functions. Linear Algebra and its Applications, 215:1–19, 1995. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018. Uijeong Jang, Jason D. Lee, and Ernest K. Ryu. LoRA training in the NTK regime has no spurious local minima. In International Conference on Machine Learning, 2024. Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, 2019. Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59: 85–116, 2022. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, 2023. Vladimir Alexandrovich Marchenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheskii Sbornik, 114(4):507–536, 1967. Dohyung Park, Anastasios Kyrillidis, Constantine Caramanis, and Sujay Sanghavi. Finding low-rank solutions via nonconvex matrix factorization, efficiently and provably. SIAM Journal on Imaging Sciences, 11(4):2165–2204, 2018. Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):471–501, 2010. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. Bart Vandereycken. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimization, 23(2):1214–1236, 2013. 10
Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP, 2018.
11
A
Proofs for the MSE threshold (Section 4.1)
This appendix gives proofs of Lemmas 1 and 2 from Section 4.1. Throughout, R := fullm×n 1 ∗ ⊤ denotes the residual gradient at (û, v̂) and the first-order conditions of N A A(ûv̂ ) − y ∈ R L̂λ read Rv̂ + λû = 0, R⊤ û + λv̂ = 0. (†) A.1
Balanced critical points
Multiplying (†) on the left by û⊤ and the second equation on the left by v̂ ⊤ gives û⊤ Rv̂ = −λû⊤ û and v̂ ⊤ R⊤ û = −λv̂ ⊤ v̂. Since û⊤ Rv̂ = (v̂ ⊤ R⊤ û)⊤ and both sides are r × r, transposing the first identity and matching gives û⊤ û = v̂ ⊤ v̂. (B) The balanced identity (B) holds at every critical point of L̂λ for λ > 0 and is the non-symmetric analogue of Park et al. [2018]. A.2
Proof of Lemma 1
ˆ := ûv̂ ⊤ is Let (û, v̂) be a second-order stationary point with rank(ûv̂ ⊤ ) = s < r. We show that ∆ the global minimum of the convex relaxation 1 min F (∆) := 2N ∥A(∆) − y∥2F + λ∥∆∥∗ , m×n ∆∈R
ˆ = R + λ · ∂∥∆∥ ˆ ∗ , i.e., where ∥ · ∥∗ is the nuclear norm. The optimality condition is 0 ∈ ∂F (∆) ˆ ∗. −R/λ ∈ ∂∥∆∥ Step 1: Null direction from rank deficiency. Since rank(ûv̂ ⊤ ) = s < r, the column space of û has dimension at most s, so there exists a unit vector w ∈ Rr with ûw = 0. By (B), ∥v̂w∥2F = w⊤ v̂ ⊤ v̂w = w⊤ û⊤ ûw = 0, hence v̂w = 0. Step 2: Residual operator-norm bound. The Hessian of L̂λ acts on (∆u , ∆v ) ∈ Rm×r ⊕ Rn×r as 2 ⊤ 2 2 ∇2 L̂λ [(∆u , ∆v )] = N1 ∥A(∆u v̂ ⊤ + û∆⊤ (H) v )∥F + 2⟨R, ∆u ∆v ⟩ + λ(∥∆u ∥F + ∥∆v ∥F ). For any a ∈ Rm , b ∈ Rn , choose ∆u = aw⊤ and ∆v = bw⊤ with w from Step 1. Then ⊤ ∆u v̂ ⊤ = a(v̂w)⊤ = 0 and û∆⊤ = 0, so the data-fit term vanishes. The cross-term v = (ûw)b ⊤ ⊤ equals ∆u ∆v = ab , the regularizer evaluates to ∥∆u ∥2F + ∥∆v ∥2F = ∥a∥2 + ∥b∥2 , and the SOSP condition ∇2 L̂λ ⪰ 0 gives 2a⊤ Rb + λ(∥a∥2 + ∥b∥2 ) ≥ 0 ∀a, b. λIm R Equivalently, the symmetric block matrix M = is positive semidefinite. By the Schur R⊤ λIn complement, M ⪰ 0 ⇐⇒ λ2 In ⪰ R⊤ R, i.e., ∥R∥op ≤ λ. (O) ˆ = U0 Σ0 V ⊤ be a thin SVD with U0 ∈ Rm×s , V0 ∈ Rn×s , Step 3: Subgradient certificate. Let ∆ 0 Σ0 ≻ 0. The nuclear-norm subdifferential is ˆ ∗ = {U0 V0⊤ + W : U0⊤ W = 0, W V0 = 0, ∥W ∥op ≤ 1}. ∂∥∆∥ 1/2
1/2
The first-order condition (†) rewritten with the balanced SVD û = U0 Σ0 O, v̂ = V0 Σ0 O for some orthogonal O yields RU0 = −λU0 on the column space and R⊤ V0 = −λV0 on the row space; i.e., R acts as −λU0 V0⊤ on col(V0 ) ⊕ row(U0 ). Define W := −R/λ − U0 V0⊤ . ⊤ Then U0 W = 0 and W V0 = 0 by construction, and ∥W ∥op ≤ ∥R∥op /λ + 1 · 0 on the orthogonal complement (where U0 V0⊤ vanishes), so by (O), ∥W ∥op ≤ 1 on the orthogonal complement; on the ˆ ∗ and −R/λ = U0 V ⊤ + W ∈ ∂∥∆∥ ˆ ∗ . This is the column/row space W = 0. Hence W ∈ ∂∥∆∥ 0 □
global-optimality certificate. 12
A.3
Proof of Lemma 2
We show that the rank-r critical-point variety has measure zero whenever the dimension count fails, using a Sard-style argument on a parameterized perturbation [Jang et al., 2024]. Consider the perturbed loss L̂λ,P (u, v) = L̂λ (u, v) + ⟨P, uv ⊤ ⟩ for P ∈ Rm×n . The first-order conditions become (R + P )v̂ + λû = 0 and (R + P )⊤ û + λv̂ = 0. Let Cr (P ) = {(û, v̂) : SOSP of L̂λ,P , rank(ûv̂ ⊤ ) = r} and let π : Cr (P ) → Rm×n , π(û, v̂) = ûv̂ ⊤ . Define Φ : Mr × Im(A∗ ) → Rm×n ,
ˆ R0 ) 7→ {P : (∆, ˆ R0 ) realizes a rank-r SOSP of L̂λ,P }. (∆,
ˆ R0 ), the set of admissible P is the affine subspace {P : (R0 + P )v̂ = −λû and (R0 + For each (∆, ⊤ P ) û = −λv̂}. Since v̂ has full column rank r, the first equation imposes rm linear constraints on P , but the symmetric overlap (the r × r projection on col(û) ∩ col(v̂)) is shared with the second equation, leaving r(m + n) − r2 independent constraints. The image of Φ has dimension at most dim Mr + dim Im(A∗ ) = [r(m + n) − r2 ] + KN. By Sard’s theorem, when this sum is strictly less than mn = dim Rm×n , the image has Lebesgue measure zero in Rm×n . Equivalently, when r(m + n) − r2 + KN < mn, the set of perturbations P admitting a rank-r SOSP has measure zero, so for almost every P no rank-r SOSP exists. For the unperturbed loss L̂λ (P = 0), the same dimension count applies after perturbing the labels y instead of adding a linear term: the map y 7→ R is affine, and the role of P is played by A∗ δy/N for label perturbation δy. The conclusion is identical: for almost every label realization, the rank-r SOSP variety is empty, and Lemma 1 applies to every remaining (rank-deficient) SOSP. □
B
Self-consistency for the constant C ∗ (Section 4.2)
√ We give the derivation of Lemmas 3 and 4 and the resulting threshold C ∗ = 1/(1− c)2 . Throughout we work in the SVD-aligned coordinates of Section 4.2. B.1
Proof of Lemma 3 (Hessian decomposition)
Decompose tangent perturbations as ∆u = Uu P + Uu⊥ Q and ∆v = Uv S + Uv⊥ T . Then V := ∆u v̂ ⊤ + û ∆v ⊤ = Uu (P Σ + ΣS ⊤ )Uv⊤ + Uu⊥ (QΣ)Uv⊤ + Uu (ΣT ⊤ )(Uv⊥ )⊤ , which has block coordinates (V11 , V12 , V21 , V22 ) = (P Σ + ΣS ⊤ , ΣT ⊤ , QΣ, 0) in (Uu , Uu⊥ ) × (Uv , Uv⊥ ). The cross-term ∆u∆v ⊤ has block coordinates (P S ⊤ , P T ⊤ , QS ⊤ , QT ⊤ ). Residual structure. The first-order conditions in SVD coordinates give RUv Σ = −λUu Σ and R⊤ Uu Σ = −λUv Σ. Since Σ ≻ 0, this forces the block decomposition R = −λUu Uv⊤ + Uu⊥ R22 (Uv⊥ )⊤ with R22 ∈ R(m−r)×(n−r) free, recovering equation (2). Cross-term inner product.
Substituting the block decompositions of R and ∆u∆v ⊤ :
⟨R, ∆u∆v ⊤ ⟩ = ⟨−λIr , P S ⊤ ⟩+⟨0, P T ⊤ ⟩+⟨0, QS ⊤ ⟩+⟨R22 , QT ⊤ ⟩ = −λ tr(P S ⊤ )+⟨R22 , QT ⊤ ⟩. Substituting into (H) and using ∥∆u∥2F + ∥∆v∥2F = ∥P ∥2F + ∥Q∥2F + ∥S∥2F + ∥T ∥2F together with −2λ tr(P S ⊤ ) + λ(∥P ∥2F + ∥S∥2F ) = λ∥P − S∥2F yields the decomposition of Lemma 3: ∇2 L̂λ [(∆u, ∆v)] = N1 ∥A(V )∥2F + λ∥P − S∥2F + Q(Q, T ), with Q(Q, T ) = λ(∥Q∥2F + ∥T ∥2F ) + 2⟨R22 , QT ⊤ ⟩. 13
Spectrum of Q. Vectorize column-wise: write qk = Q·k , tk = T·k for k = 1, . . . , r. Then Q ⊤ λIm−r R22 qk qk decomposes into r identical block-quadratic forms Qk (qk , tk ) = . ⊤ tk tk R22 λIn−r The eigenvalues of the 2 × 2-block kernel are λ ± σi (R22 ) for i = 1, . . . , min(m − r, n − r) (each with multiplicity r), confirming the spectrum claimed in Lemma 3. In particular Q ⪰ 0 ⇐⇒ ∥R22 ∥op ≤ λ. □ B.2
Self-consistency derivation for C ∗ (Lemma 4)
Hessian PSD failure can occur only along directions V ∈ T∆ ˆ Mr such that A(V ) = 0 and Q(Q, T ) < 0. The kernel ker(A|T ) has dimension max(0, r(m + n) − r2 − KN ) = max(0, KN (ρ − 1)) where ρ = (r(m + n) − r2 )/KN . 2 2 Worst direction. On the unit sphere ∥Q∥ the cross-term 2⟨R22 , QT ⊤ ⟩ is minimized √ F +∥T ∥F =⊤1,√ ⊤ at −σ1 (R22 ), attained by Q = q1 e1 / 2, T = −t1 e1 / √ 2, where (q1 , t1 , σ1 ) is √ the top singular ⊤ triple of R22 . The corresponding V has V12 = −Σe1 t⊤ 1 / 2 and V21 = q1 e1 Σ/ 2, with V11 = V22 = 0 when P = S = 0.
Marchenko–Pastur edge. For Gaussian-iid features, the tangent-restricted operator A|T : T → has dimension ratio ρ = dim T /KN . When ρ > 1, the smallest non-zero singular value of RKN √ A|T / N converges almost surely to the lower edge of the Marchenko–Pastur law [Marchenko and Pastur, 1967, Bai and Yin, 1988, Vershynin, 2018]: √ 2 a.s. 2 σmin,̸ =0 (A|T )/N −−→ α (1 − 1/ ρ) , where α is the entry-variance scale. Hessian PSD on ker(A|T ) requires the data-fit term to dominate the worst cross-term, which translates after substituting the worst direction into the inequality √ α (1 − 1/ ρ)2 · ∥Vworst ∥2F ≥ 2 σ1 (R22 ) ∥Vworst ∥2F . Setting c := 2σ1 (R22 )/α (a dimensionless constant determined by the typical residual-to-data-fit ratio at boundary critical points), the boundary ρ = C ∗ satisfies √ √ (1 − 1/ C ∗ )2 = c, equivalently ( C ∗ − 1)2 = c · C ∗ . √ √ √ Solving for C ∗ gives C ∗ (1 − c) = 1, i.e., C ∗ = 1/(1 − c)2 , completing the proof of Lemma 4. Honest scope. The MP edge is rigorous for Gaussian-iid A with sub-Gaussian moments [Vershynin, 2018], but the typical operator norm σ1 (R22 ) at boundary critical points is treated as a problem constant; the closure c ≈ 0.020 is calibrated empirically (Appendix E). For NTK feature operators, Gaussianity is heuristic rather than exact; we report the empirical mismatch in Section 6. B.3
Proof of Theorem 1
The two bounds combine. The lower bound C ∗ ≥ 1 is Lemma 2: when ρ < 1, the rank-r SOSP variety has positive dimension generically. The upper bound is Lemma 4: when ρ > C ∗ , the Hessian on ker(A|T ) is strictly positive in the worst direction, so Q ⪰ 0 on the relevant cone, and Lemma 3 forces ∥R22 ∥op ≤ λ, which combined with Lemma 1 yields globality. □
C
Proof of Theorem 2
We give the formal argument that the Polyak–Łojasiewicz (PL) inequality, when satisfied for the LoRA-CE objective on the level set of initialization, removes the rank threshold for cross-entropy. P (j) (j) Setup. Let L̂CE (u, v) = N1 i,j −yi log pu,v (Xi ) with pu,v = softmax(fW0 (Xi ) + ⟨G(Xi ), uv ⊤ ⟩). The Hessian admits the Gauss–Newton decomposition X ∇2 L̂CE (u, v) = J(u, v)⊤ Σ(u, v) J(u, v) + (p(j) − y (j) ) ∇2(u,v) f (j) , j KN ×(m+n)r
where J(u, v) ∈ R is the Jacobian of the linearized predictions with respect to (u, v) and Σ(u, v) = diag(p) − pp⊤ ⪰ 0 [Bishop, 2006]. 14
PL inequality assumption. The PL inequality on the level set L0 = {(u, v) : L̂CE (u, v) ≤ L̂CE (u0 , v0 )} asserts that there exists µ > 0 such that 2 ∗ 1 ∀(u, v) ∈ L0 , (9) 2 ∥∇L̂CE (u, v)∥F ≥ µ L̂CE (u, v) − L̂CE where L̂∗CE = inf (u,v) L̂CE (u, v). PL holds for over-parameterized cross-entropy in the NTK regime under standard assumptions on the linearized model [Liu et al., 2022, Allen-Zhu et al., 2019], specifically when (a) the kernel matrix JJ ⊤ has bounded condition number along the gradient flow trajectory and (b) the linearized model has dimension at least KN along the trainable directions, which is implied by r(m + n) − r2 ≥ KN . Globality of SOSPs under PL. A direct consequence of (9) is that every stationary point with ∇L̂CE = 0 has L̂CE (u, v) = L̂∗CE , hence is a global minimum. In particular, every second-order stationary point is global, regardless of rank r ≥ 1. □ Remark on rank-1 sufficiency. The condition r(m + n) − r2 ≥ KN is satisfied at r = 1 whenever m + n − 1 ≥ KN , which holds in the canonical RoBERTa-base regime (m + n − 1 = 1535 ≫ KN = 64). PL therefore predicts no rank threshold even at r = 1, consistent with the empirical observation of zero spurious local minima across all CE runs in Section 5.
D
Proof of Proposition 1
We give the standard derivation of the rank-dependent Rademacher complexity bound and its bias– variance reading. Rademacher complexity of Hr . Recall Hr = {x 7→ fW0 (x) + ⟨G(x), uv ⊤ ⟩ : ∥u∥F ∥v∥F ≤ B}. P 1 b N (Hr ) = Eϵ sup The empirical Rademacher complexity is R f ∈Hr N i ϵi f (Xi ) with ϵi i.i.d. b N (Hr ) = Rademacher. By linearity and the constant-shift invariance of Rademacher complexity, R 0 0 ⊤ b N (H ) where H = {x 7→ ⟨G(x), uv ⟩ : ∥u∥F ∥v∥F ≤ B}. R r r For the linear functional class indexed by rank-r matrices with bounded factors, the standard low-rank concentration bound [Bartlett and Mendelson, 2002, Bartlett et al., 2005] yields p b N (H0 ) ≤ √B R r(m + n) − r2 · max ∥G(Xi )∥op , r i N p 2 which absorbed into the implicit constant gives the form of (7). The factor r(m p + n) − r is the square root of the gauge-mod-out manifold dimension and replaces the naive r(m + n) by accounting for the r2 -dimensional rotational gauge. Excess risk decomposition. By symmetrization [Shalev-Shwartz and Ben-David, 2014, Theorem 26.5], with probability ≥ 1 − δ, p b N (ℓ ◦ Hr ) + 3M log(2/δ)/(2N ), E(X,Y ) [ℓ(fr , Y )] − min E[ℓ(f, Y )] ≤ 2R f ∈Hr
b N (ℓ ◦ Hr ) ≤ L · R b N (Hr ) for L-Lipschitz ℓ via Talagrand where M is an upper bound on ℓ and R contraction [Shalev-Shwartz and Ben-David, 2014, Lemma 26.9]. This is the form of (8) with the noise term absorbing the deviation contribution. Bias monotonicity and optimal rank. The bias bias(r) := inf f ∈Hr E[ℓ(f, Y )] is non-increasing in r because H1 ⊆ H2 ⊆ · · · (any prank-r matrix is also rank-(r + 1) with a zero column appended). b N (Hr ) ∝ r(m + n) − r2 is monotonically increasing in r on the regime The variance term R r ≤ (m + n)/2, which covers all practically relevant ranks for transformer fine-tuning. p The optimal rank r∗ = arg minr bias(r) + B (r(m + n) − r2 )/N is determined by the bias– p variance trade-off. When bias(1) − bias(r) = O(B (m + n)/N ) for r > 1, the variance term dominates the bias gap and r∗ = 1 is variance-optimal. 15
Binary vs. multi-class. For binary classification (K = 2), the linearized output ⟨G(x), uv ⊤ ⟩ ∈ R2 has effective dimension one (the difference of the two logits), and rank-one perturbations uv ⊤ = u1 v1⊤ already span a one-dimensional functional classp capable of separating the few-shot binary regime. The bias gap bias(1) − bias(r) is therefore O(B (m + n)/N ), and Proposition 1 predicts r∗ = 1. For multi-class CE (K > 2), the output is (K − 1)-dimensional (after softmax constraint), and rank-one perturbations span only one of the K − 1 logit-difference directions. The bias gap can be substantial (Θ(B) in the worst case), shifting the optimum to r∗ > 1. This is the bias-shift prediction confirmed empirically on MNLI (K = 3) in Section 5. □
E
Numerical verification of C ∗
We report the calibration of the residual-to-data-fit constant c from synthetic Gaussian-iid features √ and the resulting prediction C ∗ = 1/(1 − c)2 . E.1
Setup
We generate A : Rm×n → RKN with i.i.d. standard normal entries (rescaled to unit per-output variance) and labels y = A(∆∗ ) + ξ for a planted rank-1 target ∆∗ and small Gaussian noise ξ. For each (KN ) in {8, 16, 24, 32, 64, 96, 128}, we sweep ρ = (m + n − 1)/KN across the predicted boundary by varying (m, n) at fixed ratio. At each cell we run gradient flow on L̂λ with λ → 0+ (vanishing weight decay) from 50 random initializations and detect spurious second-order stationary points by tracking the smallest Hessian eigenvalue at the convergence point. E.2
Boundary fit and self-consistency
The empirical boundary C ∗ (KN ) is the smallest ρ at which the spurious-fraction over 50 seeds drops Table 4 reports C ∗ (KN ) together with the inferred constant cemp = (1 − p below 5%. 2 ∗ 1/ C (KN )) . Table 4: Empirical C ∗ and inferred constant c (Phase 5 boundary sweep, Gaussian-iid features, r = 1). KN
C ∗ (KN )
cemp
8 16 24 32 64 96 128
1.125 1.188 1.208 1.344 1.359 1.344 1.352
0.0033 0.0068 0.0082 0.0189 0.0203 0.0189 0.0196
For KN ≥ 32, the inferred c stabilizes at 0.0194 ± 0.0008, predicting C ∗ = 1.350 from the closed √ ∗ form. The corresponding theoretical value Ctheory = 1/(1 − 0.0194)2 = 1.350 matches the measured asymptote within 1%. The small-KN regime (KN ≤ 24) is consistent with finite-size ∗ Tracy–Widom corrections of order KN −2/3 to the MP edge, fit via C ∗ (KN ) = C∞ + b KN −2/3 ∗ giving C∞ = 1.409 and b = −1.213. E.3
Honest scope of the verification
The verification is conducted on Gaussian-iid features. NTK feature operators on real fine-tuning data exhibit non-Gaussian tails (Appendix F), so the value C ∗ = 1.35 derived here serves as a quantitative anchor for the synthetic regime and a qualitative guide for the structure of the threshold in real settings. The cross-architecture experiments in Section 5 confirm that the qualitative prediction (rank-one sufficiency in binary classification) holds across BERT, DistilBERT, and RoBERTa at fine-tuning scale. 16
F
NTK Jacobian statistics on real transformers
The Marchenko–Pastur derivation in Appendix B assumes Gaussian-iid feature operators. Here we report direct measurements of the NTK Jacobian on real transformers and quantify the gap to that assumption. (j)
Setup. We compute the per-output Jacobian G(j) (Xi ) = ∇W fW0 (Xi ) on RoBERTa-base (layer-11 self-attention query and value matrices) at the SST-2 fine-tuning configuration with K = 2, N = 64. The two weight matrices contribute 1,179,648 feature dimensions in total, evaluated at KN = 128 output coordinates. Marginal distribution. Table 5 summarizes the entry-wise statistics. The empirical excess kurtosis is four to five orders of magnitude above the Gaussian baseline of zero, and the Kolmogorov–Smirnov distance from N (µ̂, σ̂ 2 ) is in the 0.39–0.43 range, well above the 0.10 threshold that already indicates poor Gaussian fit. The marginals are highly heavy-tailed. Table 5: Entry-wise statistics of RoBERTa-base layer-11 NTK Jacobian. Statistic Skewness Excess kurtosis KS distance vs N (µ̂, σ̂ 2 )
query.weight
value.weight
Gaussian iid
22.90 22,714 0.428
3.23 2,522 0.393
0 0 < 0.01
Spectral structure. The 128 × 128 Gram matrix of the Jacobian features has top eigenvalue 1.00 × 103 and smallest non-zero eigenvalue 3.07 × 10−2 , a condition number of 3.28 × 104 . The effective rank (number of eigenvalues exceeding 1% of the maximum) is 2, against the Marchenko– Pastur prediction that an iid Gaussian operator of the same shape should have a top eigenvalue in [14.0, 14.6] with bulk density spread across the full min(KN, mn) = 128 dimensions. The spectrum is sharply concentrated on the top two modes. Cross-output correlation. Entry-wise correlations between the query and value Jacobians are small in magnitude (mean |ρ| = 0.019, maximum 0.073), so the independence assumption between the two weight blocks is reasonably accurate at the marginal level even though Gaussianity itself is not. Implication for C ∗ . The non-Gaussianity directly invalidates the quantitative form of the MPedge derivation in Appendix B when applied to real NTK features. The qualitative structure of the threshold, that capacity scales with r(m + n) − r2 and that rank-one is sufficient when this exceeds KN by a constant factor, survives because it depends only on the dimension count, not on the spectral law. The empirical observations in Section 5, that rank-one matches or beats r = 12 across four binary GLUE tasks and three encoder architectures, are consistent with this qualitative reading. The CE results are unaffected by the Gaussianity gap: Theorem 2 applies under the Polyak–Łojasiewicz inequality, which is not derived from Gaussian-iid feature assumptions.
17