arXiv:2605.13652v1 [cs.LG] 13 May 2026
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training Namrata Shivagunde∗ [email protected]
Vijeta Deshpande [email protected]
Sherin Muckatira [email protected]
Anna Rumshisky [email protected]
University of Massachusetts Lowell
Abstract Pre-training large language models is dominated by the memory cost of storing fullrank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this, and the space of methods has grown rapidly. A central question remains open: do low-rank methods produce models that generalize comparably to full-rank training, or does the rank constraint fundamentally alter the solutions reached? Existing comparisons rely almost entirely on validation perplexity from single-seed runs, often carried forward from prior literature. Yet perplexity is a poor proxy for solution quality — two methods can match on perplexity while converging to very different loss landscape regions and internal representations. We close this gap by characterizing the solutions found by five low-rank pre-training methods — GaLore and Fira (memory-efficient optimizers), CoLA and SLTrain (architecture reparameterizations), and ReLoRA (adapter-style updates with periodic resets) — against full-rank training at three model scales (60M, 130M, 350M). We evaluate each along 16 metrics across four dimensions: 1-D loss landscape along random/top-K PCA directions, 1-D interpolation between checkpoints, spectral structure of the weights and learned updates, and activation similarity to full-rank training. We show that low-rank methods are not equivalent to full-rank training, nor to one another, even when validation perplexity is close. Full-rank training settles into sharper basin than low-rank methods along random directions, while the reverse holds for the top-1 PCA direction. Each method converges to a geometrically distinct basin. Low-rank activations increasingly diverge from full-rank in later layers as training progresses, with GaLore tracking full-rank most closely. Further, validation perplexity does not translate to downstream performance at every scale. Adding geometric and spectral metrics to it improves the prediction.
1
Introduction
Pre-training large language models (LLMs) is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged as a way to reduce this cost while still approximating full-rank training. This field has evolved along three main directions. Optimizer-based methods such as GaLore [28] leave the model intact and constrain only the gradient update to a low-rank subspace. Architecture-based methods such as CoLA [18] replace dense linear layers with low-rank decompositions. Adapter-based methods such as ReLoRA [15], add low-rank adapters to linear layers that are periodically merged into the weights and re-initialized. Despite this growing body of work, evaluation practices in low-rank pretraining remain limited. Methods are benchmarked against one another using validation perplexity from a single training ∗ Corresponding author. Code: https://github.com/NamrataRShivagunde/low-rank-geometry
Preprint.
run with a single random seed [28, 15, 18, 6, 19, 29, 21, 23, 13, 20]. While this is understandable given the significant compute costs of large-scale pretraining, it raises an important question: is validation perplexity alone a sufficient criterion for evaluating a low-rank pretraining method’s quality? Earlier work [16] shows that two models with the same pre-training loss can differ substantially on downstream tasks, and that loss-landscape flatness predicts this gap. Beyond validation perplexity, most prior work evaluates low-rank pre-training methods by using them as fine-tuning recipes. For example, GaLore [28], SLTrain [6], LOST [13], and VeLoRA [20] are evaluated by fine-tuning RoBERTa on GLUE, while CoLA [18] is evaluated by fine-tuning LLaMA-7B on downstream tasks. This measures how well a low-rank method fine-tunes a pre-trained model, but does not tell us how a model pre-trained with the low-rank method generalizes downstream. A few studies do fine-tune pre-trained models directly. For instance, Lialin et al. [15] compare ReLoRA against full-rank on GLUE, Liu et al. [18] compare a pre-trained CoLA against BERT, and Zhou et al. [29] compare full-rank, GaLore, and SwitchLoRA. These studies are narrow in scope, covering one or two methods at a single scale, and none span all three method families (optimizers, model architecture, adapters). Only a handful of recent studies look beyond validation perplexity, and these are narrow in scope. FlatLoRA [14] compares LoRA against full-rank training via loss landscape analysis, but only in the fine-tuning setting. Liu et al. [17] study GaLore, LoRA, and full-rank training through the Polyak-Łojasiewicz (PL) condition, but their analysis is restricted to multi-layer perceptrons and does not transfer cleanly to transformers. Both studies cover only a subset of low-rank methods and rely on a single analytical lens. We address this research gap with four contributions: (1) A diagnostic framework for low-rank pretraining, including a novel cross-method barrier metric (IMBH) that quantifies geometric separation between training algorithms — not just between seeds or epochs as in prior linear-mode-connectivity work. (2) Cross-family empirical findings that no prior work establishes: low-rank methods occupy geometrically separated basins (IMBH grows monotonically), develop distinct spectral signatures (GaLore matches full-rank, CoLA don’t), and produce divergent internal representations (CoLA orthogonal, Fira recovers only at last layer). (3) Geometry-augmented predictor of downstream performance. The perplexity ranking of low-rank methods does not translate to zero-shot downstream performance, but augmenting perplexity with geometric and spectral features from our framework improves the prediction. (4) An open-source unified codebase. We release a single pipeline integrating pre-training, metric computation, and downstream evaluation, designed for easy extension to new methods and metrics, enabling the community to study low-rank pre-training in more depth and design better methods.
2
Related Work
Low-rank pre-training methods. Memory-efficient pre-training of LLMs has motivated a fastgrowing family of low-rank methods, which broadly fall into three design categories. The first category wraps the optimizer, constraining the gradient update to a low-rank subspace while leaving the parameters at full rank. GaLore [28] projects gradients into a low-rank subspace, applies an Adam update there, and projects back to the full-rank parameter space. Fira [3] introduces a normbased scaling that uses GaLore’s low-rank Adam states to scale the full-rank gradient, recovering full-rank update directions at the same memory cost. A growing line of work extends this design idea along several axes, including Q-GaLore [27], GaLore 2 [24], GRASS [21], GaLore-mini [8], COAP [26], and FLORA [7]. The second category modifies the model itself, replacing dense weight matrices with low-rank decompositions. CoLA [18] restructures attention and MLP linear layers as bottleneck low-rank matrices with a non-linearity in between. SLTrain [6] parameterizes weights as a sparse-plus-low-rank sum, jointly trained from scratch. ReDoRA and RePiSSA [23] extend the same idea with alternative reparameterization choices. The third category adds low-rank adapters that are periodically merged back and re-initialized. ReLoRA [15], the canonical example, sparked the initial interest in low-rank pre-training; SwitchLoRA [29] extends it by periodically switching parameters of the LoRA matrices, and LoQT [19] adapts the recipe to quantized models. Toward understanding the training dynamics of low-rank methods. A handful of studies probe low-rank training beyond final validation loss, but each examines a single signal in isolation. FlatLoRA [14] and [17] take a loss-landscape view. SLTrain [6] tracks singular-value spectra to argue that the sparse-plus-low-rank parameterization preserves rank that pure low-rank loses, ReLoRA [15] inspects the rank of cumulative updates to argue that periodic merges grow effective rank beyond a 2
single low-rank stage, and WeLore [9] characterizes the non-uniform low-rank structure of pre-trained LLM weights across layers and exploits it for fine-tuning. Geometry- and spectrum-based analyses of trained models. Low-rank training methods approximate full-rank training through different design choices, ranging from optimizer design to model architecture change. [12] pioneers the visualization of the loss landscape itself and shows that design choices — such as architecture, optimizer, and batch size — shape the underlying loss landscape, and that some landscapes are far easier for an optimizer to navigate than others. Their study, however, was conducted on image classifiers, not LLMs. Subsequent work has used loss landscapes, weight spectra, and activation similarity to study different aspects of LLM training. For example, [25] uses 1-D loss profiles to argue that LLM training under warmup-stable-decay schedules traces a “river valley” geometry. [2] adapt random-direction perturbations to LLMs to characterize the basin-like structure of their loss landscape. [16] uses loss-landscape to show that two language models reaching the same pre-training loss can differ on downstream tasks, with flatness predicting the gap where perplexity does not. However, [1] and [10] show that sharpness alone correlates poorly with generalization, motivating us to include additional signals such as rank and activation metrics, which have been used to study other aspects of model behavior like model mergeability [30].
3
Evaluation framework
We evaluate Full-rank, GaLore, Fira, CoLA, SLTrain, and ReLoRA, at 60M, 130M, and 350M parameters using 16 metrics across four dimensions. Section 3.1 introduces the individual metrics. 3.1 3.1.1
Individual metrics 1-D loss landscape on random directions.
To understand the local geometry of the loss landscape around each pre-training checkpoint, we perturb the model parameters along a single randomly chosen direction and record how the validation loss varies as we move along this direction. However, since a single direction only captures a d-dimensional slice and is not necessarily representative of the surrounding geometry, we repeat the process along 100 independently sampled random directions and record the mean and variance of the validation loss across these directions. Following [2] , we adapt the formula to our pre-training setting where the loss is the C4 [22] validation loss (we pre-train our models on C4). For a single random direction δ: L(α) = Lval (θ + α δ) , δ ∼ N (0, I), where θ are the checkpoint parameters, α ∈ [αmin , αmax ] is the perturbation magnitude swept on a uniform grid, and Lval is the validation loss. δ has the same dimensions as the parameter matrix it perturbs. We average L(α) over 100 i.i.d. independently sampled directions δ1 , . . . , δD ∼ N (0, I) where D = 100: D D 2 1 X 1 X L̄(α, δ) = Lval (θ + α δi ) , σ 2 (α, δ) = Lval (θ + α δi ) − L̄(α) . (1) D i=1 D i=1 where L̄(α, δ) denotes the mean of the validation loss across the D random directions at perturbation magnitude α, and σ 2 (α, δ) denotes its variance. For convenience, we drop δ in the notation. Sharpness The 1-D loss landscape from the previous section yields a full curve L̄(α, δ) per checkpoint, which is qualitative in nature. To compare methods quantitatively across checkpoints, model sizes, and training steps, we reduce each curve to a single scalar capturing how flat or sharp the local basin is around the checkpoint. Intuitively, the rate at which L̄ grows as we move away from the checkpoint (α = 0) serves as a discrete proxy for local curvature: sharp curvature produces steep loss changes in the neighborhood, while flat curvature keeps losses close to the reference value at α = 0. Concretely, for each direction δi , we evaluate the loss at symmetric perturbations ±αj on a grid of N offsets (where N is the number of positive offsets in the perturbation grid) and measure the elevation ∆(δi , αj ) = L(δi , αj ) − L(δi , 0) relative to the checkpoint. Sharpness (S) is then the average of these elevations across all D directions and both signs of α: D−1 N −1 i 1 X Xh S= ∆(δi , +αj ) + ∆(δi , −αj ) , ∆(δ, α) = L(δ, α) − L(δ, 0). (2) 2N D i=0 j=0 A larger S indicates a sharper basin; a smaller S indicates a flatter one. 3
Direction variance The variance band in the 1-D loss landscape plots is a qualitative metric, and direction variance is its quantitative counterpart. We compute it by averaging the across-direction variance σ 2 (α, δ) from (1) over the offsets N , similar to (2)). See Appendix A.1 for equations. 3.1.2
1-D loss landscape on top-K PCA directions.
Random Gaussian directions probe most-case geometry around a checkpoint. We therefore complement them with a principled alternative: applying SVD to the weight matrix and perturbing along its top-k singular directions. Specifically, for every two-dimensional parameter tensor W ∈ Rm×n in the checkpoint θ, we compute the SVD, W = U ΣV ⊤ and form the top-k perturbation direction as δ (k) = σk (uk ⊗ vk ), where uk and vk are the k-th left and right singular vectors andσk is the corresponding singular value. The resulting 1-D loss slice is Lpca (α; k) = Lval θ + α δ (k) . 3.1.3
1-D interpolation
To assess the geometric connectivity of parameter space across training, we measure the loss along a straight-line path between two checkpoints. Given two models θA and θB , the interpolated model at coefficient β ∈ [0, 1] is θ(β) = (1 − β)θA + βθB , and we evaluate the validation loss L(β) = Lval (θ(β)) at evenly-spaced values of β. The resulting curve characterizes the loss surface geometry along the segment connecting the two models. A flat, low-lying curve indicates that both checkpoints reside in the same wide basin, whereas a curve that peaks sharply in the interior suggests that there is a barrier between the two models and the models exist in twodifferent basins. Following 1 [4], we quantify this as Barrier Height (BH) = maxβ∈(0,1) L(β) − 2 L(0) + L(1) . BH is the excess loss at the worst intermediate point above the average of the two endpoint losses. A barrier near zero means the two models are connected through a smooth valley, and a large positive barrier means the optimizer must cross a region of high loss to travel between them. We apply this metric in two settings. First, we interpolate between consecutive checkpoints of the same method, using pairs e.g., (θ1000 , θ2000 ), (θ2000 , θ3000 ) etc. throughout training. We call this Consecutive Checkpoints Barrier Height (CCBH). Second, we interpolate between checkpoints of different methods at the same training step. This cross-method interpolation, whose barrier we call the Inter-Method Barrier Height (IMBH), reveals whether different training methods converge to the same basin in weight space or to geometrically distinct solutions. To our knowledge, this is the first application of barrierheight analysis across different pre-training methods rather than across seeds, epochs, or fine-tuning runs of the same method. As methods such as ReLoRA and SLTrain store weights in a factored or sparse format, we first materialize all checkpoints into a common dense full-rank weight space before interpolating, ensuring that the linear path is geometrically meaningful across methods. We exclude CoLA in the cross-method interpolation comparison as CoLA has an activation function between low-rank matrices and cannot be refactored as a dense full-rank model. 3.1.4
Rank and spectral metrics.
Since low-rank methods are designed to approximate full-rank training, we track the spectral structure of each method’s weights across the training horizon. Following [30], we compute a family of spectral quantities on the attention and MLP weight matrices. Effective rank measures the entropy of the normalized singular value spectrum, capturing how broadly the matrix’s energy is distributed across directions. Stable rank, defined via the ratio of Frobenius to operator norms, provides a noise-robust dimensionality estimate that downweights negligible singular values. Spectral gap measures the relative separation between the top two singular values, indicating whether a single direction clearly dominates. Threshold rank counts the singular values exceeding a fixed cutoff. Along with raw weights, we also apply these metrics to the parameter updates between consecutive checkpoints. We include equations for each of these metrics in Appendix A.2. 3.1.5
Activation-based metrics.
To measure how closely a low-rank checkpoint’s internal representations match those of the full-rank baseline, we compare their hidden states on identical inputs. We compute Activation L2 distance, which measures the mean per-position Euclidean distance between the two activation matrices. Activation cosine similarity reports the mean directional alignment of activation vectors at each position, isolating angular mismatch independent of magnitude. Linear CKA [11] compares the two 4
activation matrices at the representational level and is invariant to orthogonal rotations and isotropic scaling. Equations for each is defined in Appendix A.3.
4
Experimental Setup
Training: Following [28], we pre-train LLaMA-style decoders at 60M, 130M, and 350M parameters on C4 with Chinchilla-optimal token budgets. We compare full-rank, GaLore, Fira, CoLA, SLTrain, and ReLoRA (without warm start), using rank 128 at 60M and 256 at larger scales, with other hyperparameters from the original papers. Metrics: We compute all metrics in Section 3.1 on a fixed 1000-example C4 validation subset with a shared seed. Downstream predictor: We fit a linear regression predicting the downstream mean from validation loss and the eight sign-consistent metrics (see Table 6), evaluated under leave-one-size-out (LOSO) and leave-one-method-out (LOMO) crossvalidation on 90 checkpoints (5 per method × 6 methods × 3 sizes). Downstream evaluation: We evaluate final checkpoints zero-shot with lm-evaluation-harness [5] on 11 tasks across five categories: commonsense (HellaSwag, PIQA, COPA, SWAG), world knowledge (OpenBookQA, ARC-Easy, QA4MRE-2013), reading comprehension (ReCoRD), grammar (BLiMP), and logical reasoning (LogiQA, LogiQA2). All tasks score above random performance and show gains with model scale. We use the t5-base tokenizer, automatic batch sizing, and 1000 bootstrap iterations; task scores use standard lm-eval metrics and are reported as mean ± bootstrap standard error in Tables 3–5 in Appendix B. We use an aggregate plot for each task category across all model sizes in the Figure 5. All experiments were run on NVIDIA RTX 6000 Ada Generation GPUs (48 GB). Each pre-training run and metric computation in our study fits on a single GPU.
5
Results
5.1
Individual metrics results
1-D loss landscape on random directions We report the 1D loss landscape along random directions in Figure 1a: Row 1 shows the mean centered-loss curve (L(α) − L(0)) with variance bands at five 350M checkpoints; Rows 2 and 3 show basin sharpness and direction variance across training for all three model sizes. 60M and 130M plots are shown in Appendix C.1 (Figures 6–11). At 60M, every method’s basin sharpness first decreases and then increases over training — it starts sharp at initialization, flattens during the early phase, and re-sharpens as training continues. The sharp → flat → sharp trajectory holds at 130M and 350M (except for CoLA). At 350M, the basin starts sharp at initialization, flattens through the early-to-mid phase (∼ 12k−18k steps for Fira, SLTrain, GaLore, and Full-rank), and then re-sharpens as training continues. CoLA’s sharpness rises monotonically from step 6k to 60k, never exhibiting the intermediate flattening phase. Figure 1a row-3 shows that at 60M full-rank is more anisotropic than every low-rank method we test, except for ReLoRA. The average direction variance is of order ∼ 10−5 for full-rank versus 10−7 −10−6 for low-rank pre-training methods. In other words, if we perturb a full-rank model along a random direction, the magnitude of the resulting loss change varies strongly with which direction we picked, whereas for the low-rank methods, almost any direction yields roughly the same response (loss change). This separation weakens at 130M, where the low-rank curves rise toward the full-rank baseline over training. At 350M, the gap closes more i.e., every method reaches similar direction variance by the end of training (y-axis range is different for each model scale), and Fira actually exceeds full-rank at the final checkpoint. The clear “full-rank-is-anisotropic, low-rank-is-uniform” separation at 60M is therefore a small-scale artifact, as at larger scales, low-rank methods develop anisotropies comparable to or greater than full-rank training. 1-D loss landscape on top-1 PCA direction. Figure 1b shows basin sharpness (top row) and top singular value σ1 (bottom row) along the top-1 PCA direction across training. Full-rank’s σ1 is moderate (∼ 4−7 across scales) and its sharpness along top-1 pca direction is low to moderate, sitting below every low-rank method except Fira 60M, suggesting full-rank basin is relatively flatter than low-rank methods. We see similar pattern with SLTrain. Fira is the flattest along its top-1 pca direction. Despite having one of the largest σ1 at every scale (peaks of ∼ 30 at 130M and ∼ 40 at 350M), Fira’s sharpness is the lowest which means a unit-α step nudges Fira further in parameter space than any other method, yet the loss barely responds, indicating its loss landscape is flattest. 5
step 6000
0.03
350M
step 12000
0.03
350M
step 24000
0.03
350M
step 48000
0.03
(0)
350M
0.02
0.02
0.02
0.02
0.02
( )
0.03 0.01
0.01
0.01
0.01
0.01
0.00
0.00
0.00
0.00
0.002 0.000 0.002
0.002 0.000 0.002
0.002 0.000 0.002
sharpness
60M
direction variance
0.0050 0.0025
0.002 6000
training step
8000
10000
5000
1e 6
10000
15000
training step
20000
10000 20000 30000 40000 50000 60000
training step
1e 6
2.0 1.5 1.0 0.5 0.0
4
7.5 5.0
2
2.5 0.0
2000
4000
6000
training step
8000
10000
cola
0.002 0.000 0.002
0.0075
0.004 4000
step 60000
350M
0.0100
0.006
2000
0.00
0.002 0.000 0.002
130M
0.0100 0.0075 0.0050 0.0025
350M
0
5000
fira
10000
15000
training step
galore
20000
1e 5
10000 20000 30000 40000 50000 60000
training step
fullrank
sltrain
0.007 0.006 0.005 0.004 0.003 0.002
top singular value 1
sharpness
(a)
60M
130M
0.008 0.006
0.010
0.004
0.005
0.002
10 8 6 4 2k
4k
6k
training step
cola
8k
fira
10k
40
40
30
30
20
20
10
10
0
350M
0.015
2.5k 5k 7.5k 10k 12.5k 15k 17.5k 20k
training step
galore
fullrank
0
10k
relora
20k
30k
40k
training step
50k
60k
sltrain
(b)
Figure 1: 1-D loss landscape (a) random direction (b) top-1 PCA direction.
GaLore, CoLA and ReLoRA converge to sharper basin than full-rank. GaLore has relatively smaller σ1 at every scale (∼ 3 throughout training) yet still produces moderate-to-high sharpness (∼ 0.005 − 0.007) — the loss elevates substantially, indicating a very steep loss landscape along its leading direction. Similar pattern is seen in CoLA and ReLoRA. Combining random and top-1 PCA results, three regimes emerge: Fira and SLTrain converge into flat-to-moderately-flat basins; CoLA and ReLoRA reach the sharpest basins; and GaLore straddles the two — moderately flat along random directions but sharp along its top-1 PCA direction. Full-rank shows the opposite pattern to GaLore: sharp along random directions but flat along its top-1 PCA directions. 1-D interpolation Figure 2a shows the CCBH heatmap for every pre-training method. We observe two patterns. First, the CCBH decay across all methods as training progresses (rows lighten left to right), and second, CoLA and ReLoRA have higher CCBH than other methods. Figure 2b shows the IMBH, and we find two consistent patterns. First, every method pair sustains large barriers throughout training (BH ≈ 2–8 across all sizes and steps), suggesting that all methods occupy geometrically separated basins. Second, unlike the CCBH of Figure 2a, which decay with training, IMBH increase monotonically. This means that methods drift further apart in weight space as each converges into its own basin. At 60M, full-rank vs. low-rank barriers (IMBH) are consistently higher than any low-rank vs. low-rank pair, indicating that the full-rank solution occupies a basin that is more distant from any low-rank method solution. Within the low-rank group at 60M, Fira, GaLore, 6
60M
350M
0.162 0.061 0.037 0.028 0.021 0.016 0.009 0.004 0.001
0.123 0.049 0.032 0.027 0.021 0.014 0.007 0.003 0.001
galore 0.148 0.054 0.036 0.026 0.023 0.017 0.012 0.005 0.002
0.129 0.054 0.032 0.025 0.021 0.017 0.009 0.005 0.001
0.110 0.040 0.030 0.022 0.020 0.013 0.007 0.003 0.001
0.141 0.438 0.208 0.054 0.512 0.019 0.023 0.005 0.001
0.392 0.095 0.044 0.027 0.015 0.008 0.004 0.002 0.001
10 2 10 3
8k
7k
6k
5k
4k
3k
2k
1k
10 1
9 9k k 10 k 2k 4k 4k 6k 6k 8 8k k 1 10 0k k 1 12 2k k 1 14 4k k 1 16 6k k 1 18 8k k 20 k 6k 12 12 k k 1 18 8k k 2 24 4k k 3 30 0k k 3 36 6k k 4 42 2k k 4 48 8k k 5 54 4k k 60 k
relora 0.701 0.348 0.054 0.030 0.018 0.011 0.007 0.004 0.002 8k
0.123 0.051 0.032 0.025 0.021 0.015 0.008 0.004 0.001
7k
0.145 0.060 0.035 0.029 0.024 0.017 0.010 0.005 0.001
6k
sltrain 0.197 0.062 0.039 0.029 0.024 0.017 0.012 0.005 0.002 5k
3.995 0.050 0.030 0.028 0.023 0.020 0.012 0.006 0.001
4k
0.145 0.051 0.036 0.024 0.020 0.016 0.012 0.006 0.002
0.129 0.051 0.035 0.027 0.024 0.018 0.012 0.006 0.002
3k
0.148 0.059 0.036 0.028 0.021 0.019 0.013 0.009 0.003
2k
fira 0.175 0.066 0.042 0.030 0.026 0.018 0.015 0.008 0.004 cola 0.110 0.079 0.037 0.039 0.026 0.025 0.018 0.011 0.003
100
barrier height (log scale)
130M
full-rank 0.231 0.071 0.042 0.030 0.020 0.016 0.010 0.003 0.002
step pair
step pair
step pair
(a) 130M
350M
Full-rank GaLore 4.38
7.70
6.60
6.03
5.93
6.39
6.14
6.18
6.25
6.23
1.91
2.90
3.55
3.86
4.21
4.58
4.66
4.71
4.75
4.75
3.13
4.54
5.36
6.01
7.00
7.43
7.42
7.59
7.52
7.51
Full-rankFull-rank FIRA 4.25
4.78
5.02
5.28
5.41
5.53
5.63
5.74
5.79
5.77
1.57
2.48
2.97
3.24
3.39
3.53
3.56
3.61
3.62
3.62
1.86
2.87
3.33
3.76
4.26
4.46
4.76
4.98
5.14
5.23
Full-rank SLTrain 4.64
5.91
5.93
5.79
5.84
5.88
6.05
6.13
6.13
6.12
4.33
4.81
5.34
5.58
5.98
5.99
6.11
6.22
6.29
6.28
4.95
6.12
6.32
6.76
7.39
7.74
7.79
7.86
7.94
7.99
Full-rank ReLoRA 4.34
4.86
5.28
5.51
5.76
5.83
5.93
5.94
5.97
5.92
2.57
3.48
4.13
4.24
4.47
4.61
4.66
4.68
4.72
4.72
2.92
3.75
4.19
4.46
4.67
4.78
4.87
4.95
4.96
4.99
1.77
2.21
2.64
3.02
3.18
3.34
3.37
3.41
3.43
3.20
5.25
5.77
5.93
6.16
6.09
6.13
6.22
6.12
6.10
2.46
3.35
3.64
4.18
4.77
5.12
5.54
5.90
6.19
6.39
reset3.08 FIRASame SLTrain family
3.39
3.67
3.96
4.11
4.31
4.37
4.46
4.45
4.44
4.22
5.43
5.95
6.43
6.70
6.85
6.95
7.09
7.12
7.14
5.50
6.07
6.44
6.72
7.27
7.46
7.69
7.84
7.99
8.11
vs low-rank
SameGaLore optimizer FIRA family 0.99
5.69
5.79
8.14
8.22
8.31
GaLore ReLoRA 1.21
1.67
2.05
2.23
2.38
2.48
2.55
2.58
2.60
2.61
1.64
2.44
2.75
3.13
3.36
3.53
3.67
3.77
3.83
3.87
2.02
2.86
3.26
3.55
3.81
4.00
4.20
4.37
4.47
4.53
ReLoRA SLTrain 3.55
3.56
3.91
4.11
4.44
4.59
4.57
4.70
4.72
4.69
4.40
4.61
4.67
5.27
5.50
5.59
5.66
5.82
5.93
5.95
5.10
5.66
6.14
6.24
6.48
6.55
6.81
6.84
6.88
6.88
3 × 100 2 × 100
100
00
60
00
00
54
training step
48
00
42
00
36
00
30
00
24
00
00
18
12
60
00
20
00
00
18
training step
16
00
14
00
12
00
10
80
60
40
20
00
90
10
training step
80
70
60
50
40
00
30
20
10
00
low-rank
4 × 100
0
5.54
8.30
0
5.36
8.01
0
5.18
7.47
0
4.94
6.60
0
4.69
6.19
0
4.15
5.67
0
3.53
4.63
0
2.30
7.12
0
4.41
7.06
0
4.37
7.09
00
4.30
6.96
0
4.21
6.30
0
4.07
5.95
0
3.90
5.64
0
3.69
4.81
0
3.32
4.40
00
2.93
3.46
00
1.94
4.05
00
3.07
4.04
0
3.07
4.03
00
3.03
3.98
00
2.97
3.88
00
2.88
3.63
00
2.78
3.55
00
2.61
3.26
00
2.35
2.84
00
1.95
00
FIRA ReLoRA 1.56 GaLoreCross-family SLTrain 2.29
6 × 100
barrier height
60M
(b)
Figure 2: 1-D interpolation (a) CCBH (b) IMBH and ReLoRA exhibit relatively low mutual barriers. SLTrain, by contrast, shows substantially higher barriers against all other low-rank methods, placing it in a distinctly separate valley. At 130M and 350M, the full-rank versus low-rank barriers decrease, and low-rank vs. low-rank increases. Fira and ReLoRA retain the smallest mutual barriers, GaLore occupies an intermediate position, and SLTrain remains the most geometrically isolated method at every scale. Among the low-rank methods, Fira consistently shows the lowest barrier to the full-rank solution across all model sizes (Figure 2b), suggesting it converges to a basin geometrically closest to the full-rank basin. ReLoRA ranks second — a surprise, given its highest validation loss and sharpest basin. Rank and spectral metrics. Figure 3 shows four rank and spectral metrics at the 350M. Row 1 tracks metrics on the trained weight W , Row 2 on consecutive-checkpoint updates (e.g. ∆W = W2k − W1k ), and Row 3 shows the singular-value distribution of ∆W (difference between last and first checkpoint) per projection type (attention WQ , WV and MLP Wup ), with the rightmost panel counting singular values above 0.1 per projection. Trained weights W (Row 1). Full-rank dominates on every rank metric and no low-rank method fully matches it. GaLore tracks full-rank closely on effective and stable rank but exhibits a substantially smaller spectral gap; CoLA shows the opposite profile. ReLoRA maintains higher effective and stable rank than both CoLA and SLTrain despite its low-rank constraint. Weight update ∆W (Row 2). Fira and GaLore most closely follow full-rank on effective and stable rank, while full-rank retains the largest spectral gap that no low-rank method reaches. GaLore further tracks full-rank’s count of singular values above 0.1 more faithfully than the other methods. Per-projection spectra (Row 3). Within each projection type, GaLore reproduces full-rank’s singular-value distribution more closely than competing methods. We see similar pattern at 60M and 130M scale (see plots in Appendix section D.) Activation metrics. Figure 4 summarizes activation alignment with full-rank training. Row 1 reports the layer-averaged stacked deviation per method and scale; Row 2 restricts the same quantity to the final layer; Row 3 shows per-layer × per-step heatmaps for the two extreme low-rank methods at 350M (Fira and CoLA). Full per-method, per-scale heatmaps are in Appendix A.3. Layer-averaged drift (Row 1). ReLoRA tracks full-rank representations most closely across all three scales, with GaLore and SLTrain following, while CoLA and Fira exhibit the largest total deviation. The ranking is stable across scales. Last-layer drift (Row 2). Restricting to the final layer changes the picture for Fira specifically: its last-layer deviation drops sharply (2.49 at 350M, second lowest after ReLoRA) despite having the largest layer-averaged drift. The last decoder block plus final RMSNorm reduces 7
20k
40k
60k
800 600 400 200 20k
Frequency ( W)
20
40k
training step
20k
40k
training step
stable rank of W
effective rank of W
training step
200 150 100 50
60k
60k
20k
40k
training step
0.125 0.100 0.075
60k
20k 3.0
1.5
2.0
2.5
0.0 0.0
1.0
0.5
(WQ)
1.5
2.0
0.0 0.0
full-rank
0.5 0.5
1.0
(WV)
galore
1.5
2.0
fira
0.0 0.0
600 400
60k
20k
40k
60k
40k
60k
training step
60k
800 600 400 200 0
20k
training step
1000
1.5
1.0
1.0
40k
training step
2.0
1.5
0.5
40k
0.30 0.25 0.20 0.15 0.10 0.05
2.5
0.5
20k
training step
2.0
1.0
# > 0.1 of W
40
800
0.150
# > 0.1 of W
200
60
1000
0.175
# > 0.1 of W
400
80
0.200
spectral gap of W
600
100
spectral gap of W
stable rank of W
effective rank of W
800
0.5
cola
1.0
(Wup)
1.5
2.0
sltrain
800 600 400 200 0
WQ WK WV WO WgateWupWdown
relora
Figure 3: Rank and spectral metrics at 350M. the deviation, and Fira benefits from this more than any other method. Per-layer dynamics (Row 3). For both Fira and CoLA, L2 distance grows in the later layers as training progresses, with the final layer reducing drift relative to full-rank. CoLA is directionally off at every layer (cos ≈ 0), while Fira preserves angular alignment. Linear CKA deviates most in the middle layers and degrades further later in training. Patterns hold across other methods and scales (Figures 17–19, Appendix E). 5.2
Downstream evaluation
Table 1 reports final C4 validation perplexity and Figure 5 reports zero-shot downstream performance. Validation perplexity does not predict downstream ranking. Fira achieves the lowest perplexity at every scale, even surpassing full-rank, yet CoLA leads on multiple downstream categories at 130M and 350M despite not ranking in the top two on perplexity. Geometry and spectral metrics recover what perplexity misses. Across 90 checkpoints, perplexity alone is a strong but incomplete predictor (LOSO Pearson 0.873, LOMO 0.864); augmenting it with the eight sign-consistent geometry and spectral features in Table 6 lifts LOSO to 0.913 and LOMO to 0.895 (Table 2).
6
Conclusion
Low-rank pre-training methods are not interchangeable approximations of full-rank training. Fira and SLTrain reach flat basins while CoLA and ReLoRA reach the sharpest; each method converges to a geometrically distinct basin, with inter-method barriers growing rather than shrinking over training; GaLore most faithfully reproduces full-rank’s spectral structure on weight updates, while ReLoRA tracks full-rank activations most closely. Validation perplexity shows an incomplete picture of downstream performance — but augmenting it with eight geometric and spectral features from our framework recovers the missing signal, providing a better predictor of downstream performance.
7
Limitations
We compute our metrics on < 1B LLaMA-style models, generalization to billion-scale models, other architectures, datasets, and rank choices is not explored. Due to compute limit, we report each metric from a single training run per method per scale and therefore cannot quantify run-to-run variability. We adopt two perturbation-direction choices for the 1-D loss landscape, but the most appropriate normalization for cross-method LLMs comparison remains open. 8
3.0 2.5 2.0 1.5 1.0 0.5 0.0
CoLA @ 350M layer index
FIRA @ 350M layer index
last-layer stacked deviation
layer-avg stacked deviation
60M 3.0 2.5 2.0 1.5 1.0 0.5 0.0
130M 3.0 2.5 2.0 1.5 1.0 0.5 0.0
2.90 2.54
2.50
cola
2.52
2.49
fira galore relora sltrain
60M
2.97
2.72
2.55
2.42
2.37
cola
3.0 2.5 2.0 1.5 1.0 0.5 0.0
fira galore relora sltrain
L2 distance
24
L2
350M 3.0 2.5 2.0 1.5 1.0 0.5 0.0
2.86 2.17
2.10 1.74
cola
1.71
fira galore relora sltrain
130M
2.76
2.58
2.53 2.15
2.01
cola
3.0 2.5 2.0 1.5 1.0 0.5 0.0
fira galore relora sltrain
1 cos
1 CKA
cosine similarity
24
1500
30k
42k
54k
0 6k 24 1500
0.0
18k
30k
42k
54k
18k
30k
42k
training step
350M
3.00 2.52
2.49
cola
2.41
2.39
fira galore relora sltrain
linear CKA 0.6 0.4 0.2
0 6k 24
18k
30k
42k
54k 0.6 0.4
0.1
0
54k
fira galore relora sltrain
0.2
500
6k
cola
0.1
1000
0
2.18
1.95
0.2
500
18k
2.30
24
1000
0 6k 24
2.81
2.80
0.0
6k
18k
30k
42k
54k
training step
0
0.2
6k
18k
30k
42k
54k
training step
Figure 4: Activation deviation with full-rank baseline.
Commonsense
130M 350M
Full-rank G A L ORE F IRA C O LA SLT RAIN R E L O RA
34.12 34.91 31.13 33.47 35.71 42.76
25.09 25.62 23.18 25.79 26.48 35.88
19.43 19.87 17.40 19.58 20.64 29.99
Table 1: Validation perplexity at last checkpoint, across model sizes. Lower is better. Per column, the best method is bold and second-best is underlined. Predictor val loss only geometry only (8 feats) val loss + geometry (9 feats)
World Knowledge
0.50
Accuracy / F1
60M
0.48 0.46 60M
130M
0.30
0.29
0.28
0.28
0.26
0.27
0.24
350M
Grammar (BLiMP)
Reading Comp.
0.30
0.22
0.26
0.44
Accuracy / F1
Method
0.280
60M
130M
350M
60M 0.44
0.275
0.76
0.270
0.74
0.42
0.265
0.41
0.72
0.260 60M
130M full-rank
350M galore
350M
Average
0.78
0.70
130M
Logical Reasoning 0.43
0.40 60M
130M fira
350M cola
0.39
60M
130M
sltrain
350M
relora
Figure 5: Zero-shot downstream performance. LOSO Pearson
LOMO Pearson
R2 (in-sample)
0.873 0.498 0.913
0.864 0.431 0.895
0.841 0.558 0.907
Table 2: Comparison of downstream predictors. Geometry features are the top-8 paper metrics whose per-size Spearman vs downstream is sign-consistent (Spearman correlation with downstream has the same sign at all three scales). Bold = best per column.
9
References [1] Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.07011, 2023. [2] Huanran Chen, Yinpeng Dong, Zeming Wei, Yao Huang, Yichi Zhang, Hang Su, and Jun Zhu. Unveiling the basin-like loss landscape in large language models. arXiv preprint arXiv:2505.17646, 2025. [3] Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of llms under low-rank constraint? ArXiv, abs/2410.01623, 2024. [4] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International conference on machine learning, pages 3259–3269. PMLR, 2020. [5] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. [6] Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. arXiv preprint arXiv:2406.02214, 2024. [7] Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293, 2024. [8] Weihao Huang, Zhenyu Zhang, Yushun Zhang, Zhi-Quan Luo, Ruoyu Sun, and Zhangyang Wang. Galore-mini: Low rank gradient learning with fewer learning rates. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024. [9] AJAY KUMAR JAISWAL, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. [10] Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In Proceedings on, pages 51–65. PMLR, 2023. [11] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), 2019. [12] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018. [13] Jiaxi Li, Lu Yin, Li Shen, Jinjin Xu, Liwu Xu, Tianjin Huang, Wenwu Wang, Shiwei Liu, and Xilu Wang. Lost: Low-rank and sparse pre-training for large language models. arXiv preprint arXiv:2508.02668, 2025. [14] Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang. Flat-lora: Low-rank adaptation over a flat loss landscape. arXiv preprint arXiv:2409.14396, 2024. [15] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReLoRA: High-Rank Training Through Low-Rank Updates. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [16] Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, 2022. 10
[17] Xu-Hui Liu, Yali Du, Jun Wang, and Yang Yu. On the optimization landscape of low rank adaptation methods for large language models. In The Thirteenth International Conference on Learning Representations, 2025. [18] Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, and Zheng Zhang. Cola: Compute-efficient pre-training of llms via low-rank activation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4627–4645, 2025. [19] Sebastian Loeschcke, Mads Toftrup, Michael J Kastoryano, Serge Belongie, and Vésteinn Snæbjarnarson. LoQT: Low Rank Adapters for Quantized Training. arXiv preprint arXiv:2405.16528, 2024. [20] Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections. Advances in Neural Information Processing Systems, 37:42292–42310, 2024. [21] Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, and Virginia Smith. Grass: Compute efficient low-memory llm training with structured sparse gradients. arXiv preprint arXiv:2406.17660, 2024. [22] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. [23] Namrata Shivagunde, Mayank Kulkarni, Giannis Karamanolakis, Jack G. M. FitzGerald, Yannick Versley, Saleh Soltan, Volkan Cevher, Jianhua Lu, and Anna Rumshisky. Approximations may be all you need: Towards pre-training llms with low-rank decomposition and optimizers. 2024. [24] DiJia Su, Andrew Gu, Jane Xu, Yuan Tian, and Jiawei Zhao. Galore 2: Large-scale llm pre-training by gradient low-rank projection. 2025. [25] Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192, 2024. [26] Jinqi Xiao, Shen Sang, Tiancheng Zhi, Jing Liu, Qing Yan, Linjie Luo, and Bo Yuan. Coap: Memory-efficient training with correlation-aware gradient projection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30116–30126, 2025. [27] Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients. arXiv preprint arXiv:2407.08296, 2024. [28] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. [29] Kaiye Zhou, Shucheng Wang, and Jun Xu. Switchlora: Switched low-rank adaptation can learn full-rank information. arXiv preprint arXiv:2406.06564, 2024. [30] Lorenzo Zhou, Bo Zhao, Runpeng Yu, and Emanuele Rodolà. Demystifying mergeability: Interpretable properties to predict model merging success. In arXiv preprint arXiv:2601.22285, 2026.
A
More details on metrics
We provide more details on the metrics in this section.
11
A.1
Loss landscape related metrics
i PN h 2 2 1 Direction variance equation is given below DV = 2N j=1 σ (+αj ) + σ (−αj ) , A higher DV indicates that the loss surface is highly anisotropic around the checkpoint, i.e., the basin is much steeper along some directions than others, whereas a lower DV indicates a more isotropic basin where the loss grows at a similar rate regardless of the sampled direction.
A.2
Rank related metrics
Since each low-rank pre-training method imposes a different structural prior on the weight matrices it learns, we examine the singular value distribution of the trained checkpoints to characterize how that prior manifests across the training horizon. These metrics surface complementary aspects of the spectrum: how much of the matrix’s energy concentrates in the leading components, how broadly the remaining mass is distributed, and how many directions actively contribute to the matrix’s action. Following the probe family adopted by [30] in their study of task-vector alignment, we restrict our analysis to the attention and MLP weight matrices, as these are the layers where the low-rank parametrization is enforced. For each two-dimensional weight W ∈ Rm×n , we obtain the ordered singular values σ1 ≥ σ2 ≥ · · · ≥ σr ≥ 0 with r = min(m, n) and compute the following quantities. • Effective rank: The exponential of the entropy of the normalized spectrum, treating singular values as a probability distribution: ! r X σi . (3) EffRank(W ) = exp − pi log pi , pi = P j σj i=1 This quantity equals 1 when the spectrum collapses entirely onto a single direction and approaches r as the singular values flatten toward uniformity. • Stable rank: A spectrum-based dimensionality measure defined as the ratio of the Frobenius norm to the operator norm, P 2 ∥W ∥2F i σi StableRank(W ) = = . (4) ∥W ∥22 σ12 It is upper-bounded by the algebraic rank but, unlike the latter, is insensitive to negligibly small singular values. A value of 1 corresponds to an essentially rank-one matrix, while a value approaching r reflects a near-uniform spectrum. • Spectral gap: The relative separation between the leading two singular values, SpectralGap(W ) = (σ1 − σ2 )/σ1 . A pronounced gap signals that the top direction stands out clearly from the rest of the spectrum; a vanishing gap means the first two directions carry comparable weight and are not robustly distinguishable. • Threshold rank: A hard count of singular values whose magnitude exceeds a fixed cutoff τ = 0.1, namely TRankτ (W ) = #{ i : σi > τ }. This filters out tail components that are numerically nonzero but have negligible influence on the matrix’s action, providing a noise-robust alternative to the strict rank.
A.3
Activation-based metrics.
To measure how closely a low-rank checkpoint’s internal representations match those of the full-rank baseline, we compare their hidden states on identical inputs. Using the same fixed evaluation subset and seed as in section 3.1.1, we run both the full-rank baseline fA and the low-rank target fB and extract the hidden states at every layer ℓ ∈ {0, 1, . . . , L}, where ℓ = 0 denotes the embedding output. The resulting per-layer activation matrices are (ℓ) (ℓ) HA ∈ RN ×d , HB ∈ RN ×d , respectively, where N is the total number of token positions in a sequence and d is the hidden dimension. Let (ℓ) (ℓ) (ℓ) (ℓ) hA,i and hB,i denote the i-th row of HA and HB respectively, i.e. the hidden activation vectors at position i. We then report the following per-layer metrics. • Activation L2 distance: the mean per-position Euclidean distance between the two activation matrices, N
(ℓ)
dL2 =
1 X (ℓ) (ℓ) h − hB,i . N i=1 A,i 2
(5)
A value of 0 means the two checkpoints produce identical activations on the input, and larger values indicate that the low-rank model departs further from the baseline in the absolute feature space.
12
• Activation cosine similarity: the mean per-position directional alignment between the two activation vectors, (ℓ) (ℓ) N ⟨hA,i , hB,i ⟩ 1 X cos(ℓ) = . (6) N i=1 ∥h(ℓ) ∥2 ∥h(ℓ) ∥2 + ε A,i
B,i
A value of 1 indicates that, at every position, the low-rank model points in the same direction in feature space as the baseline; values near 0 indicate orthogonal, unrelated features. Unlike the L2 distance, this is insensitive to activation magnitude and isolates the directional component of the mismatch. • Linear CKA: following [11], we compute the linear Centered Kernel Alignment between the two full (ℓ) (ℓ) activation matrices as a representation-level similarity. Let H̃A and H̃B denote the column-centered (ℓ) (ℓ) versions of HA and HB , and define the cross- and self-Gram matrices (ℓ) ⊤ (ℓ) (ℓ) ⊤ (ℓ) (ℓ) ⊤ (ℓ) GAB = H̃B H̃A , GAA = H̃A H̃A , GBB = H̃B H̃B . Linear CKA is then
∥GAB ∥2F . (7) ∥GAA ∥F ∥GBB ∥F Linear CKA is invariant to orthogonal rotations and isotropic scaling of the hidden features, so it captures whether the two checkpoints encode the same representational subspace even when individual neurons have been permuted or re-scaled. A value of 1 indicates perfectly aligned subspaces and a value near 0 indicates unrelated ones. CKA(ℓ) =
B
Downstream evaluation
We report downstream performance per task for 60M, 130M, 350M in Table 3, 4, and 5, respectively.
Table 3: 60M models final checkpoint evaluation on downstream tasks. Per-task results as presented as mean ± bootstrap standard error. Accuracy/F1 metrics are shown in %. Section abbreviations: CS = Commonsense, WK = World Knowledge, RC = Reading Comprehension, Gr. = Grammar, LR = Logical Reasoning, IT = Instruction / Truthfulness. Metric symbols: † length-normalised accuracy, ‡ F1, ↓ lower-is-better. Per task, the best method is bold and the second-best is underlined; n/a marks tasks for which bootstrap stderr is unavailable. The Random column gives the chance-level accuracy of each task in %; “–” marks tasks without a well-defined chance baseline (F1 / perplexity). Task
Random
LL A MA
G A L ORE
F IRA
C O LA
SLT RAIN
R E L O RA
CS HellaSwag† PIQA COPA SWAG†
25.0 50.0 50.0 25.0
26.86 ± 0.44 58.49 ± 1.15 54.00 ± 5.01 36.83 ± 0.34
28.00 ± 0.45 59.63 ± 1.15 53.00 ± 5.02 36.78 ± 0.34
27.06 ± 0.44 60.28 ± 1.15 61.00 ± 4.90 37.26 ± 0.34
27.90 ± 0.45 60.45 ± 1.15 55.00 ± 5.00 37.09 ± 0.34
27.92 ± 0.45 60.07 ± 1.15 55.00 ± 5.00 36.08 ± 0.34
27.30 ± 0.44 58.43 ± 1.16 56.00 ± 4.99 34.33 ± 0.34
WK OpenBookQA† ARC-Easy QA4MRE-2013†
25.0 25.0 20.0
28.40 ± 2.02 29.60 ± 2.04 30.80 ± 2.07 29.60 ± 2.04 28.40 ± 2.02 31.80 ± 2.08 28.83 ± 0.93 30.22 ± 0.94 29.42 ± 0.93 29.17 ± 0.94 28.87 ± 0.94 28.49 ± 0.93 20.77 ± 2.41 26.06 ± 2.61 21.83 ± 2.46 22.54 ± 2.48 21.83 ± 2.46 22.89 ± 2.50
RC ReCoRD‡
–
36.39 ± 0.48 37.36 ± 0.48 39.17 ± 0.49 40.93 ± 0.49 38.06 ± 0.48 40.32 ± 0.49
Gr. BLiMP
50.0
74.76 ± 0.15 74.08 ± 0.15 76.37 ± 0.14 73.40 ± 0.15 72.99 ± 0.15 70.14 ± 0.16
LR LogiQA† LogiQA2†
25.0 25.0
26.27 ± 1.73 28.73 ± 1.77 28.11 ± 1.76 25.81 ± 1.72 27.50 ± 1.75 26.27 ± 1.73 25.45 ± 1.10 25.06 ± 1.09 25.38 ± 1.10 26.97 ± 1.12 25.57 ± 1.10 25.06 ± 1.09
C
More results
We report additional plots and results for all the metric families in this section.
C.1
1-D loss landscape
This section reports the full set of 1-D loss landscape plots complementing the 350M random-direction results and the top-1 PCA results in Figure 2 of the main paper. Figures 6–11 show random-direction landscapes at
13
Table 4: 130M models final checkpoint evaluation on downstream tasks. Per-task results as presented as mean ± bootstrap standard error. Accuracy/F1 metrics are shown in %. Section abbreviations: CS = Commonsense, WK = World Knowledge, RC = Reading Comprehension, Gr. = Grammar, LR = Logical Reasoning, IT = Instruction / Truthfulness. Metric symbols: † length-normalised accuracy, ‡ F1, ↓ lower-is-better. Per task, the best method is bold and the second-best is underlined; n/a marks tasks for which bootstrap stderr is unavailable. The Random column gives the chance-level accuracy of each task in %; “–” marks tasks without a well-defined chance baseline (F1 / perplexity). Task
Random
LL A MA
G A L ORE
F IRA
C O LA
SLT RAIN
R E L O RA
CS HellaSwag† PIQA COPA SWAG†
25.0 50.0 50.0 25.0
28.58 ± 0.45 60.28 ± 1.14 62.00 ± 4.88 38.72 ± 0.34
29.44 ± 0.45 59.58 ± 1.14 58.00 ± 4.96 39.82 ± 0.35
29.52 ± 0.46 60.50 ± 1.14 58.00 ± 4.96 39.34 ± 0.35
28.78 ± 0.45 60.99 ± 1.14 58.00 ± 4.96 39.02 ± 0.34
29.16 ± 0.45 59.74 ± 1.14 58.00 ± 4.96 38.90 ± 0.34
26.99 ± 0.44 58.81 ± 1.15 54.00 ± 5.01 35.05 ± 0.34
WK OpenBookQA† ARC-Easy QA4MRE-2013†
25.0 25.0 20.0
29.20 ± 2.04 32.00 ± 2.09 29.80 ± 2.05 30.80 ± 2.07 29.40 ± 2.04 30.60 ± 2.06 30.01 ± 0.94 31.10 ± 0.95 29.50 ± 0.94 30.68 ± 0.95 30.47 ± 0.94 28.91 ± 0.93 23.24 ± 2.51 21.48 ± 2.44 24.30 ± 2.55 19.37 ± 2.35 25.00 ± 2.57 25.35 ± 2.59
RC ReCoRD‡
–
43.98 ± 0.49 39.00 ± 0.49 45.86 ± 0.50 49.60 ± 0.50 42.48 ± 0.49 39.02 ± 0.49
Gr. BLiMP
50.0
75.96 ± 0.14 76.00 ± 0.14 77.39 ± 0.14 78.19 ± 0.14 76.71 ± 0.14 74.62 ± 0.15
LR LogiQA† LogiQA2†
25.0 25.0
28.26 ± 1.77 25.65 ± 1.71 27.19 ± 1.75 28.88 ± 1.78 26.11 ± 1.72 26.73 ± 1.74 27.61 ± 1.13 27.61 ± 1.13 26.78 ± 1.12 26.65 ± 1.12 25.64 ± 1.10 26.08 ± 1.11
all three scales (60M, 130M, 350M), each presented twice — once excluding ReLoRA and once including it. Because ReLoRA’s sharpness is one to two orders of magnitude larger than the other methods at every scale, including it compresses the y-axis and obscures the relative ordering among the remaining methods; we therefore provide both views. Figures 12–14 extend the top-k PCA-direction analysis to k ∈ {1, 5, 10, 20} at each scale.
D
Rank related metrics
This section reports the rank and spectral metric trajectories at the 60M and 130M scales, complementing the 350M results shown in Figure 3 of the main paper. Both figures follow the same layout as Figure 3 Row 1 tracks the four spectral metrics — effective rank, stable rank, spectral gap, and the count of singular values above 0.1 — on the trained weight W across training; Row 2 tracks the same metrics on the consecutive-checkpoint update ∆W = Wt+1 − Wt ; and Row 3 shows the singular value distributions of ∆W per projection type (WQ , WV , Wup ), with the rightmost panel reporting the per-projection count of singular values above 0.1. The same method ordering observed at 350M — GaLore ≈ Fira > SLTrain > CoLA > ReLoRA — holds at both smaller scales: optimizer-based methods (GaLore, Fira) most faithfully reproduce full-rank’s update geometry, while parameterization-based methods (CoLA, SLTrain) and ReLoRA systematically underpopulate the small-singular-value tail in the attention WV /WO blocks.
E
Activations
This section reports the full per-layer × per-training-step activation heatmaps for every method at every model scale, complementing the bar summaries and the Fira/CoLA-only heatmaps shown in Figure 4 of the main paper. All three figures share the same grid layout: rows index the five low-rank methods (GaLore, Fira, CoLA, SLTrain, ReLoRA) and columns index the three model scales (60M, 130M, 350M); within each panel, the x-axis is the training step and the y-axis is the decoder layer index, where layer 0 corresponds to the embedding output. Figure 17 reports the per-layer Activation L2 distance to the full-rank baseline (Eq. 5), Figure 18 reports the per-layer Linear CKA similarity (Eq. 7), and Figure 19 reports the per-layer Activation cosine similarity (Eq. 6). Across all three metrics, divergence from full-rank concentrates in the later decoder layers and grows with training, with CoLA showing the most severe directional mismatch — cosine similarity remains near zero across all layers and steps at 130M and 350M.
14
Table 5: 350M models final checkpoint evaluation on downstream tasks. Per-task results as presented as mean ± bootstrap standard error. Accuracy/F1 metrics are shown in %. Section abbreviations: CS = Commonsense, WK = World Knowledge, RC = Reading Comprehension, Gr. = Grammar, LR = Logical Reasoning, IT = Instruction / Truthfulness. Metric symbols: † length-normalised accuracy, ‡ F1, ↓ lower-is-better. Per task, the best method is bold and the second-best is underlined; n/a marks tasks for which bootstrap stderr is unavailable. The Random column gives the chance-level accuracy of each task in %; “–” marks tasks without a well-defined chance baseline (F1 / perplexity). Task
Random
LL A MA
G A L ORE
F IRA
C O LA
SLT RAIN
R E L O RA
CS HellaSwag† PIQA COPA SWAG†
25.0 50.0 50.0 25.0
31.75 ± 0.46 61.81 ± 1.13 66.00 ± 4.76 41.19 ± 0.35
31.95 ± 0.47 63.17 ± 1.13 60.00 ± 4.92 42.37 ± 0.35
34.44 ± 0.47 63.82 ± 1.12 63.00 ± 4.85 42.67 ± 0.35
32.41 ± 0.47 61.75 ± 1.13 65.00 ± 4.79 42.65 ± 0.35
31.23 ± 0.46 62.19 ± 1.13 61.00 ± 4.90 41.79 ± 0.35
27.59 ± 0.45 59.41 ± 1.15 60.00 ± 4.92 35.79 ± 0.34
WK OpenBookQA† ARC-Easy QA4MRE-2013†
25.0 25.0 20.0
29.20 ± 2.04 34.00 ± 2.12 36.40 ± 2.15 34.40 ± 2.13 32.20 ± 2.09 29.80 ± 2.05 31.06 ± 0.95 31.48 ± 0.95 30.85 ± 0.95 29.80 ± 0.94 30.51 ± 0.94 29.00 ± 0.93 23.59 ± 2.52 23.24 ± 2.51 23.94 ± 2.54 22.54 ± 2.48 25.35 ± 2.59 17.25 ± 2.25
RC ReCoRD‡
–
45.41 ± 0.50 41.16 ± 0.49 47.00 ± 0.50 55.54 ± 0.49 52.16 ± 0.50 42.37 ± 0.49
Gr. BLiMP
50.0
79.46 ± 0.14 77.06 ± 0.14 76.20 ± 0.15 78.16 ± 0.14 78.70 ± 0.14 76.55 ± 0.15
LR LogiQA† LogiQA2†
25.0 25.0
26.11 ± 1.72 27.65 ± 1.75 25.50 ± 1.71 29.03 ± 1.78 27.80 ± 1.76 27.34 ± 1.75 25.89 ± 1.11 26.27 ± 1.11 27.04 ± 1.12 26.65 ± 1.12 27.93 ± 1.13 25.64 ± 1.10
Table 6: Top-8 sign-consistent geometry features used in the combined predictor. Per-size Spearman correlation between each feature and the 11-task downstream mean. All three sizes share the same sign; features are ranked by |median ρ|.
F
#
Feature
1 2 3 4 5 6 7 8
barrier_consec (CCBH) act_cka_mean act_l2_mean stable_rank_dW eff_rank_dW threshold_rank_W spectral_gap_dW stable_rank_W
Lens
60M
130M
350M
|e ρ|
interpolation activation activation rank ∆W rank ∆W rank W rank ∆W rank W
−0.69 −0.62 +0.66 +0.28 +0.30 +0.21 −0.29 −0.20
−0.70 −0.42 +0.62 +0.62 +0.70 +0.37 −0.20 −0.11
−0.55 −0.78 +0.55 +0.35 +0.18 +0.05 −0.18 −0.17
0.69 0.62 0.62 0.35 0.30 0.21 0.20 0.17
Correlation results
15
(0) ( )
0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000
step 1000
0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000
0.002 0.000 0.002
step 5000
0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000
0.002 0.000 0.002
0.002 0.000 0.002
direction variance
0.008 0.006 0.004 2000
4000
6000
8000
training step
cola
fira
step 10000
0.002 0.000 0.002
Direction variance vs training step
8 6 4 2 0
10000
0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000
0.002 0.000 0.002
1e 6
0.010
0.002
step 8000
0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000
Expected sharpness vs training step
0.012
expected sharpness
step 3000
galore
2000
4000
fullrank
6000
training step
8000
10000
sltrain
Figure 6: 1-D loss landscape at 60M parameters. Top: centered loss profile L(α) − L(0) averaged over 100 random directions, at five training checkpoints (1k, 3k, 5k, 8k, 10k). Bottom-left: average sharpness with respect to training step. Bottom-right: average direction variance. ReLoRA is omitted from the plot as it makes it harder to view other methods. The plot including ReLoRA is given in 7 .
( )
(0)
0.20
step 1000
0.20
step 3000
step 5000
0.20
step 8000
0.20
0.15
0.15
0.15
0.15
0.15
0.10
0.10
0.10
0.10
0.10
0.05
0.05
0.05
0.05
0.05
0.00
0.00 0.002 0.000 0.002
0.00 0.002 0.000 0.002
0.00 0.002 0.000 0.002
Expected sharpness vs training step
0.04 0.02 0.00
2000
4000
6000
training step
cola
8000
fira
galore
0.002 0.000 0.002
Direction variance vs training step
6 4 2 0
10000
step 10000
0.00 0.002 0.000 0.002
8 1e 5
0.06
direction variance
expected sharpness
0.20
fullrank
2000
4000
relora
6000
training step
8000
10000
sltrain
Figure 7: 1-D loss landscape at 60M parameters for All methods. Top: centered loss profile L(α) − L(0) averaged over 100 random directions, at five training checkpoints (1k, 3k, 5k, 8k, 10k). Bottom-left: average expected sharpness with respect to training step. Bottom-right: average direction variance.
16
( )
(0)
step 2000
step 4000
step 10000
step 20000
0.020
0.020
0.020
0.020
0.015
0.015
0.015
0.015
0.015
0.010
0.010
0.010
0.010
0.010
0.005
0.005
0.005
0.005
0.005
0.000
0.000 0.002 0.000 0.002
0.000
0.000
0.002 0.000 0.002
5
direction variance
0.007 0.006 0.005 0.004 0.003 0.002 2500
5000
7500
cola
fira
0.002 0.000 0.002
0.002 0.000 0.002
Direction variance vs training step
1e 6
4 3 2 1 0
10000 12500 15000 17500 20000
training step
0.000
0.002 0.000 0.002
Expected sharpness vs training step expected sharpness
step 16000
0.020
2500
galore
5000
7500
fullrank
10000 12500 15000 17500 20000
training step
sltrain
Figure 8: 1-D loss landscape at 130M parameters. Same layout as Figure 6, with top-row checkpoints at {2k, 6k, 10k, 14k, 20k}.
( )
(0)
0.30
step 2000
0.30
step 4000
step 10000
0.30
0.30
0.25
0.25
0.25
0.25
0.25
0.20
0.20
0.20
0.20
0.20
0.15
0.15
0.15
0.15
0.15
0.10
0.10
0.10
0.10
0.10
0.05
0.05
0.05
0.05
0.05
0.00
0.00
0.00
0.00
0.002 0.000 0.002
0.002 0.000 0.002
0.002 0.000 0.002
step 20000
0.00 0.002 0.000 0.002
Expected sharpness vs training step
0.002 0.000 0.002
Direction variance vs training step
0.10
direction variance
expected sharpness
step 16000
0.30
0.08 0.06 0.04 0.02 0.00
2500
5000
7500
cola
fira
0.00010 0.00005 0.00000
10000 12500 15000 17500 20000
training step
0.00015
galore
2500
fullrank
5000
7500
relora
10000 12500 15000 17500 20000
training step
sltrain
Figure 9: 1-D loss landscape at 130M parameters for All methods.
17
( )
(0)
0.030
step 6000
0.030
step 12000
step 24000
0.030
step 48000
0.025
0.025
0.025
0.025
0.020
0.020
0.020
0.020
0.020
0.015
0.015
0.015
0.015
0.015
0.010
0.010
0.010
0.010
0.010
0.005
0.005
0.005
0.005
0.005
0.000
0.000
0.000
0.000
0.002 0.000 0.002
0.002 0.000 0.002
Expected sharpness vs training step
2.0
direction variance
0.010 0.008 0.006 0.004 0.002 10000
20000
30000
40000
50000
training step
cola
fira
0.000 0.002 0.000 0.002
0.002 0.000 0.002
Direction variance vs training step
1e 5
1.5 1.0 0.5 0.0
60000
step 60000
0.030
0.025
0.002 0.000 0.002
expected sharpness
0.030
galore
10000
20000
fullrank
30000
40000
50000
training step
60000
sltrain
Figure 10: 1-D loss landscape at 350M parameters. Same layout as 6.
( )
(0)
0.4
step 6000
0.4
step 12000
step 24000
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.1
0.0
0.0 0.002 0.000 0.002
0.0 0.002 0.000 0.002
0.0 0.002 0.000 0.002
direction variance
0.100 0.075 0.050 0.025 10000
20000
30000
40000
training step
cola
50000
fira
0.002 0.000 0.002
Direction variance vs training step
0.125
0.000
step 60000
0.0 0.002 0.000 0.002
Expected sharpness vs training step expected sharpness
step 48000
0.00015 0.00010 0.00005 0.00000
60000
galore
fullrank
10000
20000
relora
30000
40000
training step
sltrain
Figure 11: 1-D loss landscape at 350M parameters for All methods.
18
50000
60000
k=1 ( ) (0)
step 1000
k=5 ( ) (0) k = 10 ( ) (0)
step 8000
step 10000
0.02
0.02
0.02
0.01
0.01
0.01
0.01
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.004
0.004
0.004
0.004
0.004
0.002
0.002
0.002
0.002
0.002
0.000
0.000
0.000
0.000
0.05
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.00
0.05
0.002
0.002
0.002
0.002
0.000
0.000
0.000
0.000
0.000
0.00
0.001
0.05
0.05
0.00
0.05
0.001
0.000 0.00
0.05
0.00
0.05
0.001
0.000 0.05
0.05
0.00
k=1
0.05
0.05
0.00
0.00
0.05
k=5 0.0008
0.004
0.0010
0.0006
0.002
0.00
2000 4000 6000 8000 10000
dir variance
1.5 1e 5
7.5
1.0
5.0
0.5
2.5 2000 4000 6000 8000 10000
step
fullrank
galore
fira
0.05
0.05
0.05
0.00
0.05
0.05
0.00
0.05
k = 20
0.0003 0.0002
step
k=5
0.00
0.0004
2000 4000 6000 8000 10000
step
0.05
0.0005
0.0004
step
0.05
0.000 0.05
k = 10
0.0015
0.00
0.001
0.000 0.05
0.006
2000 4000 6000 8000 10000
0.05
0.001
0.000 0.05
0.05
0.000 0.05
0.002
0.05
k = 20 ( ) (0)
step 5000
0.02
0.05
expected sharpness
step 3000
0.02
1e 6
2000 4000 6000 8000 10000
step
k = 10 4
1e 6
k = 20
2 2000 4000 6000 8000 10000
step
cola
sltrain
2000 4000 6000 8000 10000
step
relora
Figure 12: 1-D loss landscape along top-k PCA directions at 60M parameters for k ∈ {1, 5, 10, 20}. Top four rows: centered loss profile at five training checkpoints. Bottom: expected sharpness (left column of summary panels) and across-component direction variance (right column) as a function of training step, per k.
19
k=1 ( ) (0)
step 2000
k=5 ( ) (0)
step 10000
step 16000
step 20000
0.02
0.02
0.02
0.02
0.02
0.01
0.01
0.01
0.01
0.01
0.00
0.00
0.00
0.00
0.05
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.00 0.05
0.00
0.05
0.05
0.00
0.05
0.0000 0.05 0.05
0.00
0.05
0.00
0.05
0.00
0.05
0.0050
0.0050
0.0050
0.0050
0.0050
0.0025
0.0025
0.0025
0.0025
0.0025
0.0000
0.0000 0.05 0.05
0.0000 0.05 0.05
0.0000 0.05 0.05
k = 10 ( ) (0)
0.05
0.00
0.00
0.00
0.004
0.004
0.004
0.004
0.002
0.002
0.002
0.002
0.002
0.05
k = 20 ( ) (0)
0.00
0.004
0.000 0.00
0.000 0.05 0.05
0.00
0.000 0.05 0.05
0.00
0.000 0.05 0.05
0.00
0.000 0.05 0.05
0.002
0.002
0.002
0.002
0.002
0.001
0.001
0.001
0.001
0.001
0.000 0.05
expected sharpness
step 6000
0.00
0.000 0.05 0.05
0.00
k=1
0.000 0.05 0.05
0.00
0.000 0.05 0.05
k=5
0.0075 0.0050 0.0025 5000 10000 15000 20000
0.0020 0.0015 0.0010 0.0005
step
dir variance
k = 20 0.0006 0.0004
0.0005 5000 10000 15000 20000
1e 5
0.0002 5000 10000 15000 20000
1e 5
k = 10
1
0.5
0
0.0
step
fira
step
6
1.0
5000 10000 15000 20000
5000 10000 15000 20000
step
k=5
galore
0.000 0.05 0.05
k = 10 0.0010
step
fullrank
0.00
1e 6
k = 20
4 2 5000 10000 15000 20000
step
cola
sltrain
0
5000 10000 15000 20000
step
relora
Figure 13: 1-D loss landscape along top-k PCA directions at 130M parameters for k ∈ {1, 5, 10, 20}. Layout identical to Figure 12.
20
k=1 ( ) (0)
step 6000
k=5 ( ) (0)
step 30000
step 48000
step 60000
0.04
0.04
0.04
0.04
0.04
0.02
0.02
0.02
0.02
0.02
0.00
0.00
0.00
0.00
0.05
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.00 0.05
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.0000 0.05 0.05
0.00
0.05
0.00
0.05
0.010
0.010
0.010
0.010
0.010
0.005
0.005
0.005
0.005
0.005
0.000
0.000
0.000
0.000
0.05
k = 10 ( ) (0)
step 18000
0.00
0.05
0.05
0.00
0.05
0.05
0.00
0.05
0.000 0.05
0.00
0.05
0.0050
0.0050
0.0050
0.0050
0.0050
0.0025
0.0025
0.0025
0.0025
0.0025
0.0000
0.0000 0.05 0.05
0.0000 0.05 0.05
0.0000 0.05 0.05
k = 20 ( ) (0)
0.05
0.00
0.002
0.002
0.000 0.05
expected sharpness
0.00
0.002
0.000 0.05 0.05
0.00
0.00
0.002
0.000 0.05 0.05
0.00
k=1
k=5
k = 10
0.001 20000
40000
step
60000
20000
dir variance
6
fullrank
40000
step
60000
20000
k=5
1e 5
2 20000
40000
step
galore
60000
fira
3 2 1 0
0.000 0.05 0.05
k = 20
0.00100 0.00075 0.00050 0.00025
40000
step
60000
k = 10
1e 5
4 0
0.00
0.0015 0.0010 0.0005
0.002
0.005
0.002
0.000 0.05 0.05
0.00
0.003
0.010
0.00
20000 2
40000
step
60000
k = 20
1e 5
1 20000
cola
40000
step
60000
sltrain
0
20000
40000
step
60000
relora
Figure 14: 1-D loss landscape along top-k PCA directions at 350M parameters for k ∈ {1, 5, 10, 20}. Layout identical to Figure 12. CoLA’s direction variance grows monotonically with training across all k, while other methods remain near zero, indicating that CoLA’s sharpness is distributed anisotropically across its top-k subspace rather than concentrated in any single direction.
21
2k
4k
6k
training step
8k
10k
4k
6k
training step
8k
10k
120
300 200 100 2k
4k
6k
8k
training step
100 80 60 40 20
10k
400
2k
0.15 0.10 0.05 0.00
2k
4k
6k
8k
training step
10k
2.5
8k
2k
0.5
0.5
0.0 0.0
0.0 0.0
0.0 0.0
0.5
full-rank
1.0
(WV)
1.5
galore
2.0
4k
6k
8k
10k
6k
8k
10k
training step
fira
300 200 100 0
10k
2k
4k
training step
500
1.0
0.5 2.0
6k
training step
1.5
1.0
1.5
4k
2.0
1.5
200
400
2k
1.5
300
10k
0.05
3.0
(WQ)
8k
0.10
2.0
1.0
6k
0.15
2.5
0.5
4k
training step
0.20
2.0
1.0
# > 0.1 of W
0.20
0.25
spectral gap of W
400
Frequency ( W)
2k
stable rank of W
effective rank of W
100
500
# > 0.1 of W
200
0.25
# > 0.1 of W
300
70 60 50 40 30 20
spectral gap of W
stable rank of W
effective rank of W
400
0.5
1.0
(Wup)
cola
1.5
2.0
sltrain
400 300 200 100 0
WQ WK WV WO WgateWupWdown
relora
10k
15k
training step
20
20k
stable rank of W
600 400 200
Frequency ( W)
effective rank of W
5k
5k
10k
15k
training step
5k
10k
15k
training step
150 100 50 0
20k
20k
5k
10k
15k
training step
20k
0.10 0.05 0.00
0.00 3.0
1.5
2.5
1.5
2.0
0.0 0.0
full-rank
10k
15k
training step
1.0
(WV)
galore
1.5
2.0
fira
0.0 0.0
0.5
cola
1.0
(Wup)
1.5
300 10k
15k
20k
10k
15k
20k
training step
600 400 200 0
2.0
sltrain
Figure 16: Rank and Spectral metrics for 130M.
22
400
5k
training step
800
0.5 0.5
500
5k
20k
1.0
0.5
(WQ)
5k
1.5
1.0
1.0
training step
600
20k
2.0
1.5
0.5
15k
0.05
2.0
0.0 0.0
10k
0.10
2.5
0.5
5k
0.15
2.0
1.0
# > 0.1 of W
40
0.15
# > 0.1 of W
200
60
700
0.20
# > 0.1 of W
300
spectral gap of W
400
80
spectral gap of W
500
0
0.25
100
600
stable rank of W
effective rank of W
Figure 15: Rank and Spectral metrics for 60M.
600 400 200 0
WQ WK WV WO WgateWupWdown
relora
60M
130M
12
24
350M
GaLore layers
8 0
0
1k 3k 5k 7k 9k
0
2k 6k 10k 14k 18k
24
12
FIRA layers
8
6k 18k 30k 42k 54k
1k 3k 5k 7k 9k
8
0 100
CoLA layers
80
2k 6k 10k 14k 18k
12
60
0
40
1k 3k 5k 7k 9k
20
0
2k 6k 10k 14k 18k
700 600 500 400 300 200 100
6k 18k 30k 42k 54k
24
1k 3k 5k 7k 9k
ReLoRA layers
2k 6k 10k 14k 18k
0
400
6k 18k 30k 42k 54k
1k 3k 5k 7k 9k training step
0
0
6k 18k 30k 42k 54k
24
12
8 0
0
2k 6k 10k 14k 18k training step
Figure 17: Activation L2 distance layer-wise
23
1000 600
SLTrain layers 0
1200 800
24
12
8
0
0
6k 18k 30k 42k 54k training step
200
Activation L2 distance
0
60M
130M
12
24
350M
GaLore layers
8 0
0
1k 3k 5k 7k 9k
0
2k 6k 10k 14k 18k
12
FIRA layers
8
6k 18k 30k 42k 54k
24
1k 3k 5k 7k 9k
8
0 0.6
CoLA layers
0.5
2k 6k 10k 14k 18k
12
0.5
0.4
0
0.3
1k 3k 5k 7k 9k
0.1
0
2k 6k 10k 14k 18k
12
6k 18k 30k 42k 54k
24
0
1k 3k 5k 7k 9k
0.4
0.5
0.3
0.4
0.2
0
0.1
24
ReLoRA layers
2k 6k 10k 14k 18k
6k 18k 30k 42k 54k
1k 3k 5k 7k 9k training step
0
0
6k 18k 30k 42k 54k
24
12
8 0
0
2k 6k 10k 14k 18k training step
Figure 18: Activation linear CKA similarity
24
0.7 0.6
SLTrain layers
8
0.2
0 0.6
Linear CKA
0
0
6k 18k 30k 42k 54k training step
0.3 0.2
60M
130M
12
24
350M
GaLore layers
8 0
0
1k 3k 5k 7k 9k
0
2k 6k 10k 14k 18k
24
12
FIRA layers
8
6k 18k 30k 42k 54k
1k 3k 5k 7k 9k
8
0.03
CoLA layers
0.02
0
2k 6k 10k 14k 18k
12
1k 3k 5k 7k 9k
0.25
24
6k 18k 30k 42k 54k
0.15
0.15
0.00
0
0.01
12
0.10
0.10
2k 6k 10k 14k 18k
0
0.05 0.00
0
1k 3k 5k 7k 9k
6k 18k 30k 42k 54k
24
1k 3k 5k 7k 9k training step
0
ReLoRA layers
0
2k 6k 10k 14k 18k
6k 18k 30k 42k 54k
24
12
8 0
0
2k 6k 10k 14k 18k training step
Figure 19: Activation cosine similarity layer-wise
25
0.25 0.20
SLTrain layers
8
0
0.20
0.01
0
0.30
Activation cosine similarity
0
0
6k 18k 30k 42k 54k training step
0.05 0.00