Published as a conference paper at ICOMP 2026
N O S UBSPACE TO T RACK : N ON -I DENTIFIABILITY AND O PTIMIZER S TATE IN L OWR ANK T RAINING Noel Thomas Mohamed bin Zayed University of Artificial Intelligence [email protected]
arXiv:2607.05872v1 [cs.LG] 7 Jul 2026
A BSTRACT Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace that is recomputed every T steps, on the assumption that this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimates of the top-r subspace computed at the same step from disjoint minibatches disagree as much as estimates √ computed T steps apart (0.73 versus 0.74 of the maximal chordal distance 2r, measured at Pythia-160M with r=128): the apparent rotation at every refresh is dominated by estimator noise. This holds across four model families in three architecture classes and from 70M to 6.9B parameters, strengthening with scale, and more weakly in a vision transformer. Only ∼39 of 128 directions are reproducible across minibatches, and averaging cannot recover the rest: under N -fold averaging the gradient’s spectral tail shrinks as N −1/4 rather than the N −1/2 of pure noise, so no averaging budget makes the subspace well defined. At Pythia-1B, a basis averaged over ten thousand steps still moves by ∼84% of the geometric maximum per refresh, and sweeping the averaging strength leaves perplexity essentially unchanged (a 0.36 perplexity span). What helps instead follows from treating each refresh as a change of coordinates for Adam’s state. Carrying the second moment blindly is provably about (r−k ⋆ )/2 worse than the best rotation-blind estimator, while the first moment transports exactly through the rotation, the optimal linear map under isotropic gradients and the rule LDAdam uses. In a three-seed comparison at 1B and 40k steps, the full LDAdam update reaches 18.7 perplexity at the default β2 =0.999, beating untransported GaLore after its best β2 fix (19.3). Shortening the second-moment memory to β2 =0.99 helps the refreshing optimizers at their reported recipes, by 1.8 to 26 perplexity, though for canonical GaLore the effect is small and recipe-sensitive; a full-rank control reverses the preference. One measurable fact, subspace non-identifiability, clarifies why GaLore works, which patches work, and what to check before trusting a low-rank assumption: the reproducible rank k ⋆ .
1
I NTRODUCTION
Training large models is increasingly bottlenecked by optimizer memory: Adam stores two moments per parameter, doubling the footprint of the weights themselves. Low-rank gradient optimizers such as GaLore (Zhao et al., 2024) sidestep this by projecting each gradient G ∈ Rm×n onto the top-r left-singular subspace of G, running Adam in the resulting r-dimensional space, and recomputing the subspace every T steps. This matches full-rank Adam at a fraction of the memory, and a family of refinements has grown around it (Robert et al., 2025; Rajabi et al., 2025b; Zmushko et al., 2024; He et al., 2024; Zhu et al., 2024). Underlying the whole approach is a geometric premise that is rarely stated and, to our knowledge, was never measured: that the gradient’s top-r subspace is a meaningful, slowly varying object that the refresh tracks. If the premise held, two consecutive estimates of the subspace would nearly coincide, and carrying optimizer state across a refresh would be a small correction. 1
Published as a conference paper at ICOMP 2026
We measure the premise and find that beyond a small reproducible core, there is no such object. The rotation between consecutive subspaces, measured as the chordal distance ∥∆U ∥F , saturates √ its geometric maximum 2r to 96.7–99.6% at every refresh and at every rank r ∈ {32, . . . , 256} during training (Pythia-1B (Biderman et al., 2023); 160M behaves the same): each refresh adopts a nearly orthogonal frame. The decisive measurement is the one that removes time. On a pretrained Pythia-160M checkpoint, two estimates√of the top-r subspace computed at the √ same step from disjoint minibatches disagree by 0.725 2r, within a few percent of the 0.742 2r the subspace rotates across a full refresh interval of 160M training (a gap the trajectory control of §3.2 confirms is small at every training state). The apparent rotation is not motion. It is estimator noise: of the r=128 directions GaLore extracts, only k ⋆ ≈ 39 are reproducible across two halves of one batch, and the rest are re-drawn with the sample. (Both checkpoint values sit below the training-time saturation for a reason the paper quantifies: a pretrained gradient’s reproducible spike is stronger, and the k ⋆ shared directions pull the two estimates together.) The cause is spectral, and it is not curable by averaging. The per-matrix gradient spectrum has a small dominant spike, a few to ∼20 directions consistent with the tiny gradient subspace documented for classification by Gur-Ari et al. (2018), followed by a smooth tail with no gap anywhere near the ranks GaLore uses: at r=128 the adjacent singular-value ratio is ≈1.005, the effective rank straddles the cutoff, and the top-r block holds only 46–68% of the gradient’s nuclear mass. A rank cut inside a gapless bulk returns near-degenerate directions that resample freely (Davis–Kahan). Averaging N gradients does not open a gap: the deep spectral tail shrinks only as ≈N −1/4 rather than the N −1/2 of pure noise, because it is itself signal with a power-law spectrum, so no averaging budget makes the top-r frame well defined. GaLore works anyway, and the reason is unglamorous: it captures a roughly constant fraction of gradient energy in a frame that is re-drawn, not tracked (§4). The redraw has a concrete cost, and the cost has remedies that we can now rank. Adam’s second moment is a slow average with memory τv = 1/(1 − β2 ) ≈ 1000 steps at the default β2 =0.999; a frame scrambled every T ≪ τv steps leaves it chronically misaligned. The obvious fix, stabilizing the basis by averaging the projection over time, fails exactly as the geometry predicts: at Pythia-1B a basis averaged over ten thousand steps still moves by 84% of maximal per refresh (against 97% with no averaging), and final perplexity is essentially flat across four decades of averaging window (a 0.36 perplexity span). What works is moving the state, not the frame. The first moment transports exactly ⊤ through the rotation R = Unew Uold ; the second moment transports through the squared entries of R, and a carry that ignores R is provably a factor ≈(r−k ⋆ )/2 worse than the best blind alternative (§5, Appendix B). In a three-seed controlled comparison at 1B, transported state (the LDAdam update, which combines transport with error feedback) at the default β2 reaches 18.73 perplexity, better than untransported GaLore after its best β2 fix (19.28). The cheaper remedy, shortening the memory to β2 =0.99, helps the refreshing optimizers we test at their matched recipes, from 1.8 to 26 perplexity depending on optimizer and recipe (both the size and, for canonical GaLore, the sign of the effect are recipe-dependent), while a full-rank control with no refreshes reverses the preference, isolating the redraw as the cause. Contributions. control showing that the GaLore subspace’s apparent • The object is not there. A same-step √ per-refresh rotation (∥∆U ∥F ≈ 2r, saturated 96.7–99.6% across ranks and scales) equals the disagreement between two simultaneous estimates: the subspace is statistically nonidentifiable beyond a small reproducible core of k ⋆ ≈ 39 of 128 directions (§3). The same holds across four model families (three architecture classes), from 70M to 6.9B parameters, and in a vision transformer, strengthening with scale (§3.6). • Why, and why averaging cannot fix it. The gradient spectrum is gapless at r, and its deep tail is power-law signal that shrinks under N -fold averaging only as ≈N −1/4 ; k ⋆ is predictable from the spectrum (within 8% of the direct count, with the per-layer ordering matching measured energy retention), and no averaging budget opens a gap (§3). At 1B, ten-thousand-step basis averaging leaves both the realized rotation and the final perplexity essentially unchanged (§5). • What to do instead, with theory and a controlled test. The optimizer state should be transported through each refresh: the first moment by R, the second moment by the squared entries of R, with blind carry provably ≈(r−k ⋆ )/2 suboptimal (Appendix B). A three-seed 2
Published as a conference paper at ICOMP 2026
2×2 at 1B/40K shows transport at the default β2 beats the β2 fix alone (18.73 vs 19.28), and the two remedies are partially independent (§5). • A prescription and a probe. Lower β2 to 0.99 for any optimizer that refreshes a gradient subspace (validated for canonical GaLore and LDAdam, with APOLLO-Mini as a directional consistency check and a full-rank control reversing the preference), and measure k ⋆ before trusting a rank: if r > k ⋆ , the excess directions are noise being refreshed (§5). The object of study is the gradient subspace these methods compute, not a LoRA-style weight parameterization (§2).
2
S ETUP AND NOTATION
The GaLore-family optimizer. We study optimizers that train a weight matrix W ∈ Rm×n by running Adam inside a periodically-recomputed low-rank subspace of its gradient. At step t the gradient is Gt = ∇W L ∈ Rm×n . Every T steps the optimizer computes the top-r left singular subspace of the current gradient, Ut ∈ Rm×r with Ut⊤ Ut = Ir (the columns are the leading left singular vectors of Gt ), and holds it fixed for the next T steps. Within an interval it projects each gradient into the subspace, G̃ = U ⊤ G ∈ Rr×n , applies the Adam update in that r-dimensional space, and maps the result back: W ← W − η U Adam(G̃). This is GaLore (Zhao et al., 2024); the same template, explicitly form a top-r gradient subspace, refresh it every T steps, underlies the family that has grown around it (Robert et al., 2025; Rajabi et al., 2025b; Zmushko et al., 2024; He et al., 2024; Zhu et al., 2024). Our claims are restricted to this family. LoRA and other low-rank parameterizations, which impose a low-rank structure on the update and never compute a gradient subspace, are out of scope. Adam’s two clocks. Adam (Kingma & Ba, 2015) carries two exponential moving averages of the projected gradient: the first moment mt = β1 mt−1 + (1−β1 )g̃t and the second moment vt = β2 vt−1 + (1−β2 )g̃t2 . Each is a low-pass filter with a relaxation time set by its decay, τm =
1 ≈ 10, 1 − β1
τv =
1 ≈ 1000 1 − β2
(β1 =0.9, β2 =0.999),
(1)
so the second moment forgets the past 100× more slowly than the first. These two timescales are the constants against which the refresh interval T must be compared, and the asymmetry between them is the source of the staleness we analyze in §5. Measuring subspace rotation. To quantify how much the basis changes at a refresh we compare the bases before and after, Uold and Unew ∈ Rm×r , through their r principal angles θ1 , . . . , θr (the angles between range(Uold ) and range(Unew )). Singular vectors carry an arbitrary per-column sign, so the raw Frobenius distance ∥Unew − Uold ∥F is dominated by sign flips rather than rotation; we align signs first (Appendix A) and report the sign-corrected distance 1/2 √ Pr ∥∆U ∥F = 2r − 2 i=1 cos θi , 0 ≤ ∥∆U ∥F ≤ 2r. (2) This is the chordal (principal-angle) Frobenius distance between orthonormal √ frames (Bendokat et al., 2024): it is 0 when the subspace is unchanged and attains its maximum 2r when the two bases are orthogonal (θi = π/2 for all i), i.e. when the new subspace shares no direction with the old. We also track the fraction of gradient energy the subspace captures, γg =
∥U U ⊤ G∥2F ∈ [0, 1], ∥G∥2F
(3)
which is what the projection actually preserves and the quantity we return to in §4.
3
T HE SUBSPACE IS NOT IDENTIFIABLE
The subspace GaLore projects onto is recomputed every T steps, and it moves. This section establishes what that motion is. The measured per-refresh basis change saturates the geometric maximum at 3
Published as a conference paper at ICOMP 2026
rotation per refresh
U F
Each refresh redraws a near-orthogonal frame 2r (orthogonal-frame maximum) measured U F per refresh
20 15
0.977
0.967
0.996
10 8
0.991
32
64
rank r
128
256
Figure 1: Measured √ subspace rotation per refresh against rank (Pythia-1B, T =50; log–log). The points sit on the 2r ceiling, the maximum possible Frobenius distance between two rank-r frames, at every rank (annotated ratios 0.967–0.996). Each refresh returns a nearly independent subspace. Exact values and the sign correction: Appendix A. every rank and scale (§3.1). A same-step control then locates the cause: two top-r estimates taken at the identical step from disjoint minibatches already disagree by as much as two estimates taken T steps apart (§3.2). The refresh is therefore not the motion of a stable object. The top-r subspace of a minibatch gradient is statistically non-identifiable beyond a small reproducible core, at the ranks these methods use. The rest of the section shows why: the gradient spectrum has no gap at r (§3.3); the identifiable part is small but measurable and consistent (§3.4); and averaging does not manufacture identifiability (§3.5). 3.1
A SCALE - INVARIANT ROTATION LAW
We measure ∥∆U ∥F at every refresh during training and find that the subspace rotates almost as far as it geometrically can. Figure 1 reports the measured rotation across an 8× range of ranks at Pythia-1B (exact values in Appendix A): in every case ∥∆U ∥F reaches 96.7–99.6% of its maximum √ 2r, the value√attained when the new basis is orthogonal to the old. The dependence on rank is ∥∆U √ ∥F ≈ 2r: for instance ∥∆U ∥F (r=256)/∥∆U ∥F (r=32) = 22.10/7.93 = 2.79, close to 8 = 2.83. The same holds from Pythia-160M to 1B. At this saturation the mean cosine, 1 − ∥∆U ∥2F /2r, is below 0.07: at each refresh the optimizer discards the frame it was using and ◦ adopts a nearly orthogonal (On a pretrained 160M checkpoint the same √one (mean angle 86–90 ). redraw sits lower, at 0.74 2r, a mean angle of ≈63◦ : a pretrained gradient’s reproducible spike is stronger, and its k ⋆ ≈ 39 shared directions pull consecutive estimates together; §3.2.) This is not slow drift. It is turnover. 3.2
T HE ROTATION IS ESTIMATOR NOISE
A near-maximal ∥∆U ∥F across a refresh has an obvious reading: the subspace genuinely rotates by a large angle in T steps. We test that reading with a control that removes time. At a single step we split the batch into two disjoint halves, compute the top-r subspace of each half’s gradient, and measure the distance between the two (protocol in Appendix F). Nothing has changed between the two estimates except the sample. If the top-r subspace were a well-defined property of the gradient, the two same-step estimates would agree. They do not. Averaged √ over four layers at Pythia-160M with r=128, the same-step split-batch disagreement is 0.725 2r.√The across-time rotation over a full refresh interval of training, at the same 160M scale, is 0.742 2r (Figure 3a), essentially the √ same-step value: the across-time rotation exceeds the same-step floor by at most a few percent of 2r. That floor is not an artifact of the 4
Published as a conference paper at ICOMP 2026
pretrained checkpoint at which the control is run: repeated at five points along Pythia-160M training √ (steps 1K to 143K), the four-layer same-step disagreement stays in 0.64–0.71 2r (Appendix F), lowest early and rising toward the pretrained value, so at every training state the bulk of the acrosstime rotation is already present with no time elapsed. The distance a subspace travels in T steps is essentially the distance two estimates of it already sit apart at a single step, so the “rotation” carries little information about elapsed time: it is dominated by estimator noise, not by motion of a stable object. A direct count confirms the same fact at the level of individual directions: averaged over the four layers, only k ⋆ ≈ 39 of the r=128 directions the top-r cut returns are reproducible across the split (48 on the query-key-value matrix, with standard error below 1 over repetitions), and the rest reshuffle with the sample. This also settles an apparent conflict with reports that the GaLore subspace is stable. Those reports measure quantities that stay small while ∥∆U ∥F is near-maximal. The rank-r subspace has r principal angles, and many can remain small while the tail angles near 90◦ dominate the chordal distance, so an angle distribution that looks mostly stable, without conversion to ∥∆U ∥F , still corresponds to a near-maximal rotation (Zmushko et al., 2024). An incremental tracker is built to move its estimate slowly, so its self-consistency across steps is a property of the tracker, not of the subspace a fresh SVD would return (Rajabi et al., 2025b). And a raw singular-vector overlap conflates the arbitrary per-column sign of the SVD with genuine rotation; once signs are aligned (§2), the rotation is the near-maximal value of Figure 1. None of these is the sign-corrected frame distance ∥∆U ∥F , which is the quantity that governs the carried optimizer state. 3.3
W HY: THERE IS NO SPECTRAL GAP AT RANK r
The recomputed top-r subspace can be reproducible across samples only if the rank-r cut separates well-determined directions from the rest, that is, if the spectrum has a gap at index r. There is not. Figure 2 shows the per-matrix gradient spectrum at Pythia-160M and 1B: a small dominant spike, then a smooth, near-isotropic tail that the rank-128 cutoff slices straight through. Across layers, steps, and both scales: • the dominant spike is small: a few to ∼20 directions hold the first half of the gradient’s squared singular-value (Frobenius) mass, consistent with the tiny gradient subspace documented for classification by Gur-Ari et al. (2018); • there is no gap at the cutoff: the adjacent singular-value ratio at r=128 is ≈ 1.005 on average, and no jump anywhere in the bulk exceeds ∼3% (max ratio 1.011–1.030); • the effective rank (entropy of the normalized spectrum) straddles the cutoff, ranging 89–358 across layers and both scales, so r=128 sits inside the flat tail, not at a boundary; • the top-r block captures only 46–68% of the gradient’s singular-value (nuclear) mass, leaving roughly a third to over half outside the subspace at all times. (The spike dominates the squared mass while the long flat tail dominates the nuclear mass; the contrast between the two norms is itself a signature of the tail’s flatness.) This agrees with independent measurements: Gur-Ari et al. (2018) establish the small dominant spike directly, and a concurrent large-model analysis reports the same spike-tail profile with the spike at ∼1.5% of directions (Huang et al., 2026), a separation at the spike rather than at r=128. The complementary evidence from the randomization side, that past the spike the SVD frame carries little more than a random one, is taken up in §4 and §6. From no gap to Haar-random redraw. When singular directions are near-degenerate they are maximally sensitive to perturbation: the Davis–Kahan and Wedin sin θ theorems bound a subspace’s rotation by the perturbation size over the gap that isolates it (Davis & Kahan, 1970; Wedin, 1972). As the gap at r vanishes, that bound diverges, and the SVD is free to return any frame within the degenerate block. Sub-threshold directions therefore behave as a Haar-random draw within the tail, whether the perturbation is a step of training or a change of minibatch. This is why two same-step estimates disagree as much √ as two across-time estimates (§3.2). The same model predicts the redraw distance approaches the 2r ceiling as r/d → 0√(Appendix A). At 160M, where r/d = 1/6, it stays below the ceiling, matching the measured 0.74 2r. The Weingarten lower bound of §5 follows for the carried second moment under the same Haar model (Theorem 1, Appendix B). The stable part of the geometry is real but small: the spike of Gur-Ari et al. (2018) is genuine, and GaLore’s r=128 5
Published as a conference paper at ICOMP 2026
Per-matrix gradient spectrum: small spike, then smooth gapless tail (no step at r = 128) Pythia-160M
100
r = 128
10 1 i/ 1
Pythia-1B r = 128
10 2 10 3 embedding (eff. rank 221) attention QKV (eff. rank 89) attention output (eff. rank 135) MLP (eff. rank 120)
10 4 10 5 0
200 400 600 singular value index i
embedding (eff. rank 358) attention QKV (eff. rank 130) attention output (eff. rank 172) MLP (eff. rank 305)
800
0
500 1000 1500 singular value index i
2000
Figure 2: Per-matrix gradient singular spectrum at Pythia-160M (left) and 1B (right): the reason the subspace is not identifiable. A small dominant spike (a few to ∼20 directions holding half the squared mass) gives way to a smooth, near-isotropic tail with no gap. The GaLore cutoff r=128 (dashed) falls inside the flat region, where the adjacent singular-value ratio is ≈ 1.005 and the top-r block holds only 46–68% of the gradient’s nuclear mass. The rank-r subspace is not separated from its complement. overshoots it by an order of magnitude, reaching past the determined directions into the gapless bulk that redraws. 3.4
T HE IDENTIFIABLE PART IS SMALL , MEASURABLE , AND CONSISTENT
The same-step count gives a direct measure of how many directions survive resampling: a four-layer mean of k ⋆ ≈ 39 of 128, ranging from 48 on the query-key-value matrix down to 31 on the attention output, with a standard error below 1 over repetitions, so the count is stable rather than noise. Reading the same data a second way reproduces the pattern: inverting the measured spectral shape through a detectability threshold fixed a priori, with one calibrated noise scale per layer and no shape parameters, recovers the same layer ordering, qkv > mlp > embed > dense (inversion 42, 37, 19, 17 against the direct 48, 43, 34, 31). Both readings derive from the same measurement, so this is a consistency check between two estimators rather than an independent prediction; both place the reproducible core well below r=128. The ordering matches the measured energy the top-r block retains in each layer (73–81% for qkv and mlp versus 42–48% for embed and dense): layers whose gradient concentrates more have more reproducible directions. k ⋆ is thus a measurable property, not an artifact of one estimator. 3.5
AVERAGING DOES NOT RESCUE IT
Identifiability might be recovered by averaging: with a large enough gradient sample the noisy tail would shrink and a gap could open at r. It does not shrink fast enough. Under N √-fold gradient averaging, pure sampling noise falls as N −1/2 , so a noise singular value scaled by N would stay flat. The deep spectral tail instead shrinks only as ≈ N −1/4 : across all four probed layers the scaled tail value grows by 2.0–3.2× from N =1 to N =64 (Figure 3). The tail is not noise being averaged away. It is signal, with a de-censored power-law spectrum of exponent α = 1.03–1.35, obtained two independent ways (a direct tail fit and the overlap-versus-N curve). A shuffled-document control, which breaks within-document correlation, retains most of the anomaly (1.7–2.2× against the 2.0– 3.2× of the true pool), so the persistent tail is mostly genuine gradient signal, with only a minor part attributable to within-document correlation. We state the limits of this claim plainly. We do not claim a closed-form law for how the subspace overlap grows with N : the naive spiked-covariance prediction fails out of sample (mean absolute error 0.073), so we report only the measured scaling, not a formula. That scaling is enough for the conclusion. Because the tail is signal that decays as a power law and shrinks under averaging only as 6
Published as a conference paper at ICOMP 2026
(a) two same-step estimates disagree as much as two time-separated ones
(b) the deep spectral tail is signal, not noise being averaged away 3 N (normalized)
across-time rotation 0.742
0.75 0.70 0.65 0.60
deep-tail s500
same-step disagreement
U F / 2r
0.80
0.55 embed attn qkv
0.50 0.45
2
0
2
1
attn dense mlp
2
2
3
2 2 averaging fold N
4
2
5
2
6
embed attn qkv
attn dense mlp
signal s N 1/4
2 1.5
1 20
21
22
23 24 averaging fold N
pure noise s N 1/2
25
26
Figure 3: Averaging cannot make the top-r subspace identifiable (Pythia-160M, r=128, four layers). √ (a) Same-step split-batch subspace disagreement ∥∆U ∥F / 2r against averaging fold N . At N =1 it is 0.725, indistinguishable from the across-time √ rotation 0.742 (dashed), and it falls only slowly with N . (b) The deep tail singular value scaled by N . Pure noise would be flat; the measured tail grows 2.0–3.2× from N =1 to 64, so it shrinks as ≈ N −1/4 , not N −1/2 . The tail is signal with power-law exponent α = 1.03–1.35, so no averaging budget opens a gap at r. N −1/4 , no finite averaging budget opens a gap at r, and no rank r lands on a gap that is not there. The top-r subspace GaLore uses is not well-defined, and no averaging budget we test recovers it; this holds from 160M to 2.8B and across GPU backends (Appendix F). 3.6
T HE NON - IDENTIFIABILITY IS GENERAL
The same-step control is measured at Pythia-160M, but the effect is not specific to Pythia, to language, or to GaLore. We ran the same split-batch probe on eleven pretrained checkpoints, all trained with fullrank Adam: seven Pythia scales (70M–6.9B), GPT-2-large, Qwen2.5-3B, Llama-3-8B, and a vision transformer (ViT-B) probed√on CIFAR-10. In every language model the same-step disagreement is 0.67–0.83 of the maximum 2r, with no spectral gap at r (mean adjacent ratio 1.004–1.009); it rises with scale (Figure 4) as the reproducible core shrinks, from k ⋆ ≈ 50 at 70M to k ⋆ ≈ 21 at 6.9B. The effect crosses architecture classes (GPT-NeoX, GPT-2, and the Llama-class Qwen2.5 and Llama-3) and holds, more weakly, in the vision transformer (0.53 of maximal on CIFAR). Because every model here is trained with full-rank Adam, the non-identifiability is a property of the gradient itself, not of the low-rank projection: the top-r subspace any method would extract is estimator noise past a small core, at every scale we measured. Per-model numbers and the protocol are in Appendix G. Forward pointer. This scaling is measured at Pythia-160M, where the tail is already flat at r=128. At 1B, where r=128 lies deeper in the tail of a wider gradient, it predicts that averaging successive bases across refreshes cannot stabilize the subspace either. We verify this directly, after the energy interlude of §4, with the second-moment remedies at 1B in §5: basis averaging leaves the per-refresh rotation near its maximum, while the two remedies that do help act on the carried optimizer state rather than on the subspace.
4
W HY G A L ORE STILL WORKS
If the subspace it projects onto is replaced by a nearly orthogonal one every T steps, why does GaLore train as well as it does? The answer is that the projection’s job is not to track a fixed subspace but to capture gradient energy, and how much it captures barely depends on which subspace it picks. The descent contribution of a projection is governed by the fraction of gradient energy it retains, γg (Eq. 3). For a subspace U spanning r directions of the gradient second-moment, the expected coverage is the cumulative spectral mass it covers, Pr λi , (4) E[γg ] = Pi=1 n i=1 λi 7
Published as a conference paper at ICOMP 2026
U F / 2r (N=1)
0.9
same-step disagreement
Non-identifiability across architectures, scales, and domains
0.6
near-orthogonal redraw
0.8 0.7
0.5
Pythia (GPT-NeoX), 70M-6.9B GPT-2-large Qwen2.5-3B Llama-3-8B ViT-B on CIFAR (vision)
0.4 0.3
0.1
0.3 1 3 parameters (billions, log scale)
10
√ Figure 4: Same-step subspace disagreement ∥∆U ∥F / 2r at N =1 (split-batch, pretrained checkpoints, r=128, mean over four layer types) across architectures, scales, and domains. It rises with scale in Pythia (70M–6.9B), sits in the same band for GPT-2-large, Qwen2.5-3B, and Llama-3-8B, and holds more weakly for a vision transformer on CIFAR. Every model is full-rank-Adam-pretrained, so the non-identifiability is a property of the gradient, not of GaLore; no spectral gap at r appears in any of them.
where λ1 ≥ · · · ≥ λn are the spectrum’s energies (Appendix D). This identity is an expectation over the gradient covariance; the per-matrix SVD energy fractions reported below are the instantaneous quantity it matches in expectation. In the presence of a sharp gap, only the gap-defined subspace captures the dominant mass and a rotation away from it would be costly. But §3.3 showed the tail past the small spike is smooth and globally near-isotropic, with no gap anywhere in the bulk rather than merely at r; it is this global flatness, not the absence of a single gap at r, that the argument needs. In that regime Eq. 4 is nearly flat in the choice of subspace: any r directions that include the spike and otherwise sample the tail capture a comparable fraction. The freshly-recomputed basis, near-orthogonal to its predecessor, is therefore an essentially equivalent projector: measured during 1B training with r=128, GaLore retains a roughly constant γg ≈ 0.67–0.73 of the squared gradient throughout, regardless of the redraw. (The nuclear fraction is lower, 46–68%, because the flat tail carries more of the nuclear mass; §3.3.) He et al. (2024) reach a compatible conclusion from the convergence side: once the true gradient shrinks relative to its noise, the SVD-selected subspace loses its advantage over a random one, and a projection chosen “without any preference” suffices in its place. Our diagnostic also reconciles a tension in more recent GaLore-family work. He et al. (2024) and a randomized Grassmann variant (Rajabi et al., 2025a) replace the exact SVD projection with a random one and report training at least as good as GaLore, the latter arguing that flat curvature makes the precise subspace unnecessary; GaLore-2 (Su et al., 2025) instead finds that a purely random projection degrades quality and retains a periodic SVD in a hybrid schedule. The reproducible rank k ⋆ reconciles them: the exact SVD earns its cost only to the extent the subspace is identifiable, and is redundant once k ⋆ ≪ r. Because the subspace grows less identifiable as training proceeds (the same-step floor rises across the trajectory; Appendix F), a random projection is safest late in training, while any residual advantage of the exact SVD is concentrated early. This is a quantitative form of the flat-curvature intuition, and of the boundary the hybrid schedule approximates. Low-rank projection is known to introduce a per-step bias relative to full Adam (He et al., 2024; Pan et al., 2025); our account is orthogonal, since we do not claim the projection is unbiased, only that the choice of which r directions it selects barely changes the energy captured. This reframes what GaLore is. It is not tracking a persistent low-rank structure in the gradient; there is none to track at rank r. It is a rotating energy-capture projector: a frame that turns over completely at every refresh while preserving the energy fraction that drives descent. The descent direction survives the redraw. What does not survive is the optimizer state carried across the rotation, and, as we show next, the second moment in particular pays for it. 8
Published as a conference paper at ICOMP 2026
5
W HAT WORKS AND WHAT DOES NOT
If the rank-r frame that GaLore projects onto is redrawn nearly orthogonally at every refresh (§3), the optimizer state carried across that refresh points at a target that no longer exists. The first moment ⊤ survives the move: it is a vector in the subspace and transports exactly by the rotation R = Unew Uold (Proposition 3), the change of basis that LDAdam (Robert et al., 2025) and SUMO (Refael et al., 2025) apply. The second moment does not. It is a per-coordinate variance whose optimal new-basis value passes through the squared entries of R, and a carry blind to R sits a factor ≈ (r−k ⋆ )/2 above the best any blind estimator can reach (Theorem 1). With memory τv = 1/(1 − β2 ) ≈ 1000 steps at the default and refreshes every T ≈ 50–200 steps, the second moment averages statistics from many dead subspaces and is chronically misaligned; its lag grows with τv on the same curve as full-rank Adam (Appendix H). Three remedies present themselves: average the basis so it stops moving, transport the state through the rotation, or shorten the second moment’s memory. We test all three at Pythia-1B. Averaging fails; transport and a shorter memory both work, and we measure why. 5.1
AVERAGING THE BASIS DOES NOT WORK
The direct fix for a subspace that moves too much is to stop it from moving. We build the EMAbasis variant: instead of the top-r singular subspace of the current gradient, the basis is the top-r eigenvectors of an exponential moving average of GG⊤ with decay β3 , so that β3 =0 recovers GaLore exactly and larger β3 averages the subspace over a growing window. At Pythia-1B trained from scratch on WikiText (Merity et al., 2016) (r=128, T =160, learning rate 4×10−3 , warmup 4000, 40K steps, AMD MI210, one seed), the final perplexity does not move with β3 . At the default β2 =0.999 the row is flat: {21.088, 21.025, 21.074, 21.026, 21.385} for β3 = {0, 0.9, 0.99, 0.999, 0.9999}, spanning 0.36 across four decades of averaging window (0.06 excluding the longest, 104 -step window, which alone sits 0.36 above the minimum). The flat row replicates on a second cluster (NVIDIA L40S, same recipe): 20.39 at β3 =0 against 20.52 at β3 =0.99 (we compare only within a cluster, never across). The measurement of why is direct. Averaging the covariance does slow the basis, but not enough to matter: late in training (steps ≥ 20K) the mean√ realized rotation is ∥∆U ∥F = 15.48, 14.51, 13.42 for β3 = 0, 0.999, 0.9999, against the ceiling 2r = 16.0. Even a 10,000-step average of GG⊤ leaves the refresh-to-refresh basis change at 84% of maximal (against 97% with no averaging). Basis averaging cannot stabilize what is not identifiable: past the dominant spike the spectrum is gapless (§3.3), so no amount of temporal averaging carves a stable rank-128 frame out of the isotropic tail. 5.2
T RANSPORTING THE STATE WORKS
If the basis will not hold still, move the state with it. Table 1 crosses the two arms that matter, {carry, transport} against {β2 =0.999, β2 =0.99}, three seeds per cell at Pythia-1B/40K. This table uses warmup 1440; its carry-β2 =0.999 cell (22.07) therefore sits about 0.8–1.0 PPL above the same nominal configuration in the averaging and gasket experiments (warmup 4000: 21.09 at one seed, §5.1, and 21.27 over three seeds, §5.3), purely through the warmup difference. Magnitudes shift between recipes; directions do not. We compare β2 and state policies only within a fixed recipe, never perplexities across recipes. Read the table three ways, in order. First, transporting the state at the default β2 =0.999 reaches 18.73, better than lowering β2 without transport (19.28): moving the state is worth more than the hyperparameter fix on its own. Second, the second-moment cost of the wrong β2 under a plain carry is large, 22.07 against 19.28: a 2.79 PPL gap, about twenty times the standard error of the difference (two-sample t ≈ 19 on 4 degrees of freedom). Third, that gap does not vanish once the state is transported: it shrinks to 1.81 but persists; the two remedies are therefore partially independent rather than substitutes. (The transport row is the LDAdam update, so this 1.81 is the same measurement that recurs below as LDAdam’s matched-40K β2 gap, §5.3.) One caveat is mandatory: the transport arm is the full LDAdam update, which applies the change-of-basis transport of both moments together with error feedback, and this experiment does not separate the contribution of transport from that of error feedback. The transport rule itself is LDAdam’s; our contribution here is the optimality statement behind it (Proposition 3, Theorem 1) and the controlled 2 × 2 comparison, not the rule. 9
Published as a conference paper at ICOMP 2026
Table 1: Final validation perplexity at Pythia-1B under the two independent remedies: transporting the optimizer state across each refresh and shortening the second moment’s memory. Rows are the state policy at a refresh; columns are β2 . Each cell is mean±standard deviation over three seeds. Recipe (identical across cells): Pythia-1B trained from scratch on WikiText, rank r=128, refresh interval T =160, learning rate 4 × 10−3 , warmup 1440, 40K steps. “Carry” is canonical GaLore (the second moment is carried unchanged across refreshes); “transport” is the LDAdam update (Robert et al., 2025) (change-of-basis transport of both moments plus error feedback). Transporting at the default β2 =0.999 (18.73) already beats lowering β2 alone (19.28); the carry β2 gap is 2.79, about twenty times the standard error of the difference, and persists under transport (1.81). Transport here is the full LDAdam update; the table does not separate state transport from error feedback. State at refresh Carry (canonical GaLore) Transport (LDAdam rule)
5.3
β2 =0.999 (default)
β2 =0.99
22.07 ± 0.17 18.73 ± 0.13
19.28 ± 0.18 16.92 ± 0.16
A SHORTER MEMORY WORKS , WITH A RECIPE CAVEAT
The second remedy, lowering β2 from 0.999 to 0.99, is the cheaper of the two, and it helps the refreshing optimizers we test at their matched recipes. For canonical GaLore the effect is small (about 2 perplexity) and hardware-sensitive: it is three-seed robust on our primary cluster, while a matched-recipe run on a different GPU backend does not reproduce it, consistent with the projection’s periodic SVD being ill-conditioned and resolving differently across backends. We therefore present the shorter memory as the secondary, less robust remedy, behind state transport. The forgetting-factor reason is standard (Guo et al., 1993): when the target is reset every T ≪ τv steps, a long memory keeps integrating pre-rotation statistics that are now bias, and the error-minimizing forgetting factor drops below the no-reset optimum. A bias–variance argument specialized to a self-reset target makes the direction precise (Appendix I): the minimizer is interior and moves down as the refresh interval shrinks. Empirically the β2 gain does not depend on the averaging window of the previous experiment: across the β3 grid the gap between β2 =0.999 and β2 =0.99 is 1.94, 2.04, 2.16, 2.27, 2.13 PPL at β3 = 0, 0.9, 0.99, 0.999, 0.9999 (seed 0), so averaging neither substitutes for the memory fix nor interferes with it. At the gasket recipe repeated over seeds (same recipe, warmup 4000), canonical GaLore gives 21.27 ± 0.17 at β2 =0.999 against 19.04 ± 0.13 at β2 =0.99 (three seeds each), a 2.23 PPL gap. Across the family. The direction holds for every optimizer we tried (Figure 5; exact values and configurations in Appendix E, Table 3). For LDAdam, whose error feedback makes the second moment’s calibration load-bearing, lowering β2 from 0.999 to 0.99 improves perplexity from 57.69 ± 0.11 to 31.27 ± 0.31 over three seeds (10K steps, learning rate 10−3 ): a 26.4 PPL gap against per-cell seed spreads of 0.1–0.3, two orders of magnitude larger than the noise. The length caveat below is specific to canonical GaLore’s small margin; the LDAdam gap is present at both horizons, 26.4 at its own 10K recipe and 1.81 at the common 40K recipe (Table 1), so it is not a short-run artifact. We do not rank optimizers by gap size: at the common recipe LDAdam’s gap (1.81) is smaller than canonical GaLore’s (2.79), and the 26.4 belongs to its own recipe only. APOLLO-Mini moves from 97.83/101.77 at β2 =0.999 to 96.76/97.20 at β2 =0.99 (two seeds); it resamples a random Gaussian projection every T rather than an SVD top-r subspace, so two consecutive frames are near-orthogonal by construction and the refresh geometry differs. We read APOLLO as a directional consistency check only, not a third strong leg: two seeds, small margins, high absolute perplexity. The direction is nonetheless the same. A length caveat, stated plainly. For canonical GaLore the margin is training-length dependent. It is present at 40K (above) but absent, even slightly reversed, in shorter 10–20K runs: the staleness cost accrues over rotations, and the second-moment bias-correction transient occupies a larger fraction of a short run. The effect is real but integrates with training; short-horizon tuning understates it. The causal control. The check that ties the β2 effect to the refresh, and not to these training runs in general, is full-rank AdamW: the same model, data, and schedule with no subspace refresh. There 10
Published as a conference paper at ICOMP 2026
Lowering 2 from 0.999 to 0.99 helps each refreshing optimizer shown LDAdam (10k) Canonical GaLore (40k)
31.27
19.04
57.69
21.27
APOLLO-Mini (2 seeds)
97
Full-rank AdamW control
97.9
99.8
118
20
30 50 100 final validation perplexity (log scale)
2 = 0.999 (default)
2 = 0.99: refreshing, better
200
2 = 0.99: control, reverses
Figure 5: Lowering β2 from the 0.999 default to 0.99 at Pythia-1B (final validation perplexity, log scale). The axis is logarithmic and each optimizer sits at its own training recipe (steps, learning rate, and length differ across the three, and the full-rank control); the comparison is within-optimizer, where each β2 pair shares its recipe, not across optimizers. For each refreshing optimizer shown, the lower value is better: LDAdam by 57.69 → 31.27 at its own 10K recipe (three seeds; at the common 40K recipe its gap is 1.81, Table 1), canonical GaLore by 21.27 → 19.04 (three seeds), APOLLOMini directionally (two seeds). Gap sizes are not comparable across optimizers because recipes differ. The full-rank AdamW control, which runs with no subspace refresh, reverses the preference back to 0.999, directionally isolating the periodic redraw as the cause. The canonical-GaLore advantage is flat across the EMA-basis averaging window (β2 gap 1.94–2.27 PPL over β3 ∈ {0, . . . , 0.9999}, §5.1), so a shorter memory and basis averaging act independently. Exact values, seed counts, and recipe: Appendix E, Table 3. the preference reverses to the standard β2 =0.999 at both learning rates tested (97.9 against 118.0 at learning rate 6 × 10−4 , and 158.3 against 200.9 at 10−3 ; 14.4K steps, one seed), exactly as the forgetting-factor argument predicts when the target is never reset. The control is directional (one seed, two learning rates, shorter horizon) but the margins are far outside seed noise, and its absolute perplexities are not comparable to the 40K GaLore runs above. The lower-β2 advantage is therefore specific to subspace refreshing. A second moment that a refresh periodically invalidates is what makes 0.999 the wrong default. 5.4
P RESCRIPTION
The geometry gives a recipe. Measure how many gradient directions your layer actually resolves, the small determined spike of §3.3 rather than the nominal rank; call it k ⋆ . If the projection rank r exceeds k ⋆ , the directions beyond k ⋆ sit in the gapless tail and are re-drawn at each refresh, so the state carried across the refresh goes stale. Then do two things. Transport the optimizer state through the rotation: the first moment by R (Proposition 3), the second moment by the symmetric-square rule of Theorem 1. And lower β2 toward 0.99. Both help, and their gains are partly independent (Table 1), so applying both is worth more than either alone. Do not expect basis averaging to recover the difference: when the rank sits in the gapless tail there is no stable subspace to average toward (§3.3), and four decades of averaging window leave both the perplexity and the realized rotation unchanged (§5.1).
6
R ELATED WORK
The GaLore family and its premise. GaLore (Zhao et al., 2024) and its refinements share one move: run an optimizer inside an explicitly recomputed top-r gradient subspace. The variations are in what they carry across the refresh: LDAdam transports both moments and adds error feedback (Robert et al., 2025), SubTrack++ tracks the subspace incrementally (Rajabi et al., 2025b), FRUGAL keeps a reduced state (Zmushko et al., 2024), and GoLore (He et al., 2024) and APOLLO (Zhu et al., 2024) alter the projection itself, while Q-GaLore quantizes it and adapts the refresh rate per layer (Zhang 11
Published as a conference paper at ICOMP 2026
et al., 2024b). Memory-efficient optimizers outside the projection family instead compress the second moment directly, by low-rank factorization (Shazeer & Stern, 2018) or block-wise learning-rate sharing (Zhang et al., 2024a). The shared, usually unstated, premise is that this subspace is a meaningful object that persists across refreshes. Our contribution is orthogonal to these methods: we do not propose another member of the family but measure the premise they rest on, find it false at the ranks they use, and show both why they nonetheless succeed (energy capture, §4) and what the redraw costs them (second-moment staleness, §5). Robert et al. (2025) note from the theory side that GaLore requires a “strong stable-rank assumption” but do not measure it; we measure it. LDAdam also supplies the state-transport rule we build on: it transports both moments by the change of basis and adds error feedback. We do not prove that its second-moment rule is optimal; we prove that first-moment transport is the MMSE map and that a rotation-blind second-moment carry is a factor ≈ (r−k ⋆ )/2 suboptimal (Theorem 1, Appendix B), a bound LDAdam escapes because it observes the rotation. That LDAdam already defaults to β2 =0.99 is, on our account, not a tuning accident but a symptom of exactly the staleness we characterize. What prior “stability” measurements measured. Reports that the GaLore subspace is stable rest on quantities that remain small while the sign-corrected chordal rotation is near-maximal: a distribution of principal angles (Zmushko et al., 2024), which leaves the dominant tail contribution to chordal distance unmeasured, and incremental-tracker self-consistency (Rajabi et al., 2025b), a property of the tracker rather than of a fresh SVD (§3.2). None measured ∥∆U ∥F , the quantity that determines what happens to carried state. Random and averaged subspaces. Two recent works reach compatible conclusions from the randomization side. Rajabi et al. (2025a) argue that the gradient subspace landscape has nearly flat curvature and that much of the gradient energy lies outside any core subspace, so random lowrank projections preserve the relevant geometry and match SVD-based selection; they exploit this to navigate the Grassmannian with random steps. Pan et al. (2025) independently diagnose that noise-dominated singular directions make the SVD projection unreliable, though they debias GaLore by probabilistically sampling full-rank layer updates rather than by randomizing the projection. Both locate an unreliability in the SVD-selected frame past the spike that we trace to the same-step non-identifiability of the top-r cut. A separate line averages the subspace rather than randomizing it: Gong et al. (2025) maintain an adaptively estimated low-dimensional subspace instead of a fresh SVD at each refresh, closest in spirit to the averaged-basis variant we test in §5.1. Our EMA-basis measurement is our own implementation at Pythia-1B with r=128 and speaks only to that setting; we make no claim about the performance reported for Alice. Gradient geometry and subspace perturbation. That optimization concentrates in a small gradient subspace is established (Gur-Ari et al., 2018), and gradient spectra serve directly as optimization diagnostics (Liu et al., 2026); we reconcile it (§3.3): the small determined subspace is the spike, and r overshoots it into the gapless tail. The link from a vanishing spectral gap to maximal subspace sensitivity is the Davis–Kahan and Wedin sin θ theory (Davis & Kahan, 1970; Wedin, 1972); the √ maximal-rotation value 2r is the diameter of the Stiefel manifold (Bendokat et al., 2024; Meckes, 2019). The Weingarten calculus on the orthogonal group (Collins & Matsumoto, 2009) enters separately, bounding the second-moment error that the rotation induces (§5). Forgetting factors and tracking. Treating β2 as a forgetting factor for a second moment that tracks an abruptly-reset target places our staleness result in the adaptive-filtering tradition: the bias–variance behavior of the forgetting-factor RLS algorithm under parameter jumps is classical (Guo et al., 1993), and our prediction β2⋆ < 0.999 under a periodically re-drawn subspace is its specialization to GaLore’s periodic refresh. The distinctive element here is that the “jumps” are not external nonstationarity but are imposed by the optimizer itself at every refresh. GaLore’s periodic SVD is likewise the batch counterpart to incremental subspace tracking (Oja’s rule (Oja, 1982), GROUSE (Balzano et al., 2010), PAST (Yang, 1995)); the same-step result bounds all of these, since a subspace two simultaneous batches cannot agree on is recoverable by no tracker. Unlike convergence analyses of low-rank optimizers (He et al., 2024; Pan et al., 2025), we characterize the identifiability of the projected subspace itself. 12
Published as a conference paper at ICOMP 2026
7
L IMITATIONS
Five boundaries of the evidence are worth stating plainly. First, the full identifiability apparatus (the four-layer same-step control and the N −1/4 averaging-budget test) is developed at Pythia-160M; the same-step disagreement and k ⋆ are additionally measured on pretrained checkpoints up to 6.9B (§3.6), and at 1B we further verify that basis averaging neither stabilizes the subspace nor improves perplexity. The N −1/4 averaging-budget sweep, developed at 160M, is confirmed at 410M, 1B, and 2.8B and reproduced across GPU backends (Appendix F); only the closed-form out-of-sample fit is reported solely at 160M. Second, the basis-averaging sweep is one seed per cell; we read it as a direction, supported by the measured rotation and a second-cluster replication, not as a headline number. Third, the transport arm of our 2×2 is the full LDAdam update, which combines state transport with error feedback; the comparison bounds their joint effect and does not separate them. Fourth, perplexity results are single-recipe per table (recipes stated in each caption; magnitudes shift between recipes, directions do not), on WikiText and Pythia only. Fifth, the canonical-GaLore β2 margin is training-length dependent: present at 40K steps, absent at 10–20K, so short-horizon tuning will not see it. Our claims are scoped to optimizers that explicitly compute and refresh a gradient subspace; LoRA and other parameterizations make no such assumption, and our measurements say nothing about them.
8
C ONCLUSION
The premise beneath GaLore-family optimizers is that there is a low-rank gradient subspace to track. We measured the premise. Beyond a reproducible core of roughly 39 of 128 directions, the object is not there: two estimates of the subspace taken at the same instant disagree as much as estimates taken a full refresh apart, the spectrum offers no gap for a rank cut to rest on, and the tail that a larger sample might average away turns out to be signal, shrinking four times too slowly for any budget to rescue. What looked like a subspace rotating is a subspace being re-drawn. Once that is said, the engineering sorts itself. Stabilizing the basis is chasing the phantom: at 1B, four decades of averaging window change neither the realized rotation nor the loss. The state, not the frame, is what must move: the first moment transports exactly through the rotation, the second through its squared entries, blind carry is provably a factor (r−k ⋆ )/2 worse, and the transported optimizer (the LDAdam update, transport plus error feedback) beats the best untransported configuration in a controlled three-seed comparison. The cheaper fix, shortening the second moment’s memory to β2 =0.99, helps the refreshing optimizers we tested at their reported recipes (small and recipesensitive for canonical GaLore) and reverses in a full-rank control, because a memory of a thousand steps is a liability precisely when the coordinates it averages over are re-drawn every 160. The takeaway we want a reader to carry out of this paper is the probe, not the numbers. Before building on a top-r gradient subspace, measure how much of it is real: split a batch, compute the subspace twice, and count the directions that agree. If your rank exceeds that count, the excess is noise that will be re-drawn at every refresh, and the right response is not a better tracker but transported state and a shorter memory. Low-rank training works, and it works for a reason; the reason is just not the one its geometry seemed to promise.
R EFERENCES Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for non-null complex sample covariance matrices. Annals of Probability, 33(5):1643–1697, 2005. Laura Balzano, Robert Nowak, and Benjamin Recht. Online identification and tracking of subspaces from highly incomplete information. In Allerton Conference on Communication, Control, and Computing, 2010. arXiv:1006.4046. Thomas Bendokat, Ralf Zimmermann, and P.-A. Absil. A Grassmann manifold handbook: Basic geometry and computational aspects. Advances in Computational Mathematics, 2024. arXiv:2011.13699. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, et al. Pythia: A suite for analyzing large language models across training and scaling. In ICML, 2023. 13
Published as a conference paper at ICOMP 2026
Benoı̂t Collins and Sho Matsumoto. On some properties of orthogonal Weingarten functions. Journal of Mathematical Physics, 50(11):113516, 2009. Chandler Davis and William M. Kahan. The rotation of eigenvectors by a perturbation. III. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970. Wenbo Gong, Meyer Scetbon, Chao Ma, and Edward Meeds. Towards efficient optimizer design for LLM via structured Fisher approximation with a low-rank extension. arXiv:2502.07752, 2025. arXiv:2502.07752. Lei Guo, Lennart Ljung, and Pierre Priouret. Performance analysis of the forgetting factor RLS algorithm. International Journal of Adaptive Control and Signal Processing, 7(6):525–537, 1993. Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv:1812.04754, 2018. Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv:2410.11289, 2024. Zhendong Huang, Hengjie Cao, Fang Dong, et al. Spectra: Rethinking optimizers for LLMs under spectral anisotropy. arXiv:2602.11185, 2026. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. Andy Zeyi Liu, Elliot Paquette, and John Sous. Spectral lens: Activation and gradient spectra as diagnostics of LLM optimization. arXiv:2605.05683, 2026. Elizabeth S. Meckes. The Random Matrix Theory of the Classical Compact Groups, volume 218 of Cambridge Tracts in Mathematics. Cambridge University Press, 2019. ISBN 9781108419529. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.07843, 2016. Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15(3):267–273, 1982. Rui Pan, Yang Luo, Yuxing Liu, Yang You, and Tong Zhang. Unbiased gradient low-rank projection. arXiv:2510.17802, 2025. arXiv:2510.17802. Sahar Rajabi, Nayeema Nonta, and Sirisha Rambhatla. Geometrically principled randomized optimization for efficient LLM training. arXiv:2510.01878, 2025a. arXiv:2510.01878. Sahar Rajabi, Nayeema Nonta, and Sirisha Rambhatla. SubTrack++: Gradient subspace tracking for scalable LLM training. arXiv:2502.01586, 2025b. Yehonathan Refael, Guy Smorodinsky, Tom Tirer, and Ofir Lindenbaum. SUMO: Subspace-aware moment-orthogonalization for accelerating memory-efficient LLM training. arXiv:2505.24749, 2025. Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. LDAdam: Adaptive optimization from low-dimensional gradient statistics. In ICLR, 2025. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv:1804.04235, 2018. DiJia Su, Andrew Gu, Jane Xu, Yuandong Tian, and Jiawei Zhao. GaLore 2: Large-scale LLM pre-training by gradient low-rank projection. arXiv:2504.20437, 2025. Per-Åke Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics, 12(1):99–111, 1972. Bin Yang. Projection approximation subspace tracking. IEEE Transactions on Signal Processing, 43 (1):95–107, 1995. 14
Published as a conference paper at ICOMP 2026
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv:2406.16793, 2024a. Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. Q-GaLore: Quantized GaLore with INT4 projection and layer-adaptive low-rank gradients. arXiv:2407.08296, 2024b. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. In ICML, 2024. Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, et al. APOLLO: SGD-like memory, AdamW-level performance. arXiv:2412.05270, 2024. Philip Zmushko, Aleksandr Beznosikov, Martin Takac, and Samuel Horvath. FRUGAL: Memoryefficient optimization by reducing state overhead for scalable training. arXiv:2411.07837, 2024.
A
T HE SIGN - CORRECTED ROTATION METRIC AND THE
√ 2r SCALING
Sign artifact in the raw distance. Singular vectors carry an arbitrary per-column sign: the i-th column of U may be negated without changing the subspace. Computed directly, ∥Unew − Uold ∥F is therefore dominated by sign inconsistencies rather than genuine rotation. For two m×r orthonormal matrices with independent random column signs, ⊤ E ∥Unew − Uold ∥2F = 2r − 2 E tr(Unew Uold ) ≈ 2r, ⊤ since the r diagonal entries √of R = Unew Uold have independent random signs and mean zero. The raw metric thus returns ≈ 2r regardless of T , a sign artifact rather than a measurement of rotation.
Corrected metric.PThe quantity we compute and report is the principal-angle (chordal) form ⊤ ∥∆U ∥F = (2r − 2 i cos θi )1/2 with cos θi = σi (Unew Uold ) the singular values of the cross-Gram matrix (Eq. 2), which is sign-invariant by construction and measures genuine rotation rather than the arbitrary column sign. (A per-column sign alignment si = sign(e⊤ i R ei ) followed by a Frobenius difference upper-bounds this quantity, with equality when the columns coincide with the principal vectors; we do not rely on that route.) We use ∥∆U ∥F for the singular-value form throughout.
Table 2: Subspace rotation per refresh, Pythia-1B, T =50, averaged over refresh events (the data √ plotted in Figure 1). ∥∆U ∥F reaches 96.7–99.6% of its frame maximum 2r at every rank; the implied mean angle is 86–90◦ . √ √ Rank r ∥∆U ∥F (measured) 2r (max) ∥∆U ∥F / 2r mean angle 32 64 128 256
7.93 ± 0.05 11.27 ± 0.01 15.47 ± 0.03 22.10 ± 0.01
8.00 11.31 16.00 22.63
0.991 0.996 0.967 0.977
89◦ 90◦ 86◦ 87◦
√ The 2r scaling. Table 2 reports ∥∆U ∥F measured at refresh events for Pythia-1B across ranks; the mean-angle column is arccos(1 − ∥∆U ∥2F /2r), the angle implied p by the mean √ cosine. The ratio ∥∆U ∥F (r=256)/∥∆U ∥F (r=32) = 22.10/7.93 ≈ 2.79 matches 256/32 = 8 ≈ 2.83 to 1.5%, √ √ √ confirming ∥∆U ∥F ∝ r. All ranks reach 96.7–99.6% of 2r. The 2r ceiling is the maximum Frobenius distance between two m×r orthonormal matrices, attained when all r principal angles equal π/2, the independent-Haar-sampling upper bound (Meckes, 2019). That correlated adjacent SVD bases nonetheless operate near this ceiling at every refresh is the non-trivial empirical fact established in §3. 15
Published as a conference paper at ICOMP 2026
B
B LIND - ESTIMATOR LOWER BOUND FOR THE SECOND MOMENT (W EINGARTEN )
This bound formalizes why carrying the second moment unchanged across a refresh is structurally suboptimal when the estimator does not observe the rotation, using the Weingarten calculus for Haar-random orthogonal matrices (Collins & Matsumoto, 2009). Theorem 1 (Blind-estimator lower bound). Let vold = λ = (λ1 , . . . , λr ) be the true per-coordinate second moments in the current subspace U (λi = E[gi2 ] for g = U ⊤ ∇L). Let the new subspace be U ′ = U R with R ∼ Haar(O(r)), and let v̂ be any estimator measurable with respect to σ(λ) only P 2 (i.e. blind to R). Then the optimal new-basis values are v ∗ (U ′ )i = j λj Rji , and 2r Var(λ) 2r Var(λ) ER ∥v̂ − v ∗ ∥2 = ∥v̂ − λ̄1∥2 + ≥ , (5) r+2 r+2 with equality iff v̂ = λ̄1 (the mean predictor). Canonical carry v̂ = λ incurs ER [∥λ − v ∗ ∥2 ] = r(r + 4) Var(λ)/(r + 2), a factor (r + 4)/2 ≈ r/2 above the floor. Proof. By the Weingarten calculus (Collins & Matsumoto, 2009), ER [v ∗ (U ′ )i ] = λ̄ and VarR [v ∗ (U ′ )i ] = 2 Var(λ)/(r + 2). Since v̂ is independent of R, the bias–variance decomposition gives ER [(v̂i − vi∗ )2 ] = (v̂i − λ̄)2 + 2 Var(λ)/(r + 2). Summing over i and minimizing over v̂ yields (5); substituting v̂ = λ gives the canonical-carry cost. Remark 2 (Representation-theoretic reading). The first moment transforms in the defining repre2 sentation of O(r) (m 7→ Rm), whereas the optimal second moment transforms via M (R)ij = Rji , which fixes 1 and decomposes into a trivial component (the mean λ̄) plus a sum-zero part. A blind estimator sees only the rotation-invariant λ, so any O(r)-equivariant linear prediction lies in the trivial component: the unique equivariant blind predictor is λ̄1. By Schur’s lemma the first moment admits an equivariant transport (Rm) but the second moment does not. This is the structural reason canonical carry is (r/2)-suboptimal rather than merely worse in practice. Methods that observe R (Robert et al., 2025; Rajabi et al., 2025b) fall outside the blind class and are not bound by (5). The bound takes R ∼ Haar(O(r)), the fully-degenerate limit; with a reproducible core of k ⋆ directions (§3.4) the Haar-resampled block has dimension r − k ⋆ and the effective suboptimality factor scales as (r − k ⋆ )/2, still ∼45× at the measured k ⋆ ≈ 39, r=128. Worked moments. The two moments the proof invokes follow from the second- and fourth-order statistics of a single Haar-random frame. A column R·i of R ∼ Haar(O(r)) is a uniform point on the unit sphere S r−1 , so its entries xj := Rji satisfy x2j ∼ Beta 12 , r−1 and, for j ̸= k, 2 1 3 1 E[x2j ] = , E[x4j ] = , E[x2j x2k ] = . (6) r r(r + 2) r(r + 2) P P P 3 These are consistent with j x2j = 1: summing gives j E[x4j ] + j̸=k E[x2j x2k ] = r+2 + r−1 r+2 = 1. P P P ∗ ′ 2 2 Writing v (U )i = j λj Rji and S1 = j λj , S2 = j λj , the first moment is ER [v ∗ (U ′ )i ] = S1 /r = λ̄, and X X 2 3S2 + (S12 − S2 ) 2S2 + S12 4 2 2 ER v ∗ (U ′ )2i = λj E[Rji ]+ λj λk E[Rji Rki ]= = , r(r + 2) r(r + 2) j j̸=k
2S2 + S12 S2 S 2 2 Var(λ) 2 VarR v ∗ (U ′ )i = − 21 = S2 − 1 = , (7) r(r + 2) r r(r + 2) r r+2 P using r Var(λ) = S2 − S12 /r. The canonical carry v̂ = λ then costs ER [∥λ − v ∗ ∥2 ] = i (λi − λ̄)2 + VarR (vi∗ ) = r Var(λ) + 2r Var(λ) = r(r+4) r+2 r+2 Var(λ), whereas the mean predictor v̂ = λ̄1 2r attains the floor r+2 Var(λ); their ratio is (r + 4)/2, the factor quoted in Theorem 1.
C
F IRST- MOMENT TRANSPORT IS THE MMSE LINEAR MAP
⊤ Proposition 3 (MMSE first-moment transport). Let m = Uold g with g ∼ N (0, σ 2 I), and consider ⊤ estimating m̂ = Unew g from m. Among linear estimators Am, the minimum-mean-squared-error ⊤ choice is A = Unew Uold = R, i.e. m̂ = R m.
16
Published as a conference paper at ICOMP 2026
⊤ ⊤ ⊤ Proof. Write g = Uold m + (I − Uold Uold )g, so m̂ = Rm + Unew (I − Uold Uold )g. The second term has zero mean and is uncorrelated with m under isotropy, so E∥m̂ − Am∥2 = E∥Rm − ⊤ ⊤ Am∥2 + σ 2 ∥Unew (I − Uold Uold )∥2F . The second term is irreducible; the first is minimized uniquely at A = R.
The isotropy assumption is exact only in the near-degenerate tail of the gradient spectrum, which is precisely the part of the subspace that redraws (§3.3); the determined spike is anisotropic but stable and does not rotate, so R is the optimal transport over exactly the directions that move.
D
C OVERAGE EQUALS CUMULATIVE SPECTRAL MASS
Proposition 4 (Coverage growth). If the gradient second-moment Pr PnΣg has eigenvalues λ1 ≥ · · · ≥ λn and U spans the top-r eigenvectors, then Eg [γg ] = i=1 λi / i=1 λi . Proof. E[∥U U ⊤ g∥2 ] = tr(U U ⊤ Σg U U ⊤ ) =
Pr
2 i=1 λi and E[∥g∥ ] = tr(Σg ) =
Pn
i=1 λi .
When a few eigenvalues dominate (the spike of §3.3), E[γg ] rises steeply for r up to the spike width and then flattens: past the spike, adding tail directions of nearly equal energy raises coverage only marginally and almost independently of which tail directions are chosen, the property that makes a re-drawn frame an essentially equivalent projector (§4).
E
E XPERIMENTAL CONFIGURATIONS AND REPRODUCIBILITY
Models and corpora. Spectrum and drift measurements use Pythia-160M; the β2 validation and the ∥∆U ∥F rotation law use Pythia-1B. Language-model training is on WikiText; the drift-rate invariance check additionally uses C4 (below). Runs were executed on two independent clusters, NVIDIA L40S and AMD MI210, which we use to cross-check the staleness measurements. β2 runs (Figure 5). Table 3 gives the exact values; each row has its own recipe, so perplexities are comparable within rows only (LDAdam’s 57.69 here and its 18.73 in Table 1 are the same optimizer at 10K/lr 10−3 and 40K/lr 4 × 10−3 respectively). Canonical GaLore uses the configuration r=128, refresh T =160, learning rate 4 × 10−3 , projection scale 0.25, cosine schedule with warmup 4000, 40K steps, three seeds per β2 ; LDAdam uses the faithful official implementation at r=128, learning rate 10−3 , 10K steps, three seeds per β2 ; APOLLO-Mini uses two seeds. The full-rank AdamW control runs the identical data and schedule with no projection. Table 3: Lowering β2 from the 0.999 default to 0.99 at Pythia-1B (the data of Figure 5). For each refreshing optimizer shown, the lower value is better; for full-rank AdamW the preference reverses. LDAdam’s gap is largest at its own 10K recipe, but gap sizes are not comparable across recipes and we do not read it as a larger staleness effect (§5.3). APOLLO-Mini (below the second rule) refreshes a random Gaussian projection rather than an SVD top-r subspace and enters as a directional check only. The canonical-GaLore margin is training-length dependent (§5.3). Optimizer
rot.
β2 =0.999
β2 =0.99
∆ (PPL)
steps/seeds
LDAdam (Robert et al., 2025) Canonical GaLore (Zhao et al., 2024)
yes yes
57.69 ± 0.11 21.27 ± 0.17
31.27 ± 0.31 19.04 ± 0.13
26.4 2.23
10k/3 40k/3
APOLLO-Mini (Zhu et al., 2024)
yes
99.8
97.0
2.8
–/2
Full-rank AdamW (control)
no
97.9
118.0
−20.1 (reversed)
14.4k/1
Spectrum probe (Figure 2). At Pythia-160M and 1B we take the per-matrix gradient G for the embedding, attention query/key/value, attention output, and MLP weight matrices, compute its singular values, and report: the adjacent-ratio σr /σr+1 at r=128; the maximum adjacent ratio anywhere in the bulk (maxi σi /σi+1 over the smooth region, “max bulk gap”); the effective rank 17
Published as a conference paper at ICOMP 2026
exp(H) P with H the PShannon entropy of the normalized squared spectrum; and the top-r nuclear fraction i≤r σi / i σi . The two scales agree closely: mean σr /σr+1 at r=128 is ≈ 1.005 at both scales (1.001–1.010 at 160M, 1.001–1.017 at 1B); max bulk gap stays in 1.011–1.022 (160M) and 1.011–1.030 (1B); effective rank spans 89–280 (160M) and 97–358 (1B), straddling r at both; and the top-r nuclear fraction is 0.52–0.68 (160M) and 0.46–0.65 (1B). Measurements are taken over four layer types at steps 80/100/120. Rotation probe (∥∆U ∥F , αdrift ). Both quantities use the same primitive, the mean principal angle ⊤ between two orthonormal bases, θ = arccos svdvals(Unew Uold ) averaged over columns. ∥∆U ∥F (Eq. 2) applies it across a refresh of the imposed GaLore subspace; αdrift (Appendix J) applies it per step to the intrinsic top-r gradient-covariance subspace.
F
I DENTIFIABILITY PROBE PROTOCOLS
The same-step control (§3.2), the reproducible count k ⋆ (§3.4), and the averaging-budget scaling (§3.5) all read from one construction, described here so they reproduce without the code. All three run on a pretrained Pythia-160M in float32, evaluation mode (no dropout; gradients still flow), r=128, on the four layer types {embed in, layer-0 attention query key value, attention dense, MLP dense h to 4h}. Gradient pool. We draw K=128 minibatches, each B=8 sequences of length 512 taken from disjoint, non-overlapping slices of WikiText-103, and for each compute the per-matrix gradient of the next-token cross-entropy loss. For a gradient G we work on the smaller side, M = G if rows ≤ cols else G⊤ , and define its top-r subspace as the top-r eigenvectors of M M ⊤ (equivalently the top-r left singular vectors). Because every minibatch is evaluated at the same checkpoint, two subspaces built from two pools differ only in the sample, never in elapsed training. Same-step disagreement and k ⋆ (Figure 3a). For an averaging fold N ∈ {1, 2, 4, 8, 16, 32, 64} and 4 repetitions, we draw 2N distinct pool indices, split them into two halves of NP , average each half ⊤ 1/2 into ḠA , ḠB , and take their top-r subspaces U , U . We report ∥∆U ∥ = (2r −2 1 2 F i σi (U1 U2 )) P ⊤ 2 ⋆ (Eq. 2), the subspace overlap ov = i σi (U1 U2 ) /r ∈ [0, 1], and k (N ) = r · ov, the √ effective number of shared directions. At N =1 this is the same-step split-batch disagreement (0.725 2r over four layers; a four-layer-mean k ⋆ ≈ 39, 48 on the query-key-value matrix, per-layer standard error √ below 1 over the 12 repetitions of the trajectory run below). The across-time reference 0.742 2r is a separate measurement: the sign-corrected rotation ∥∆U ∥F across one refresh interval of 160M training, not on the frozen pool. The frozen same-step value is the noise floor; the across-time rotation exceeds it only marginally. The floor is not specific to the pretrained checkpoint. A natural objection is that the same-step floor is read from a pretrained checkpoint while the across-time rotation is read during training, so the two sit at different optimization states. We therefore reran the N =1 probe unchanged (same four layers, K=128 pool, r=128, 12 repetitions) at five points along the released Pythia-160M training trajectory (revisions step1000, step10000, step40000, step80000, step143000). The four-layer same√ step disagreement is {0.648, 0.668, 0.700, 0.708, √ 0.710} 2r: lowest early in training and rising toward the pretrained value, but never below 0.64 2r. The reproducible core is correspondingly larger early (query-key-value k ⋆ 50.9, 58.5, 52.8, 51.4, 51.6 across the same checkpoints). At every training state √ the across-time rotation (0.742) therefore exceeds the same-step floor by at most a few percent of 2r, so the rotation is dominated by non-identifiability rather than by motion of a stable subspace, and the conclusion does not depend on the state at which the floor is measured. N -averaging spectra and the budget test (Figure 3b). The same run additionally logs the singular spectrum of each averaged matrix ḠA (densely over the top 300 ranks, log-spaced in √the tail). Three analyses follow. (i) Deep-tail scaling: a tail singular value at fixed rank, scaled by N , is flat if the tail is pure sampling noise (σ ∝ N −1/2 ); the measured value instead rises 2.0–3.2× from N =1 to 64, i.e. σ ∝ N −1/4 . (ii) Out-of-sample prediction: a spiked-covariance (BBP) model (Baik et al., 2005) with a single noise scale ν1 per layer, calibrated only to reproduce the measured overlap at 18
Published as a conference paper at ICOMP 2026
√ N =1, predicts ov(4N ) from the de-censored spectrum of ḠN (setting the noise floor to ν1 / N ); over 20 held-out points the mean absolute error is 0.073, above the pre-registered pass threshold of 0.03, which is why we report a measured scaling and not a closed-form law. (iii) De-censored exponent: the log–log slope of ranks 3–60 of the N =64 spectrum gives a signal power law with exponent α = 1.03–1.35, agreeing with the exponent implied by the overlap-versus-N curve. The averaging-budget result holds across scale and GPU backend. We reran the same-step and N -averaging probe unchanged on pretrained checkpoints at 410M, 1B, and 2.8B. At every scale, averaging to N =64 (64× the gradient sample) removes only 24–29% of the single-batch same-step disagreement (Table 4), the same weak decay measured at 160M, so no averaging budget restores identifiability at any scale tested. As a numerics control, rerunning the 160M probe on a second GPU backend (AMD MI210, ROCm) reproduces the reference measurement (NVIDIA L40S, CUDA) to within 0.3% (0.709 versus 0.707 at N =1), confirming the persistent tail is a property of the gradient, not of the linear-algebra backend. Table 4: Same-step disagreement decay under N -fold averaging, across scale (pretrained checkpoints, r=128, four-layer mean). Entries are the N =64 disagreement as a fraction of the N =1 disagreement: 1.0 would mean averaging changed nothing and 0 would mean it fully resolved the subspace. Averaging removes only about a quarter at every scale; the reproducible core k ⋆ separately shrinks with scale (Table 5). Scale
√
∥∆U ∥F / 2r at N =64, relative to N =1
G
160M
410M
1B
2.8B
0.76
0.74
0.72
0.71
G ENERALITY OF SAME - STEP NON - IDENTIFIABILITY
Table 5 gives the per-model same-step disagreement (§3.6, Figure 4). The probe is the same-step protocol of Appendix F run unchanged on pretrained checkpoints (all trained with full-rank Adam), r=128, K=32 gradient matrices of B=4 sequences, four layer types (embedding, attention input, attention output, MLP), four repetitions; the language models use WikiText, the vision transformer √ uses CIFAR-10 images. We report the mean over layers of the N =1 disagreement ∥∆U ∥F / 2r, the mean adjacent singular-value ratio at r, and the mean reproducible count k ⋆ . The disagreement rises with scale and k ⋆ falls; no model shows a spectral gap at r. Table 5: Same-step subspace disagreement across architectures, scales, and domains (pretrained, full-rank-Adam checkpoints; r=128; mean over four layer types). The Pythia-160M row reproduces the main-text control (0.73 here at B=4, K=32 versus 0.725 at B=8, K=128; k ⋆ shifts by one under the same change of settings, 38 here versus the four-layer mean 39 of the main text). Model
Architecture
√ ∥∆U ∥F / 2r (N =1)
adj. ratio at r
k⋆
Pythia-70M Pythia-160M Pythia-410M Pythia-1B Pythia-1.4B Pythia-2.8B Pythia-6.9B
GPT-NeoX GPT-NeoX GPT-NeoX GPT-NeoX GPT-NeoX GPT-NeoX GPT-NeoX
0.67 0.73 0.76 0.81 0.80 0.80 0.83
1.006 1.006 1.006 1.006 1.005 1.007 1.004
50 38 33 25 25 25 21
GPT-2-large Qwen2.5-3B Llama-3-8B
GPT-2 (learned-pos, GELU) Llama-class (RoPE/SwiGLU) Llama-3
0.71 0.78 0.81
1.007 1.007 1.009
43 29 25
ViT-B (CIFAR-10)
vision transformer
0.53
1.022
76
Rank versus k ⋆ : transport, do not truncate. A natural reading of k ⋆ is that the rank should be reduced toward it. It should not. In a single-seed sweep at Pythia-1B (β2 =0.99, T =160, 14K 19
Published as a conference paper at ICOMP 2026
steps), canonical GaLore reaches 27.9 perplexity at r=128 against 29.8 at r=64, and worse below (this recipe is not comparable to the 40K runs elsewhere; we compare only within it). The result is directional but consistent with the coverage argument (§4): the directions beyond k ⋆ are individually non-identifiable yet still capture useful gradient energy, so the excess rank is worth keeping. The fix for the stale state it carries is to transport it (§5.2), not to truncate the rank.
H
T HE SECOND MOMENT LAGS ON THE SAME CURVE AS FULL - RANK A DAM
The mechanism of §5 predicts that the second moment’s staleness should be governed by its memory τv = 1/(1 − β2 ), and that a GaLore-family optimizer at a given β2 should be no more anomalous than full-rank Adam at the same β2 . We confirm both. We define the per-cycle staleness lag as the mean distance between the carried second moment and its post-rotation optimum over a refresh cycle. Lag rises monotonically with τv : for full-rank Adam at Pythia-160M (AMD MI210, 4K steps, two seeds), lag = 0.11, 0.14, 0.19, 0.25, 0.26, 0.29 at β2 = 0.9, 0.95, 0.97, 0.99, 0.995, 0.999 (τv = 10, 20, 33, 100, 200, 1000), with seeds agreeing to ∼0.01. On a second cluster (NVIDIA L40S), GaLore at matched β2 lies on the same curve as full-rank Adam measured there: 0.13 against 0.16 at β2 =0.95, and 0.30 against 0.36 at β2 =0.99 (within-cluster comparisons only). The lag is also flat across the refresh interval (0.27–0.31 over an 8× range of T ), as expected for a quantity set by τv rather than by T . This replicates across both clusters (NVIDIA L40S and AMD MI210, different framework versions and data pipelines). We restrict this overlay claim to β2 ≤ 0.99. At β2 = 0.999 the instantaneous lag shows a large transient that does not persist to steady state; we therefore do not read it as a steady-state divergence, and the loss consequence of high-β2 staleness is the integrated effect documented in §5.3, not an instantaneous lag explosion.
I
A FORGETTING - FACTOR BIAS – VARIANCE ARGUMENT FOR β2
This appendix supports the qualitative claim of §5.3 that a subspace re-drawn every T ≪ τv steps pushes the error-minimizing second-moment decay below the no-reset optimum. It is a specialization of the classical forgetting-factor bias–variance tradeoff (Guo et al., 1993) to a target that the optimizer resets itself. We do not derive an optimal β2 ; we show the minimizer is interior and moves down as the reset becomes more frequent, which is all the prescription rests on. Model. Fix one projected coordinate. The second moment follows the exponential moving average vt = β2 vt−1 + (1 − β2 )xt , where xt = g̃t2 is the instantaneous squared projected gradient, an unbiased noisy reading of the true per-coordinate second moment with E[xt ] = λ∗t and Var(xt ) = s2 . The target λ∗t is piecewise constant, holding for the T steps of a refresh interval and jumping at each refresh when the basis is re-drawn. Under the Haar model of Appendix B the coordinate’s target after a refresh has the same mean λ̄ and a mean-squared change ∆2 := E[(λ∗new − λ∗old )2 ] = Θ(Var(λ)) across the jump. The two error terms. Averaged over a refresh interval, the steady-state per-coordinate meansquared error E[(vt − λ∗t )2 ] splits into a tracking-noise term and a lag term: MSE(β2 ) =
1 − β2 2 1 − β22T s + ∆2 . 1+β T (1 − β22 ) | {z2 } | {z } noise
(8)
lag
The noise term is the stationary variance of an EMA of i.i.d. readings, Var(v∞ ) = (1−β2 )s2 /(1+β2 ); it decreases in β2 , so a longer memory averages away more sampling noise. The lag term is the mean-squared residual bias after a target jump: right after a refresh the bias decays as β2k over the k steps since the jump, and averaging ∆2 β22k over k = 0, . . . , T − 1 gives ∆2 (1 − β22T )/ T (1 − β22 ) ; it increases in β2 . Why the optimum moves down under refresh. Write τv = 1/(1 − β2 ). For β2 near 1, 1 − β22 ≈ 2/τv and β22T ≈ e−2T /τv , so the lag term is 12 ∆2 (τv /T ) 1−e−2T /τv , the same τv (1−e−2T /τv )/T 20
Published as a conference paper at ICOMP 2026
shape that governs any EMA tracking a reset target. Two limits fix its behavior. When resets are rare relative to the memory, T ≫ τv , the lag term is O(τv /T ) → 0 and MSE is monotone decreasing in τv : the minimizer is β2 → 1, the standard no-reset preference. When resets are frequent, T ≪ τv , 1 − e−2T /τv ≈ 2T /τv and the lag term saturates at ∆2 , its ceiling, once τv ≳ T . Past that point raising β2 buys only the residual O(s2 /τv ) of noise reduction while holding near-maximal lag, so the minimizer β2⋆ is interior and decreases as T shrinks. This is the direction §5.3 confirms empirically: the second moment, whose target GaLore re-draws every T ≪ τv steps, prefers β2 =0.99 over 0.999, while the full-rank control, whose target is never reset (T = ∞), keeps the 0.999 preference. We do not claim 0.99 is the derived optimum; the argument fixes only the sign of the shift.
J
T HE EIGENBASIS DRIFT RATE IS ARCHITECTURE - INTRINSIC
√ The rotation law ∥∆U ∥F ≈ 2r concerns the imposed GaLore subspace. For completeness we also measure the drift rate of the model’s intrinsic gradient geometry: the per-step mean principal-angle rotation of the top-r gradient-covariance eigenbasis, αdrift . At Pythia-160M this is αdrift ≈ 0.05 rad/step, and it is strikingly invariant: 0.049–0.054 across all β2 , 0.057 (random sampling) versus 0.054 (sequential), and 0.0558 (WikiText) versus 0.0556 (C4), a difference below 0.5%. The drift rate is thus a property of the architecture, not of the optimizer setting, the sampling order, or the corpus. Remark 5 (Scope of the “staleness” axis). Because αdrift is architecture-set and we have not varied it across architectures, we do not claim a fully bivariate universal staleness law in (αdrift , τv ). The axis we validate is the second-moment memory τv = 1/(1 − β2 ) (§5, Appendix H); the αdrift axis is reported as a measured constant √ for the architectures studied. We also note that the imposed GaLore rotation arrives in bursts of size 2r every T steps, whereas intrinsic drift is continuous, so the two enter a shared staleness picture only up to this burst-versus-continuous distinction.
21