arXiv CSopen access

SLORR: Simple and Efficient In-Training Low-Rank Regularization

SLORR: Simple and Efficient In-Training Low-Rank Regularization

1

David González-Martínez1,2,3,4 Shiwei Liu1,3,4 Max Planck Institute for Intelligent Systems 2 University of Tübingen 3 ELLIS Institute Tübingen 4 Tübingen AI Center [email protected] [email protected]

arXiv:2607.08754v1 [cs.LG] 9 Jul 2026

Abstract Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.

1

Introduction

Low-rank factorization is a widely used approach for reducing the computational and memory costs of neural networks: a dense weight matrix is replaced by a product of smaller factors, yielding cheaper inference when the retained rank is sufficiently small [1–4]. However, compression quality depends on the spectral properties of weight matrices: discarding meaningful spectral directions can degrade model performance. Thus, while moderate post-training compression is generally possible, modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. This motivates training-time methods that encourage low-rank structure before compression. A natural formulation is to encourage sparsity in the singular values of weight matrices, concentrating spectra into fewer dominant directions. Although direct spectral regularization can be effective [5, 6], large singular value decompositions (SVDs) are needed at every training iteration, which is prohibitively expensive for modern training pipelines. Several ways of avoiding SVDs have been proposed [7–9]; however, these generally introduce new limitations that hinder practicality. For example, some works replace layers with factorized parameterizations, changing the architecture [7, 8], but this generally increases the number of trainable parameters and alters optimization dynamics. A further class of methods maintains cached spectral quantities that are only periodically updated [9], reducing but not eliminating SVD cost, and introducing statefulness and additional hyperparameters. Preprint.

Table 1: Comparison of different low-rank-inducing regularizers. Detailed discussion in Section 2. Method Changes arch. SVD Efficiency & scalability Stateful Prior target rank SVD-based [5] No Yes Poor No No Factorize + reg. [7] Yes No Good No No LoRITa [8] Yes No Good No No Q3R [9] No Periodic Moderate Yes Yes SLORR (ours) No No Good No No

Our proposed method, SLORR, addresses these limitations. In particular, SLORR operates directly on the original weight matrices without altering the architecture, is SVD-free and efficient, and does not maintain any cached quantities, providing a versatile framework. The key idea is simple: we approximate the spectral quantities needed for the regularizer forward and backward passes using GPU-efficient polar factor approximations. We explore two main variants: SLORR-Hoyer and SLORR-Nuc, which respectively correspond to the squared Hoyer sparsity metric [10] and the nuclear norm, two common low-rank penalties. For both variants, the resulting approximations admit guarantees for the regularizer values and their gradients (Proposition 3.1). A summary of directly relevant related works appears in Table 1, with a more extensive discussion in Section 2. Empirically, we evaluate SLORR across vision and language-modeling settings. On ImageNet1K [11], we study short-horizon continued training of ResNet-50 [12], ViT-B/16, and ViT-L/16 [13], as well as ResNet-18 pretraining [12]. Across these settings, SLORR improves post-training compressibility over unregularized training while adding less than 8% training overhead. We further evaluate SLORR-Hoyer on LLM pretraining at the 135M and 560M scales, where SLORR-trained models preserve performance substantially better after compression than unregularized models, with less than 1% average training overhead. Overall, our contributions are as follows. First, we introduce an SVD-free framework for low-rankinducing regularization using GPU-friendly approximations. Second, we show how this framework can be used to regularize the spectra of the original weight matrices without architectural changes, cached spectral quantities, or a pre-specified target rank. Third, we demonstrate across vision and language-modeling experiments that SLORR improves post-training low-rank compressibility with small training overhead.

2

Background and Related Work

Various factorization approaches have been extensively studied over time [1–3]. In this work, we focus on factorization schemes that separate a single matrix multiplication into two sequential ones: f (X; W ) = XW ≈ fˆ(X; W0 , W1 ) = (XW0 )W1 , where X ∈ RB×Di is the input, W ∈ RDi ×Do the weight of the original layer f , and W0 ∈ RDi ×P and W1 ∈ RP ×Do are the weights of the new approximate layer fˆ. With sufficiently small P , we can achieve inference speedups and memory reduction. In particular, we focus on those that employ the SVD of weight matrices [1, 14], which is optimal in weight space according to the Eckart–Young–Mirsky theorem [15]. In particular, the factors can be obtained from the SVD of W : 1/2

1/2

⊤ W1 = Σ:P,:P V:,:P .

W0 = U:,:P Σ:P,:P ,

In convolutional neural networks (CNNs) [16], SVD-based factorization can be performed by reshaping parameter tensors into matrices and subsequently decomposing them (normally called channel-wise decomposition) [7, 8]; we follow this approach throughout this work. Other recent works, such as activation-aware methods [4, 17–20], introduce different ways of computing factors that can lead to better overall accuracy. 2.1

Inducing Low-Rank Parameters

Since compressing layers by SVD truncation essentially amounts to truncating their singular values, the accuracy degradation incurred is directly related to their spectral properties: a desirable property is that the layer weight is (approximately) low-rank. Hence, an intuitive approach to obtaining more 2

compressible layers is to regularize the model to gradually decrease the (approximate) rank of its weight matrices during training. One can roughly think of the rank minimization objective as a sparsity objective on the singular values of the parameters, i.e., X X L = Ltask + λ rank(W i ) = Ltask + λ ∥σ(W i )∥0 , i

i

where σ denotes the singular values of W i . This objective is considered to be intractable [21]. Prior work has addressed this by using proxy objectives, such as minimizing the nuclear norm [5, 6]; however, this generally requires computing the SVDs of each weight at each training step, which is computationally prohibitive for larger models. Several solutions have been proposed. Some approaches rely on modifying the architecture. For example, Yang et al. [7] obtain the SVD U , V , σ of each layer and substitute each layer with a factorized form (retaining all singular values). The authors then incentivize σ to be sparse through the Hoyer sparsity metric [10] (they also explore the ℓ1 norm, akin to nuclear-norm regularization). An additional regularizer is required to keep U and V orthogonal and maintain general SVD semantics, making it less practical. A recent approach, QN named LoRITa [8], reparametrizes each layer weight W with an ordered product i Wi = W and penalizes the Frobenius norm of each Wi , which can be shown to work as a proxy for the nuclear QN norm minimization objective of the overall i Wi . A similar approach was proposed in Kliegl et al. [22] for depth-2 decompositions. These approaches generally increase the number of trainable parameters and can alter optimization dynamics. Another line of work reuses computed factors instead of recomputing them at every iteration. For example, the recent Q3R [9] uses a surrogate objective that can be approximated with periodic SVDs1 . However, it is stateful (as it needs to store the temporary factors until they are recomputed). It also requires specifying a target rank in advance, making hyperparameter tuning challenging. Other approaches for approximate computations exist: Li et al. [23] propose randomized differentiable surrogates for generalized low-rank regularization, applying them to tasks including matrix completion, video foreground/background separation, and denoising. While the method is SVD-free in theory, it is still impractical, requiring backpropagation through long chains of matrix multiplications. Moreover, its practical implementation2 still requires a (smaller) SVD and matrix inverse. This makes it infeasible for weight regularization, even at moderate scales. Qin et al. [24] propose a method to make models amenable to activation-aware methods (in particular, SVD-LLM [18]), but it is focused on very short-horizon post-training, while we focus on longer training periods (from continued training to pretraining). Relatedly, low-rank induction can also happen (either explicitly or implicitly) at the optimizer level. For example, Zimmer et al. [25] use a Frank–Wolfe-based optimizer to promote low rank in weight matrices using partial SVDs. Dolatabadi et al. [26] find that, surprisingly, Muon-trained LLMs are more compressible and that truncated Muon updates can further improve compressibility. Here, we restrict our focus to explicit regularization techniques, typically formulated either as additional loss terms or as decoupled penalties applied outside the primary gradient update, analogous to decoupled weight decay [27]. In Table 1, we compare relevant direct low-rank regularization methods. Our proposed approach is simpler: it operates on the original model, is SVD-free and efficient in practice, and is stateless. Lastly, a related area, but fundamentally different in nature, is that of low-rank training, where the rank is generally pre-defined before training, and a low-rank architecture is directly trained. Different variants, such as full-rank warmup or combinations with sparse training, and aspects like initialization, have been investigated over time [28–33].

3

Our Framework

For a weight matrix W ∈ RI×O , we use U ∈ RI×R , Σ ∈ RR×R , V ∈ RO×R to denote its thin SVD factors (W = U ΣV ⊤ ), where R = rank(W ) ≤ min(O, I). Additionally, we use σ to denote 1 Their main experiments are run with a refresh period of 5 iterations, which is expensive. In our main Q3R experiments, we also use 5; however, we found that using a larger period (which is less expensive) can also work well (Appendix F). The main concern is scalability. As one uses larger models, maintaining a low cost with Q3R requires enlarging the refresh period. It is expected that, with a sufficiently large refresh period, its effectiveness will be reduced. 2 https://github.com/naiqili/EDLRR

3

the vector of singular values (σ = diag(Σ)). Since the SVD is not unique, when mentioning it or any factor, we implicitly refer to an arbitrary valid choice. We view the problem from the perspective of inducing sparsity in singular values. We adopt (1) the ℓ1 norm of singular values, i.e., the nuclear norm, and (2) the squared ℓ1 /ℓ2 -ratio (the nonsquared version is also known as the unnormalized Hoyer metric [10]) of singular values, which is effective as a sparsity-inducing regularizer [34] (the nonsquared version has also been shown to work well in factorization-based low-rank regularization [7]). We have: X X X Lhoyer = ∥σ i ∥21 /∥σ i ∥22 = ∥W i ∥2∗ /∥W i ∥2F , Lnuc = ∥W i ∥∗ , i

i

i

where i denotes the layer index. From now on, we will omit layer indices for brevity. We optimize these regularizers together with the task loss. Strictly speaking, the regularizers are not differentiable everywhere. We use the natural backpropagation rule, which corresponds to the minimum-Frobenius-norm element of the Clarke generalized gradient (see Appendix A for the derivations). We use gradient notation for simplicity. We have that:   ∥W ∥∗ 1 ∥W ∥∗ ⊤ ∇Lhoyer (W ) = 2 UV − W , ∇Lnuc (W ) = U V ⊤ . (1) ∥W ∥F ∥W ∥F ∥W ∥3F Both gradients have terms that are generally computed Listing 1: Example PyTorch implementathrough SVDs, which is prohibitively expensive to do tion of SLORR-Hoyer. at every iteration. Fortunately, one can (through iteraLowRankRegularizer(torch.autograd.Function): tive methods) approximate every term needed in Equa- class@staticmethod def forward(ctx, W, eps, steps=6): tion (1). The resulting approximations have provable frob = torch.linalg.norm(W, ord="fro") + eps convergence guarantees (Proposition 3.1). UV = polar_express(W, steps=steps) nuc = torch.sum(W * UV) reg = nuc / frob ctx.save_for_backward(W, UV, frob, nuc, reg) return reg ** 2

Computing the polar factor. Notice that we do not need to compute the factors of U V ⊤ separately; we @staticmethod only need their product3 , which is generally known as def backward(ctx, grad_output): the generalized polar factor (we use “polar factor” for W, UV, frob, nuc, reg = ctx.saved_tensors G = (UV / frob) - (nuc / (frob**3)) * W short) of W . It can be computed efficiently on GPUs return grad_output * 2 * reg * G, None, None using iterative methods (e.g., [36]; a similar approach is used in the Muon optimizer [37] to obtain polar factors of gradients). In particular, we use the recent Polar Express [38], which approximates the polar factor iteratively, to efficiently compute an approximation of U V ⊤ . Polar Express depends on several hyperparameters, including the number of iterations T , most of which we set to their recommended values. It first normalizes the matrix so that its largest singular value is at most 1. This is achieved by dividing by the Frobenius norm, i.e., it uses W /(∥W ∥F + ε)4 . Moreover, it also needs a lower bound estimate of the smallest nonzero singular value of the normalized matrix, ℓ, which we set to 10−3 following their recommendations. We provide a more detailed discussion of our usage of the Polar Express in Appendix D.1. Computing the nuclear and Frobenius norms. After we have (an approximation of) U V ⊤ , we can compute an estimate of the nuclear norm with elementwise multiplications and a summation. In particular, it holds that (for any exact U V ⊤ ) sum(W ⊙ U V ⊤ ) = tr(W ⊤ U V ⊤ ) = tr(V ΣU ⊤ U V ⊤ ) = tr(Σ) = ∥W ∥∗ , where ⊙ denotes elementwise multiplication, and the intermediate steps qPfollow from the cyclic 2 property of the trace operator. The Frobenius norm is simply ∥W ∥F = i,j Wi,j . Combining everything. We now have all the elements needed to compute approximate values and approximate backpropagation rules for both variants, which can be implemented, for example, as a custom PyTorch function. Forward and backward computations of SLORR-Hoyer are summarized 3 Note that we are using the left and right singular vectors corresponding to the nonzero singular values of the matrix. This quantity is unique for a given matrix [35]. 4 In Listing 1, normalization is performed inside the Polar Express function.

4

in Listing 15 ; SLORR-Nuc admits a similar implementation. Note that we do not backpropagate through the Polar Express iterations. Instead, the approximate polar factor is used in place of the U V ⊤ term that appears in the regularizer values and in the gradient rules in Equation (1). During training, we compute the task loss and regularization loss, weighted by the regularization strength λ, and use gradient-based optimization as usual6 . In the case of Adam [39], one can also use a decoupled low-rank regularization term, similar to Q3R [9]; see Algorithm 1. Computational cost. In terms of matrix multiplication FLOPs (the main cost), each iteration of Polar Express (with our exact settings) with a matrix of shape (M, N ) uses 4P 2 Q + 2P 3 FLOPs, where P = min{M, N } and Q = max{M, N }, and we generally use 6 iterations. For convolutional layers, we reshape kernels into matrices (details in Appendix E.1); in which case we get M = Co and N = Ci Kw Kh , denoting the output and input channels, as well as the width and height of the convolutional kernel, respectively. We discuss practical SLORR overhead measures in Section 4.1.1 and Section 4.2, where we find that it generally remains below or around 8% in our ImageNet-scale settings and 1% in LLM pretraining at the 135M and 560M scales, whereas SVD remains prohibitively expensive. Approximation guarantees. We remark that the computed forward and backward passes of both variants are approximate in practice. Polar Express has worst-case approximation guarantees, which can be used to provide guarantees on the values and gradients of both SLORR variants (Proposition 3.1). We provide further discussion of this result and its proof in Appendix C. Proposition 3.1 (Approximation guarantees). Let W ̸= 0 have thin SVD W = U ΣV ⊤ , and assume the nonzero singular values of a normalized version of W lie in [ℓ, 1]. Let Pb denote the Polar (q+1)T

Express approximation of the polar factor U V ⊤ after T steps, and define δ = 1 − ℓ2 , where d = 2q + 1 is the Polar Express polynomial degree. Additionally, let n̂ := tr(W ⊤ Pb ) denote the approximate nuclear norm estimate, and let n := ∥W ∥∗ and f := ∥W ∥F . Moreover, let R denote b its approximation. Then the exact SLORR-Hoyer gradient and R |n̂ − n| ≤ δ, ∥Pb − U V ⊤ ∥2 ≤ δ (SLORR-Nuc), n   2 |n̂2 /f 2 − n2 /f 2 | b 2 ≤ 2δ(2 + δ) n + n ∥W ∥2 ≤ δ(2 + δ), ∥R − R∥ (SLORR-Hoyer). n2 /f 2 f2 f4 In particular, errors vanish as T → ∞. Practical remarks. The singular value range in Proposition 3.1 is a theoretical assumption on the normalized matrix, inherited from the Polar Express analysis in Amsel et al. [38]. In practice, Polar Express treats ℓ as an approximate lower bound parameter; the authors report that inaccurate guesses are typically not severe. Therefore, the displayed worst case bound should be interpreted as applying to normalized nonzero singular values lying in [ℓ, 1], while our choice ℓ = 10−3 follows their recommended value. We also emphasize that this is a worst case bound in an idealized setting. In practice, a smaller number of iterations works well even when the bound is not tight, and floating point error and implementation details may introduce additional approximation error. We follow the practical implementation choices of Amsel et al. [38], including their recommended parameter settings and numerical stabilizations. Further discussion is provided in Appendix D.1 and Appendix C. 3.1

Understanding SLORR-Hoyer

Our analysis is performed in an exact regularizer-only gradient descent setting. In this section, η > 0 denotes the step size of the regularizer-only update, with the training-time regularization strength absorbed into it. Additional comments and proofs appear in Appendix B. We let W = U diag (σ)V ⊤ be a weight matrix and its thin SVD. Additionally, let W + be the weight matrix after 5 As shown in Listing 1, in practice, we stabilize the forward and backward expressions separately (another option is to use a stabilized forward expression and derive the exact backward expression from it) by substituting ∥W ∥F 7→ ∥W ∥F + ε. This is done to avoid a possible division by zero. 6 The autograd-based implementation is generally easy to integrate into existing codebases. However, in practice, for PyTorch-based implementations, one can save memory by following the approach in Appendix D.3. This approach is used for our vision experiments.

5

a regularizer-only gradient step from W , and define σ + , U + , and V + as its thin SVD. We assume ordered singular values in decreasing order (σ1 ≥ · · · ≥ σr > 0) and W ̸= 0. We first remark that SLORR-Hoyer is scale-invariant (in the same way that the original version for vector sparsity is [10, 34]).7 Intuitively, this means it aims to redistribute the spectral energy rather than grow or shrink it. Note, however, that in a practical discretized optimization setting, the overall norm of the weight tends to increase (note that the gradient is orthogonal to the weight matrix): ∥W − η∇Lhoyer (W )∥2F = ∥W ∥2F − 2η ⟨W , ∇Lhoyer (W )⟩F + η 2 ∥∇Lhoyer (W )∥2F ≥ ∥W ∥2F . In preliminary experiments, we tried a norm-correction step that did not seem to help in practice. Proposition 3.2 (SLORR-Hoyer shrinkage and growth of singular values). For a sufficiently small regularizer-only gradient step, it holds that σi+ ⋄ σi ⇐⇒ σi ⋄

∥W ∥2F , ∥W ∥∗

for ⋄ ∈ {<, >, =}.

Proposition 3.2 suggests that SLORR-Hoyer enlarges “large” singular values and shrinks “small” ones, where “large” and “small” are determined by the dynamic quantity ∥W ∥2F /∥W ∥∗ , which varies as the weights evolve. Intuitively, this concentrates the spectrum onto the already large singular values. In turn, under certain assumptions, it induces low-rank compressibility (Proposition 3.3). Proposition 3.3 (Sufficiently small steps of SLORR-Hoyer induce compressibility). Let τ = ∥W ∥2F /∥W ∥∗ . For a sufficiently small regularizer-only gradient step, if σk > τ , then + + +⊤ ∥U:,:k diag(σ:k )V:,:k − W + ∥2F

∥W + ∥2F

4

<

⊤ ∥U:,:k diag(σ:k )V:,:k − W ∥2F . ∥W ∥2F

Experiments

Our goal is to evaluate the effectiveness of SLORR in inducing compressibility during training. Our experiments cover two main domains: image classification on ImageNet-1K [11], including both continued training from a checkpoint and pretraining, and LLM pretraining. Our general pipeline consists of first training regularized and unregularized models, and then compressing them and assessing their retained performance. Unless stated otherwise, regularization and compression are applied to all linear and convolutional layers, except for the first and last layers; for transformers, this corresponds to the layers inside the transformer blocks. In our experiments, SLORR-Hoyer-D denotes the decoupled variant, following Appendix D.2. Our code is available at https://github.com/davidgonmar/SLORR. 4.1

Image Classification Experiments

We first evaluate SLORR on ImageNet-1K classification. We cover two settings: pretraining and continued training, across four architectures. We compare against recent baselines, which have largely focused on the image classification domain. In total, we perform more than 150 runs across settings. Baselines and comparisons. We compare SLORR against Q3R [9] and LoRITa [8]. These baselines are used because: (1) they are the most recent relevant methods, to the best of our knowledge, and (2) they allow us to compare different objectives, whereas factorization-based approaches [7] generally have similar objectives to ours but optimize them after altering the architecture. For Q3R, following their experiments, the regularization term is added to the gradients in a decoupled form. We sweep over both the regularization strength and their so-called “target rank”, fixing its refresh interval to 5 (following their main experiments). We note, however, that in an ablation, we found that higher (and cheaper) refresh intervals can also work well in the continued training setting (see Table 10 for overhead measurements and Appendix F for the Q3R ablation). While the authors of Q3R recommend smaller regularization strengths for their settings (λ ∈ [0.001, 0.01]), we found in our initial experiments that higher values provided substantially better performance. 7 This is easy to see: ∥αW ∥2 /∥αW ∥2 = α2 ∥W ∥2 /α2 ∥W ∥2 = ∥W ∥2 /∥W ∥2 . ∗ ∗ ∗ F F F

6

75

ResNet-50 Uniform

75

ResNet-50 Energy

75

50

50

25

25

25

ImageNet-1K Top-1 Accuracy (%)

50 00.0

75 50 25 00.0 60 40 20 00.0

0.2

0.4

0.6

0.8

Parameters ratio

ViT-B/16 Energy

0.2

0.4

0.6

0.8

Parameters ratio

1.0 00.0 75 50 25 1.0 00.0

ResNet-18 Energy

0.2

0.4

0.6

FLOPs ratio

0.8

LoRITa

0.2

0.4

0.6

0.8

Parameters ratio

ViT-B/16 Energy

0.2

0.4

0.6

FLOPs ratio

0.8

1.0 00.0 60 40 20 1.0 00.0

ViT-L/16 Uniform 75 50 25 1.0 00.0

0.2

0.4

0.6

0.8

Parameters ratio

SLORR-Hoyer

ResNet-50 Energy

0.2

0.4

0.6

FLOPs ratio

0.8

ResNet-18 Uniform

0.2

0.4

0.6

0.8

Parameters ratio

75 50 25 1.0 00.0 60 40 20 1.0 00.0

ViT-L/16 Energy 75 50 25 1.0 00.0

SLORR-Hoyer-D

0.2

0.4

0.6

0.8

Parameters ratio

SLORR-Nuc

ViT-B/16 Uniform

0.2

0.4

0.6

0.8

Parameters ratio

1.0

ResNet-18 Energy

0.2

0.4

0.6

0.8

Parameters ratio

1.0

ViT-L/16 Energy 75 50 25 1.0 00.0

Q3R

0.2

0.4

0.6

FLOPs ratio

0.8

1.0

Baseline

Figure 1: Accuracy–compression curves of regularized ResNet-50, ViT-B/16, ViT-L/16, and ResNet-18 models on ImageNet-1K. The y-axis reports ImageNet-1K validation top-1 accuracy of the compressed model, and the x-axis reports either the retained parameter ratio or the retained FLOPs ratio, as indicated in each panel. Factorization is performed using the energy and uniform rank-selection criteria (see Section 4 for details). Faint curves indicate different hyperparameter selections, while highlighted curves indicate Pareto frontiers. For uniform compression, the parameter and FLOPs ratios are equal up to a small shift, so separate FLOPs ratio panels are omitted. For LoRITa, we follow the approach described in their paper; however, we use a more efficient and practical implementation (their original implementation significantly increases runtime and memory costs). We also adapt it to allow initialization from a trained checkpoint, which is necessary for our continued training setting. We fix N = 2 and vary the regularization strength. For SLORR, we use 6 Polar Express steps (following their recommended setup, which we found to work well; an ablation is presented in Appendix D.1) and sweep over the regularization strength λ. We also include an unregularized baseline following the same training recipe. Details are given in Appendix E. While LoRITa (fixing N = 2) and SLORR have one hyperparameter, Q3R has two that play a strong role together, which makes tuning significantly more difficult. Details regarding selection, which mainly followed a best-effort manual approach, as well as the final configurations, can be found in Appendix E and Appendix G, respectively. Compression. After training, we perform compression using two different approaches. The first is uniform compression, in which every layer retains (approximately) a uniform ratio of parameters and FLOPs across layers. Separately, we perform energy-based truncation, in which a retention threshold ρ is used to compress each layer to the minimum rank such that the weight relative squared Frobenius approximation error is at most 1 − ρ. Details are given in Appendix E. ImageNet-1K continued training. We perform continued training on ViT-B/16 [13] and ResNet50 [12] using pretrained checkpoints on the ImageNet-1K dataset [11]. We train for 30 epochs and compare against Q3R and LoRITa. We use the AdamW optimizer [27], a batch size of 1024, cosine learning rate decay, and linear warmup. For ViT-B/16, we use base learning rates of 10−5 and 5 × 10−5 ; for ResNet-50, we use a base learning rate of 5 × 10−5 . We also use the standard data augmentation protocol for each model. The exact training details and motivation for these hyperparameters are provided in Appendix E. The base learning rates are chosen to be sufficiently small so that an unregularized baseline maintains its pretrained performance. However, we found that Q3R did not perform well on ViT-B/16 with the smaller learning rate, so we also conducted experiments using the larger learning rate. We additionally perform continued training on ViT-L/16 [13], comparing SLORR with LoRITa and an unregularized baseline. Unless otherwise noted, we use the same settings as in the ViT-B/16 experiments, with a learning rate of 10−5 . We use a batch size of 896 because LoRITa caused out-of-memory errors at a batch size of 1024. To provide a matched comparison, we also use a batch size of 896 for SLORR. 7

For SLORR, we used both the SLORR-Nuc and SLORR-Hoyer variants. For ResNets, we also include the SLORR-Hoyer-D variant, which applies regularization in a decoupled way (see Appendix D.2). For ViT-B/16, we tried it preliminarily, but it did not seem to help, so we did not continue exploring it. We did not explore the decoupled variant for SLORR-Nuc.

LoRITa SLORR Q3R(5) Q3R(20) Q3R(50) Q3R(100) Q3R(300)

4 3 2 1

1.05

ViT-L

101

1.04 1.03 1.02

ViT-B

ViT-S

ViT-T

Normalized Peak Memory

Compression results. We report results for the different settings in Figure 1. The accuracy–compression Pareto frontier is highlighted, with fainter lines showing trajectories for each hyperparameter configuration. In general, for all methods, there appears to be an accuracy/compressibility tradeoff. Under our settings, we find that at least one variant of SLORR generally performs close to Q3R, sometimes surpassing it. LoRITa is relatively weak in most of our continued training settings, though it is more effective in the ResNet-18 pretraining setting. Note that, in some circumstances, some runs of different methods underperform the baseline in terms of compressibility (in particular, Q3R seems not to perform well on the ViT-B/16 runs with smaller learning rates; see Table 8 in Appendix G). Interestingly, each exact setting appears to favor different methods, including different SLORR variants, suggesting that there might not be a single one-size-fits-all objective.

Normalized Runtime

ImageNet-1K pretraining. Additionally, we pretrain ResNet-18 for 110 epochs on ImageNet-1K (the rest of the settings are similar to ResNet-50, but we use a base learning rate of 10−3 ). We compare against Q3R and LoRITa and conduct experiments with SLORR-Nuc, SLORR-Hoyer and ViT-H ViT-H SLORR-Hoyer-D. 2.0 Runtime zoom

102

Parameters (Millions)

1.8 1.6 1.4 1.2 1.0

ViT-L ViT-B ViT-S

1.01 ViT-T 1.00

101

102

Parameters (Millions)

Figure 2: Time and memory overhead of different regularizers across ViT model scales. Runtime (left) and peak memory (right) are normalized to unregularized training. For Q3R, the number in parentheses denotes the SVD refresh interval. The inset shows a zoomed view of the normalized-runtime curves near ViT-H scale. Model

ViT-B/16

ResNet-50

4.1.1 Training Overhead for Classification Models

Method SLORR LoRITa Q3R(5) Q3R(50) Q3R(300) SLORR LoRITa Q3R(5) Q3R(50) Q3R(300) SLORR LoRITa Q3R(5) Q3R(50) Q3R(300)

Time ×1.037 ×1.007 ×1.656 ×1.106 ×1.055 ×1.048 ×1.005 ×1.222 ×1.062 ×1.049 ×1.055 ×1.010 ×3.059 ×1.301 ×1.139

Mem. (GB) +0.34 +0.57 +1.14 +1.14 +1.14 +0.09 +0.13 +0.31 +0.31 +0.31 +1.21 +2.02 +4.05 +4.04 +4.04

ViT-L/16 Overhead in our training runs. We measure the overhead in a controlled environment (full details and measurements can be found in Appendix H). In Table 2, we report the Table 2: Training overhead in our runs. We report overhead of different methods8 . We find that normalized time vs. unregularized training and inSLORR consistently maintains a small traincrease in peak GPU memory. For Q3R, the number 9 ing overhead of less than 8% on average . Furin parentheses denotes the SVD refresh interval. All thermore, in these experiments, SLORR conmethods achieve near-zero overhead on ResNet-18. sistently achieves the smallest memory overhead. Although we use a refresh period of 5 in our Q3R experiments, which is expensive, we also note that larger periods, significantly cheaper than what Ghosh et al. [9] used in their main experiments, appear to work well in the settings we tried (as discussed earlier); these timings are included in the table for transparency.

Scaling up: overhead across different ViT scales. We evaluate how regularization overhead scales on modern hardware by measuring the relative time and memory costs on B200 GPUs from ViT-T through ViT-H scale (Figure 2). This reveals a key scaling tradeoff: while Q3R with large 8We remark that, in general, the training overhead is related to different factors. For example, a slow data-loading pipeline might mask a large overhead. Moreover, smaller batch sizes will generally suffer from higher relative overhead, as the regularization time is fixed. Similarly, implementation specifics can slightly alter the exact results. 9 Our implementation of the decoupled variant has a slightly larger overhead. For the default variant, overhead is less than 6% on average

8

Llama 135M Plain SVD

Perplexity

150

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001 =0.00005

100 50 0 0.2

0.4

0.6

Parameters ratio

0.8

Accuracy (%)

0.25

0.50

0.75

Parameters ratio

Accuracy (%)

0.4

ARC Easy

1.00

30

0.25

0.50

0.75

Parameters ratio

1.00

0.8

1.0 0.2

0.50

0.75

Parameters ratio

1.00

0.25

0.50

0.75

Parameters ratio

1.00

0.50

0.75

1.00

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001

1.0 0.2

0.4

OpenBookQA

0

0.50

0.75

Parameters ratio

1.00

0.25

0.50

0.75

Parameters ratio

1.00

0

0.25 35

0.50

0.75

Parameters ratio

1.00

30 0.25

0.50

0.75

Parameters ratio

1.00

25

50

0.50

0.75

Parameters ratio

1.00

0.25

0.50

0.75

Parameters ratio

50

1.0

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001

1.00

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001

60 0.25

0.8

PIQA

60

25 0.25

0.6

Parameters ratio 70

30

20

30

Parameters ratio

0.8

Llama 560M SVD-LLM (whitened)

35

40

0.25

0.6

Parameters ratio

20

30 0.25

0.4

LAMBADA

40

50 40 30

25

0.6

Parameters ratio

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001

HellaSwag

50 40 30

25

Llama 560M Plain SVD

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001 =0.00005

1.0 0.2

ARC Challenge 30

Llama 135M SVD-LLM (whitened)

0.25

0.50

0.75

Parameters ratio

1.00

Figure 3: LLM compression results with SLORR-Hoyer regularization. Colors indicate the SLORR-Hoyer regularization strength λ used during pretraining. λ = 0 corresponds to the unregularized baseline. Top row: FineWeb-Edu validation perplexity for Llama 135M and Llama 560M after compression with either plain SVD or SVD-LLM whitening, as indicated in each panel title. Perplexity values are clipped at 150 for visibility; full tabular results are reported in Appendix K. Middle row: Downstream zero-shot accuracy for Llama 560M compressed with plain SVD. Bottom row: Downstream zero-shot accuracy for Llama 560M compressed with SVD-LLM whitening. periods can be efficient, larger refresh periods are needed to maintain efficiency as model size increases. While large refresh periods work in our settings, we expect sufficiently infrequent refreshes to eventually reduce regularization quality. Although we do not reach that “saturation point” in our scales, this is an important aspect to take into account in terms of scalability.10 4.2

LLM Pretraining

We now study regularized LLM pretraining using SLORR-Hoyer. In particular, we train standard Llama-like [40] transformer models on FineWeb-Edu [41], using the AdamW optimizer and distributed data parallelism (DDP), following standard practice. Exact training details are provided in Appendix I. Our main models are trained on approximately 20 training tokens per parameter, following the compute-optimal scaling rule of Hoffmann et al. [42]. We train 135M and 560M models using both unregularized standard pretraining and SLORR-Hoyer for different regularization strengths λ. We find that, across both scales, SLORR incurs less than 1% training overhead on average; see Appendix L for concrete details. After training, we compress models using both plain SVD and the recent SVD-LLM [18], which uses activation statistics to compute a whitening transformation before compressing each layer, generally enabling stronger compression11 . Plain SVD is known to be impractical for LLMs, as degradation is significant even at very small compression ratios [18]. After compression, we evaluate the models. For both scales, we report validation perplexity on a held-out FineWeb-Edu set of approximately 2M tokens. For the 560M model, we additionally report zero-shot downstream task accuracy using the lm-evaluation-harness library [43]. Downstream tasks include ARC-Easy and ARC-Challenge [44], HellaSwag [45], LAMBADA [46], OpenBookQA [47], and PIQA [48]. Compression results. Results for the 135M and 560M models are presented in Figure 3. The reported parameter ratios are computed over the transformer blocks. In particular, embeddings and the final layer are not compressed, following common practice in this area. We find that SLORRHoyer improves compressibility under both plain SVD and SVD-LLM compression. Plain SVD is particularly brittle for LLMs, as perplexity degrades sharply even when only a small fraction of block parameters is removed. Although SLORR-trained models have slightly higher uncompressed perplexity, they outperform unregularized models after mild compression, e.g., 10% parameter removal, corresponding to a retained parameter ratio of 0.9. The gap widens significantly at lower parameter count ratios. 10While Q3R relies on exact SVDs, it might be an interesting avenue for future work to try approximate SVDs, which might improve scalability. 11We use a slightly modified version. Details are provided in Appendix I.

9

For the 560M model, we observe similar trends in downstream task performance. While the unregularized model quickly suffers from significant degradation, regularized models maintain substantially higher downstream performance in general, even at higher compression ratios, at the cost of a slight hit in uncompressed performance on some tasks. Longer training. As noted above, our main models are trained to the compute-optimal token budget [42]. We also train 135M models for longer, specifically for 4 and 8 times the computeoptimal token budget, to explore whether the gains transfer to longer training. We find that, as models are trained for longer, they become less compressible in our setting, both under unregularized and regularized training. However, regularized models remain more compressible. These results are shown in Appendix J, with tabular values in Appendix K. Visualizing the effect of SLORR on weight spectra. SLORR promotes low rank with the objective of making models amenable to factorization. Naturally, this is directly reflected in the visual properties of weight spectra, where stronger regularization induces faster singular value decay. We include plots for all Llama 135M layers in the appendix (Figure 5), where the spectral concentration effect is apparent. Perhaps surprisingly, there are visible spectral differences in the embedding matrix, but not in the final linear layer. Since the embedding layer was not regularized, this indicates that modifying the spectra of regularized layers can affect other parts of the model.

5

Discussion

Limitations. Weight low-rank regularization may not be effective in every setting. In exploratory experiments, we found that fine-tuning LLMs with low-rank regularization while preserving general capabilities was challenging, suggesting that the data and training setup are important, as one might expect. Additionally, there is generally a tradeoff between final accuracy and compressibility, and excessive regularization can cause training instabilities, particularly for the Hoyer variant. Moreover, although our experiments are significantly more extensive than those of prior work (more than 150 runs on ImageNet-1K), the performance of regularizers can vary depending on the training setting and hyperparameters. Similarly, we did not explore every SLORR variant in every setting. Therefore, although we provide a broad best-effort comparison, our results should not be interpreted as a definitive ranking of all methods. Finally, our theory, while informative, relies on specific assumptions that may not hold in every practical case. Future work. One promising direction is to combine regularization with complementary techniques, such as periodic truncation, similar to Xu et al. [6], or multiple rounds of retraining and compression. Regularization strength schedules, as in Dolatabadi et al. [26], would likely also yield better results. Moreover, the need for orthogonalization in optimization has sparked research into more effective polar-function approximations, e.g., [49], which could be applied to SLORR. Finally, we believe our work could motivate broader research beyond the scope of this paper. Although the main application of this work is to directly regularize weights, the versatility of SLORR may open new avenues in settings where one cannot rely on architectural modifications or the validity of cached quantities. For example, low-rank regularization of activations via the nuclear norm has been shown to enhance generalization [50]; while that work uses SVDs, an efficient version could potentially be computed using our approach. Similarly, although nontrivial, it would be interesting to extend low-rank regularization to encourage activations to lie in a subspace, enabling better use of activation-aware factorization methods. More broadly, an accessible way to regularize rank may motivate work on broader applications of rank regularization beyond those studied here, including settings where encouraging a higher rank may be beneficial. Conclusion. We have introduced SLORR, a framework for directly regularizing weight matrices to be low rank. The idea is simple yet highly versatile: it does not require SVDs, remains efficient, operates directly on the matrix being regularized, and is stateless. Our experiments on ImageNet-1K image classification and LLM pretraining show that this regularizer is effective at producing more compressible models while maintaining training efficiency. More broadly, although we focused on weight compression, we are excited about the avenues that this versatility may open in other use cases. 10

Acknowledgments and Disclosure of Funding We would like to thank Keyu Wang, Philipp Nazari, and Neeratyoy Mallik for helpful discussions and feedback on this work.

References [1] Matan Ben Noach and Yoav Goldberg. Compressing Pre-trained Language Models by Matrix Decomposition. In Kam-Fai Wong, Kevin Knight, and Hua Wu (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 884–889, Suzhou, China, December 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. aacl-main.88. URL https://aclanthology.org/2020.aacl-main.88/. [2] Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’14, pp. 1269–1277, Cambridge, MA, USA, 2014. MIT Press. [3] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up Convolutional Neural Networks with Low Rank Expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014. doi: http://dx.doi.org/10.5244/C.28.88. [4] Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. DRONE: Dataaware Low-rank Compression for Large NLP Models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 29321–29334. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ f56de5ef149cf0aedcc8f4797031e229-Paper.pdf. [5] Jose M. Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pp. 856–867, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964. [6] Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. TRP: Trained Rank Pruning for Efficient Deep Neural Networks. CoRR, abs/2004.14566, 2020. URL https://arxiv.org/abs/2004.14566. [7] Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. CoRR, abs/2004.09031, 2020. URL https://arxiv.org/ abs/2004.09031. [8] Ismail Alkhouri, Xitong Zhang, and Rongrong Wang. Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/ forum?id=1KCrVMJoJ9. [9] Ipsita Ghosh, Ethan Nguyen, and Christian Kümmerle. Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id= ZtzWvNKOCr. [10] Patrik O. Hoyer. Non-negative Matrix Factorization with Sparseness Constraints. J. Mach. Learn. Res., 5:1457–1469, December 2004. ISSN 1532-4435. [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848. 11

[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. doi: 10.1109/CVPR.2016.90. [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy. [14] Ajay Kumar Jaiswal, Yifan Wang, Lu Yin, Shiwei Liu, Runjin Chen, Jiawei Zhao, Ananth Grama, Yuandong Tian, and Zhangyang Wang. From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=fsf7LhbYdf. [15] Gene H. Golub and Charles F. Van Loan. Matrix Computations - 4th Edition. Johns Hopkins University Press, Philadelphia, PA, 2013. doi: 10.1137/1.9781421407944. URL https: //epubs.siam.org/doi/abs/10.1137/1.9781421407944. [16] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278–2324, 1998. URL https://api. semanticscholar.org/CorpusID:14542261. [17] Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=kws76i5XB8. [18] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=LNYIUouhdt. [19] Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4287–4296, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long. 217. URL https://aclanthology.org/2025.naacl-long.217/. [20] Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, 2025. URL https://arxiv.org/abs/2312.05821. [21] M. Fazel, H. Hindi, and S. Boyd. Rank minimization and applications in system theory. In Proceedings of the 2004 American Control Conference, volume 4, pp. 3273–3278 vol.4, 2004. doi: 10.23919/ACC.2004.1384521. [22] Markus Kliegl, Siddharth Goyal, Kexin Zhao, Kavya Srinet, and Mohammad Shoeybi. Trace norm regularization and faster inference for embedded speech recognition RNNs, 2018. URL https://arxiv.org/abs/1710.09026. [23] Naiqi Li, Yuqiu Xie, Peiyuan Liu, Tao Dai, Yong Jiang, and Shu-Tao Xia. Efficient Differentiable Approximation of Generalized Low-rank Regularization. In James Kwok (ed.), Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pp. 5616–5624. International Joint Conferences on Artificial Intelligence Organization, 8 2025. doi: 10.24963/ijcai.2025/625. URL https://doi.org/10.24963/ijcai.2025/625. Main Track. [24] Haoran Qin, Shansita Sharma, Ali Abbasi, Chayne Thrash, and Soheil Kolouri. Low-Rank Prehab: Preparing Neural Networks for SVD Compression, 2025. URL https://arxiv.org/ abs/2512.01980. 12

[25] Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Compression-aware Training of Neural Networks using Frank-Wolfe, 2024. URL https://arxiv.org/abs/2205.11921. [26] Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Chamin P Hewa Koneputugodage, Shamane Siriwardhana, Violetta Shevchenko, Karol Pajak, James Snewin, Gil Avraham, and Alexander Long. NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training, 2026. URL https://arxiv.org/abs/2603.03597. [27] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7. [28] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. ReLoRA: High-Rank Training Through Low-Rank Updates, 2023. URL https://arxiv.org/abs/ 2307.05695. [29] Mikhail Khodak, Neil A. Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and Regularization of Factorized Neural Layers. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=KTlJT1nof6d. [30] Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N. Gomez. Exploring Low Rank Training of Deep Neural Networks, 2022. URL https://arxiv.org/abs/2209.13569. [31] Xiuying Wei, Skander Moalla, Razvan Pascanu, and Caglar Gulcehre. Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers. In The Thirtyeighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.net/forum?id=WxLVYZbIew. [32] Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MXze4H7opg. [33] Jiaxi Li, Lu Yin, Li Shen, Jinjin Xu, Liwu Xu, Tianjin Huang, Wenwu Wang, Shiwei Liu, and Xilu Wang. LOST: Low-rank and Sparse Pre-training for Large Language Models, 2025. URL https://arxiv.org/abs/2508.02668. [34] Huanrui Yang, Wei Wen, and Hai Li. DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rylBK34FDS. [35] Wen Li and Weiwei Sun. Perturbation Bounds of Unitary and Subunitary Polar Factors. SIAM Journal on Matrix Analysis and Applications, 23(4):1183–1193, 2002. doi: 10.1137/ S0895479801394623. URL https://doi.org/10.1137/S0895479801394623. [36] Nicholas J. Higham. The matrix sign decomposition and its relation to the polar decomposition. Linear Algebra and its Applications, 212-213:3–20, 1994. ISSN 0024-3795. doi: https://doi. org/10.1016/0024-3795(94)90393-X. URL https://www.sciencedirect.com/science/ article/pii/002437959490393X. [37] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/. [38] Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=yRtgZ1K8hO. [39] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2017. URL https://arxiv.org/abs/1412.6980. 13

[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023. URL https://arxiv.org/abs/2302.13971. [41] Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024. URL https://arxiv.org/abs/2406.17557. [42] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models, 2022. URL https://arxiv.org/ abs/2203.15556. [43] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The Language Model Evaluation Harness, 07 2024. URL https://zenodo.org/records/ 12608602. [44] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018. URL https://arxiv.org/abs/1803.05457. [45] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472/. [46] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/ P16-1144/. [47] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, OctoberNovember 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1260. URL https://aclanthology.org/D18-1260/. [48] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language, 2019. URL https://arxiv.org/abs/ 1911.11641. [49] Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram Newton-Schulz, 2026. URL https://dao-ailab.github.io/blog/2026/gram-newton-schulz/. [50] Zhenmei Shi, Yifei Ming, Ying Fan, Frederic Sala, and Yingyu Liang. Domain Generalization via Nuclear Norm Regularization. In Conference on Parsimony and Learning (Proceedings Track), 2024. URL https://openreview.net/forum?id=hJd66ZzXEZ. [51] Frank H. Clarke. Optimization and Nonsmooth Analysis. Society for Industrial and Applied Mathematics, 1990. doi: 10.1137/1.9781611971309. URL https://epubs.siam.org/doi/ abs/10.1137/1.9781611971309. 14

[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, HighPerformance Deep Learning Library, 2019. URL https://arxiv.org/abs/1912.01703. [53] Ross Wightman. PyTorch Image Models, 2019. URL https://github.com/huggingface/ pytorch-image-models. [54] TorchVision maintainers and contributors. TorchVision: PyTorch’s Computer Vision library, November 2016. URL https://github.com/pytorch/vision. [55] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2018. URL https://arxiv.org/abs/1706.02677. [56] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations, 2017. URL https://openreview. net/forum?id=Skq89Scxx. [57] David González-Martínez. BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression, 2025. URL https://arxiv.org/abs/2509.25136. [58] Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=BvJkwMhyInm. [59] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 Furious, 2024. URL https://arxiv.org/abs/2501.00656. [60] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/20-074.html. [61] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale, 2019. URL https://arxiv.org/abs/ 1907.10641.

15

A

Backpropagating Through the Regularizers

As noted before, the SLORR regularizers are not differentiable everywhere. In practice, one can informally use “natural” backpropagation rules. We note that, for SLORR-Hoyer, the computed rule simply corresponds to backpropagating through the whole expression (and, when backpropagating through the nuclear norm, using the generalized polar factor). These correspond to minimum-norm choices in the generalized gradient of the corresponding functions, in the sense of Clarke [51, Chapter 2], which we denote by ∂f (W ). Throughout this section, when U and V appear, they refer to the thin SVD of W , as in the rest of the paper. We also assume W ̸= 0, so that ∥W ∥F > 0. First, note that both ∥W ∥∗ and ∥W ∥F are locally Lipschitz. Moreover, ∥W ∥2F is continuously differentiable, with ∇∥W ∥2F = 2W . By [51, Proposition 2.2.4], this implies ∂∥W ∥2F = {2W }. We use Clarke’s notion of regularity [51, Definition 2.3.4]. Continuously differentiable functions and convex functions are regular [51, Proposition 2.3.6]. Therefore, ∥W ∥2F , ∥W ∥∗ , and ∥W ∥2∗ are regular. The nuclear norm is convex. Hence, by [51, Proposition 2.2.7], its Clarke generalized gradient coincides with the subdifferential from convex analysis. Generalized gradient of SLORR-Nuc. The SLORR-Nuc regularizer is the nuclear norm, Lnuc (W ) = ∥W ∥∗ . Let

W = U ΣV ⊤

be the thin SVD of W . The generalized gradient is  ∂∥W ∥∗ = U V ⊤ + Q : U ⊤ Q = 0, QV = 0, ∥Q∥2 ≤ 1 . Since Q is Frobenius-orthogonal to U V ⊤ , the minimum-norm element is obtained by taking Q = 0. Thus, the backpropagation choice is ∇Lnuc (W ) = U V ⊤ ∈ ∂Lnuc (W ). Generalized gradient of SLORR-Hoyer. can be written as

The Hoyer-type SLORR regularizer is not convex. It

Lhoyer (W ) = where

a(W ) = ∥W ∥2∗ ,

a(W ) , b(W )

b(W ) = ∥W ∥2F .

Since W ̸= 0, we have a(W ) > 0 and b(W ) > 0. Moreover, since ∥W ∥2∗ is convex (and locally Lipschitz), its generalized gradient corresponds to the convex subdifferential [51, Proposition 2.2.7]: ∂a(W ) = 2∥W ∥∗ ∂∥W ∥∗ . Also, ∇b(W ) = 2W . By the quotient rule for generalized gradients [51, Proposition 2.3.14],   a(W ) b(W )∂a(W ) − a(W )∇b(W ) ∂ = . b(W ) b(W )2 16

Note that equality holds because a and −b are regular, and because a(W ) > 0 and b(W ) > 0. Substituting, we obtain  
∥W ∥2∗ ∥W ∥∗ ⊤ ⊤ UV + Q − 2 W : U Q = 0, QV = 0, ∥Q∥2 ≤ 1 . ∂Lhoyer (W ) = 2 ∥W ∥2F ∥W ∥4F The Q term is again orthogonal to the remaining terms. Hence, the minimum-norm element is again obtained by taking Q = 0. Thus, the backpropagation choice is ∇Lhoyer (W ) = 2

∥W ∥∗ ∥W ∥2∗ UV ⊤ − 2 W ∈ ∂Lhoyer (W ). 2 ∥W ∥F ∥W ∥4F

Equivalently, ∥W ∥∗ ∇Lhoyer (W ) = 2 ∥W ∥F



1 ∥W ∥∗ UV ⊤ − W ∥W ∥F ∥W ∥3F

 .

We then use these expressions when backpropagating through the regularizers. We remark that, intuitively, these correspond to backpropagating individually through the composition of functions in PyTorch, and setting the gradient of the nuclear norm to the generalized polar factor. We use this specific choice because it is the one that is generally (approximately) computed by polynomial approximations such as Polar Express.

B

Analysis of SLORR-Hoyer

Here, we provide proofs and additional discussion of the results shown in Section 3.1. We restate the results for the convenience of the reader. We remark, following Section 3.1, that our analysis is performed in a regularizer-only gradient descent setting. η > 0 denotes the step size of the regularizer-only update, with the training-time regularization strength absorbed into it. We let W = U diag (σ)V ⊤ be a weight matrix and its thin SVD, with singular values σ. Additionally, let W + be the weight matrix after a gradient step from W , and define σ + , U + , and V + as its thin SVD. We assume ordered singular values in decreasing order (σ1 ≥ · · · ≥ σr > 0) and W ̸= 0. Proposition 3.2 (SLORR-Hoyer shrinkage and growth of singular values). For a sufficiently small regularizer-only gradient step, it holds that σi+ ⋄ σi ⇐⇒ σi ⋄

∥W ∥2F , ∥W ∥∗

for ⋄ ∈ {<, >, =}.

Proof. Define C = 2∥W ∥∗ /∥W ∥F for brevity. First, write our gradient update as W + = W − η∇Lhoyer (W )   C C∥W ∥∗ =W −η UV ⊤ − W ∥W ∥F ∥W ∥3F C C∥W ∥∗ = U ΣV ⊤ − η UV ⊤ + η U ΣV ⊤ ∥W ∥F ∥W ∥3F   C C∥W ∥∗ =U Σ−η I +η Σ V ⊤. ∥W ∥F ∥W ∥3F Since the matrix in the “sandwich” is diagonal (as it is a sum of two diagonal matrices), we can conclude that, for small enough updates such that no inner term becomes negative, σi+ = σi − η

C∥W ∥∗ C +η σi . ∥W ∥F ∥W ∥3F

Moreover, note that the order of singular values is preserved. 17

Now, C C∥W ∥∗ σi ⋄ σi +η ∥W ∥F ∥W ∥3F C∥W ∥∗ C +η σi ⋄ 0 ⇐⇒ −η ∥W ∥F ∥W ∥3F C C∥W ∥∗ ⇐⇒ − + σi ⋄ 0 (η > 0) ∥W ∥F ∥W ∥3F 1 ∥W ∥∗ σi ⋄ 0 (C > 0) ⇐⇒ − + ∥W ∥F ∥W ∥3F

σi+ ⋄ σi ⇐⇒ σi − η

⇐⇒ −∥W ∥2F + ∥W ∥∗ σi ⋄ 0 ⇐⇒ ∥W ∥∗ σi ⋄ ∥W ∥2F ⇐⇒ σi ⋄

∥W ∥2F , ∥W ∥∗

for ⋄ ∈ {<, >, =}. This concludes the proof.

Proposition 3.3 (Sufficiently small steps of SLORR-Hoyer induce compressibility). Let τ = ∥W ∥2F /∥W ∥∗ . For a sufficiently small regularizer-only gradient step, if σk > τ , then + + +⊤ ∥U:,:k diag(σ:k )V:,:k − W + ∥2F

∥W + ∥2F

<

⊤ ∥U:,:k diag(σ:k )V:,:k − W ∥2F . ∥W ∥2F

Proof. For brevity, define ε=η

2∥W ∥2∗ . ∥W ∥4F

Note that (see the proof of Proposition 3.2) σi+ = σi − η

2∥W ∥∗ 2∥W ∥2∗ + η σi . ∥W ∥2F ∥W ∥4F

Then η

2∥W ∥∗ 2∥W ∥2∗ ∥W ∥2F =η · = ετ. 2 ∥W ∥F ∥W ∥4F ∥W ∥∗

Therefore, 2∥W ∥∗ 2∥W ∥2∗ + η σi ∥W ∥2F ∥W ∥4F = σi − ετ + εσi = σi + ε(σi − τ ).

σi+ = σi − η

For convenience, now define A+ =

X

=

X

2

(σi + ε(σi − τ )) =

X  σi2 + 2εσi (σi − τ ) + ε2 (σi − τ )2 i≤k

i≤k

σi2 +

i≤k

X X   2εσi (σi − τ ) + ε2 (σi − τ )2 = A + 2εσi (σi − τ ) + ε2 (σi − τ )2 , i≤k

i≤k

| {z } A

18

B+ =

X

=

X

2

(σi + ε(σi − τ )) =

i>k

X  σi2 + 2εσi (σi − τ ) + ε2 (σi − τ )2 i>k

σi2 +

i>k

X X   2εσi (σi − τ ) + ε2 (σi − τ )2 = B + 2εσi (σi − τ ) + ε2 (σi − τ )2 . i>k

i>k

| {z } B

The A term.

Note that for the A term, (σi − τ ) > 0,

as required in the statement of this result. Hence, A+ − A = εa1 + ε2 a2 (as every other term is nonnegative), where a1 , a2 > 0. The A + B term. First note

We will show that, for sufficiently small ε, the A + B increases less than A. !

X

X

2εσi (σi − τ ) = 2ε

i

σi2 − τ

i

X

σi

= 2ε

X

i

i

∥W ∥2F X σi σi2 − ∥W ∥∗ i

! = 0.

Now, A+ + B + − (A + B) =

X  X 2  2εσi (σi − τ ) + ε2 (σi − τ )2 = ε (σi − τ )2 = bε2 , i

i

with b > 0. As the A term has a strictly positive linear term in ε, while the total increment is only quadratic, taking ε small enough gives A+ − A > (A+ + B + ) − (A + B). Also note that A+ − A > 0. Hence, A+ A > . + + A +B A+B Final step.

Now, separate the sums P P + 2 + 2 A+ i≤k (σi ) i≤k (σi ) , P + 2 =P P + 2 + 2 = + A + B+ i (σi ) i≤k (σi ) + i>k (σi ) P P 2 2 A i≤k σi i≤k σi P 2 =P P 2+ 2 = A + B. σ σ σ i i i≤k i i>k i

Finally, it follows that 1−

+ + +⊤ ∥U:,:k diag(σ:k )V:,:k − W + ∥2F

∥W + ∥2F

P + 2 2 ⊤ ∥U:,:k diag(σ:k )V:,:k − W ∥2F i≤k (σi ) i≤k σi . P + 2 > P 2 = 1− ∥W ∥2F i σi i (σi )

P =

The result follows from subtracting 1 and swapping sides.

C

Approximation Error Guarantees

We restate the result for the convenience of the reader. Proposition 3.1 (Approximation guarantees). Let W ̸= 0 have thin SVD W = U ΣV ⊤ , and assume the nonzero singular values of a normalized version of W lie in [ℓ, 1]. Let Pb denote the Polar Express approximation of the polar factor U V ⊤ after T steps, and define δ = 1 − ℓ2 19

(q+1)T

, where

d = 2q + 1 is the Polar Express polynomial degree. Additionally, let n̂ := tr(W ⊤ Pb ) denote the approximate nuclear norm estimate, and let n := ∥W ∥∗ and f := ∥W ∥F . Moreover, let R denote b its approximation. Then the exact SLORR-Hoyer gradient and R |n̂ − n| ≤ δ, ∥Pb − U V ⊤ ∥2 ≤ δ (SLORR-Nuc), n   2 |n̂2 /f 2 − n2 /f 2 | b 2 ≤ 2δ(2 + δ) n + n ∥W ∥2 ≤ δ(2 + δ), ∥R − R∥ (SLORR-Hoyer). n2 /f 2 f2 f4 In particular, errors vanish as T → ∞. Proof. Given W , by assumption, a normalized version M satisfies σ(M ) ⊆ [ℓ, 1] (where only the nonzero singular values are taken into account). By Theorem 3.3 of Amsel et al. [38], Pb − U V ⊤

≤ 1 − ℓ2

(q+1)T

,

2

where T is the number of Polar Express steps (6 in our main experiments) and d = 2q +1 is the degree of the Polar Express polynomial (5 in our experiments). We can now extend it to the approximation of the nuclear norm. In particular, fix the PE parameters and let δ = 1 − ℓ2

(q+1)T

.

First, we have   |n̂ − n| = tr W ⊤ (Pb − U V ⊤ ) ≤ ∥W ∥∗ Pb − U V ⊤

2

≤ δ ∥W ∥∗ . Therefore, |n̂ − n| ≤ δ. n Similarly, we can bound the error in the SLORR-Hoyer value. Using |n̂| ≤ n + |n̂ − n| ≤ (1 + δ)n, we obtain n̂2 − n2 = |n̂ − n| |n̂ + n| ≤ |n̂ − n| (|n̂| + n) ≤ δn ((1 + δ)n + n) = n2 δ(2 + δ). Therefore, we get (dividing by n2 ) n̂2 − n2 ≤ δ(2 + δ), n2 or equivalently, n̂2 /f 2 − n2 /f 2 ≤ δ(2 + δ). n2 /f 2 Now, we can bound the error on the SLORR-Hoyer gradient. Recall that (for the exact polar factor):   n 1 n ⊤ R=2 UV − 3 W , f f f and note that its Polar Express approximation is   b = 2 n̂ 1 Pb − n̂ W . R f f f3 20

We have b R−R

n = 2 f 2



1 n UV ⊤ − 3 W f f



n̂ −2 f



1 b n̂ P − 3W f f

2n 2n̂ b 2n̂2 2n2 ⊤ = U V W − − P+ 4 W f2 f4 f2 f

 2

2

2n̂ 2n 2n2 2n̂ 2n̂ b 2n̂2 ⊤ ⊤ ⊤ + − ≤ U V − U V U V − + W + W P f2 f2 f2 f2 f4 f4 2 2

2

2n 2 |n̂| 2δ(2 + δ)n2 2δ(n + |n̂|) 2δ(2 + δ)n2 ≤δ 2 + 2 δ+ ∥W ∥2 ≤ + ∥W ∥2 . 4 f f f f2 f4 Now, to get rid of n̂, note that |n̂| ≤ n + |n̂ − n| ≤ (1 + δ)n. Hence,

2δ(n + (1 + δ)n) 2δ(2 + δ)n2 + ∥W ∥2 f2 f4 2 2δ(2 + δ)n 2δ(2 + δ)n2 = + ∥W ∥2 f2 f4   n n2 = 2δ(2 + δ) + 4 ∥W ∥2 . f2 f In particular, as T → ∞, the error tends to 0, as expected. b R−R

≤

Remark about zero singular values. The Polar Express guarantee we use only concerns the nonzero singular subspace, corresponding to the generalized polar factor (i.e., using the thin SVD) used throughout this paper. Exact zero singular values are therefore excluded from the result. In exact arithmetic, the Polar Express updates do not create components in this zero singular subspace, since the scalar polynomial maps used in their iteration satisfy p(0) = 0. Thus, zero singular values remain zero under the idealized iteration, and hence our approximation guarantee concerns only the nonzero singular values. Remark about normalization and the lower bound parameter. The singular value range [ℓ, 1] is inherited from the analysis in Amsel et al. [38]. For a nonzero matrix W , we apply Polar Express to the normalized matrix W f = W . ∥W ∥F + ε This positive rescaling leaves the polar factor unchanged and ensures that the largest singular value f is at most one. However, it does not by itself guarantee that every nonzero singular value is of W at least ℓ. The parameter ℓ should therefore be interpreted as the lower bound parameter used by Polar Express. If the actual smallest nonzero normalized singular value is below ℓ, the displayed worst-case bound does not apply literally to that singular direction. We note that Amsel et al. [38] mention that inaccurate lower bound guesses are typically not severe in practice. Remark about practical considerations. The guarantee above is a worst-case bound for the idealized Polar Express iteration and should not be read as a tight estimate of the implemented approximation error. In our experiments, we use a small number of Polar Express iterations together with the practical adaptations of Amsel et al. [38]. These choices slightly deviate from the idealized theorem, and floating point arithmetic can introduce additional error. Empirically, this setting works well in our experiments; see Appendix D.1 for an ablation and implementation details.

21

D

General Experimental and Implementation Details

The code used to produce our experimental results is available at https://github.com/ davidgonmar/SLORR. It includes the implementation of our method and baselines, as well as the scripts, hyperparameters, and configurations used for our experiments. Our code for both the vision and LLM experiments is based on the PyTorch framework [52]. D.1

On our Use of Polar Express

Polar Express has several tunable knobs. The first is the lower bound on singular values, ℓ, along with the degree and number of iterations. Following Implementation 1 in Amsel et al. [38], we set ℓ = 10−3 and the degree to d = 5. They note that choosing ℓ too small can delay convergence for smaller singular values, but that inaccurate guesses are typically not severe in practice. We found these values to work well in our experiments. We use bfloat16, as recommended. An important hyperparameter of Polar Express is the number of iterations. The authors recommend 5 or 6 iterations. Throughout our experiments, we used 6 Polar Express iterations, which add minimal overhead. In early experiments, we found that this quantity worked well. Here, we explicitly ablate the number of iterations. Following our ViT-B/16 experiments, we fix λ = 0.002 and repeat our uniform compressibility experiments using runs with {1, 2, 4, 6, 8, 10} Polar Express iterations. Results are presented in Table 3. In this setting, after 6 iterations, results across compression ratios change little, suggesting diminishing returns from additional Polar Express iterations in this setting. Table 3: Effect of Polar Express iterations (ViT-B/16). We repeat the main ViT-B/16 SLORR-Hoyer experiment with λ = 0.002 while varying only the number of Polar Express iterations. Columns denote retained parameter ratios, and entries are ImageNet-1K top-1 accuracy. Steps 21% 31% 41% 51% 60% 70% 80% 90% Full 1 2 4 6 8 10

D.2

0.1 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.1 0.1 0.1

0.6 0.2 0.1 6.6 5.2 7.0

14.4 0.3 0.2 66.2 67.7 68.9

49.0 1.6 3.8 79.5 79.5 79.4

67.9 32.3 18.6 81.1 80.9 80.9

77.1 63.7 82.6 81.5 81.3 81.2

81.0 72.6 83.3 81.7 81.5 81.4

84.9 83.4 83.7 81.7 81.6 81.5

Implementation Details for the Decoupled Variant

Algorithm 1 details how the gradient of the SLORR regularizer can be applied in a decoupled form. This follows the rationale of AdamW [27], where weight decay is applied based on the pre-update weights. SLORR-Hoyer-D follows this approach, but note that this approach can also be applied to SLORR-Nuc. Note that, in practice, the regularizer value (not only the gradient) can also be computed (at the same time as the gradient) and, for example, logged. Algorithm 1 Decoupled SLORR update Require: Weight set W, regularized weight set Wreg ⊆ W, learning rate η, regularization strength λ Require: Task gradients ∇W Ltask 1: for all W ∈ W \ Wreg do 2: if ∇W Ltask exists then 3: Apply AdamW update to W using ∇W Ltask 4: end if 5: end for 6: for all W ∈ Wreg do b pre b 7: G SLORR ← GSLORR (W ) 8: if ∇W Ltask exists then 9: Apply AdamW update to W using ∇W Ltask 10: end if b pre 11: W ← W − ηλG ▷ uses the pre-update SLORR gradient SLORR 12: end for

22

D.3

Implementation Details for the Memory-Efficient Variant

Algorithm 2 details a more memory-efficient approach for PyTorch environments. When implemented as a loss function (i.e., not in decoupled form, as detailed in Appendix D.2), using an autograd function is easy to implement. However, if one computes each layer-wise loss, sums them, and then backpropagates, PyTorch keeps intermediate values for all layers. This can consume additional memory proportional to the total size of the regularized layers. Again, this can be applied to both SLORR-Nuc and SLORR-Hoyer. Similar to Appendix D.2, the regularizer value can also be computed at the same time, if needed. Algorithm 2 Iterative memory-efficient SLORR implementation Require: Regularized weight set Wreg , regularization strength λ 1: for all W ∈ Wreg do b SLORR (W ) from the current value of W 2: Compute G 3: if W .grad exists then b SLORR (W ) 4: W .grad ← W .grad + λG 5: else b SLORR (W ) 6: W .grad ← λG 7: end if 8: end for

23

E

Vision Experimental and Implementation Details

For the vision experiments, we use standard ImageNet training pipelines built on the model implementations provided by timm [53]. E.1

Interpreting Convolutional Layers

For factorization and regularization, we interpret each convolutional kernel as a matrix by reshaping W ∈ RCo ×Ci ×Hk ×Wk as reshape(W; (Co , −1)). A low-rank factorization under this interpretation corresponds to replacing the original convolution with two sequential convolutions. This is the standard channel-wise decomposition used in prior low-rank factorization work, including LoRITa. E.2 E.2.1

Implementation Details for Baselines LoRITa

We implement LoRITa largely as described in Alkhouri et al. [8], with a few adaptations. We fix N = 2, where N denotes the number of factorized layers used to replace each original layer. Since we initialize from pretrained checkpoints, we choose the factors so that the initial network is functionally equivalent to the original model. Specifically, for a weight matrix W , we compute its thin SVD and initialize W0 = U Σ1/2 , W1 = Σ1/2 V ⊤ . This corresponds to the “spectral initialization” of Khodak et al. [29] in the case where all singular values are preserved. For ResNet-18 pretraining, we first initialize each corresponding dense layer using the same random initializer as in the dense baseline. We then compute its factorization and initialize the LoRITa factors using the spectral initialization above. This matches the initial function of the dense baseline while using LoRITa’s factorized parameterization, allowing us to control for initialization effects. Following Ghosh et al. [9] and Alkhouri et al. [8], we implement the LoRITa regularizer through the optimizer’s weight decay term. The value reported for LoRITa in the result tables is this coefficient. As we use AdamW, it is applied in a decoupled way. Parameters that are not replaced by LoRITa factors use the default weight decay for the corresponding experiment, while normalization parameters and biases are not decayed. This keeps the LoRITa penalty separate from the ordinary optimizer weight decay used for the rest of the model. For convolutional layers, we follow the construction described in Alkhouri et al. [8], which coincides with the general interpretation of convolutional layers discussed in Appendix E.1. We make one implementation modification to improve memory efficiency, which allows us to run LoRITa on larger models. In the original sequential implementation12 , LoRITa replaces a layer by a sequence of factorized layers. With N = 2, this introduces an additional intermediate activation and can substantially increase memory use at realistic batch sizes. Even when memory is sufficient, the sequential implementation can remain computationally expensive; see the LoRITa authors’ comments at https://openreview.net/forum?id=1KCrVMJoJ9. We describe the modification for a linear layer without bias; the convolutional case follows the same matrix interpretation as in Appendix E.1. Let the original layer be parameterized by W ∈ RI×O , and let R = min{I, O}. LoRITa replaces this layer with two trainable matrices, W0 ∈ RI×R ,

W1 ∈ RR×O .

For an input X ∈ RB×I , with batch and token dimensions folded into a single effective batch dimension, the original computation is Y = XW , while the sequential LoRITa implementation computes Y = (XW0 )W1 . 12 https://github.com/XitongSystem/LoRITa

24

Instead, in each forward pass we first merge the factors and then apply the merged matrix: Y = X(W0 W1 ). This avoids storing the intermediate activation XW0 and reduces LoRITa’s overhead to effectively zero in our experiments. The merged computation defines the same function of W0 and W1 as the sequential implementation, so the forward pass and gradients are equivalent up to floating point differences. Moreover, the original LoRITa implementation uses R = I. We use R = min{I, O}, which is the natural full-rank dimension for the matrix factorization above and enables the checkpoint-preserving initialization described here. The main theoretical results supporting LoRITa still apply (see Section 4.3 of Alkhouri et al. [8]). We restrict LoRITa to N = 2. Extending the initialization to N > 2 would require choosing how to split the singular values across more than two factors, introducing an additional experimental choice. Since the LoRITa paper reports only marginal ImageNet ResNet gains from N = 3, including for ResNet-18, we keep N = 2 throughout and leave larger N as a limitation of our study. E.2.2

Q3R

Following the authors’ recommended setup, we use their AdamQ3R optimizer. AdamQ3R combines Adam-style optimization with the Q3R regularizer, applying the regularization effect in a decoupled form motivated by AdamW [27]. We copied the optimizer and regularizer implementation from the authors’ official codebase13 and left the linear-layer path unchanged, except that we added support for decoupled weight decay to match our AdamW baselines. We use a refresh interval of 5 for all main experiments, unless noted otherwise, following the setting used by the Q3R authors in their main experiments. We also found that larger, cheaper refresh intervals can work well in some settings; see Appendix F. The original implementation does not support convolutional layers. We extend it to convolutions by interpreting convolutional layers as matrices, as described in Appendix E.1. This extension only inserts the necessary reshape operations and leaves other parts of the code untouched. When we refer to Q3R’s “target rank”, we mean the value passed to the AdamQ3R optimizer. This value can be a ratio between 0 and 1 rather than an absolute rank. E.3

Training Details and Hyperparameters

We first describe the general training setup and then list specific hyperparameters for each model. Regularization and factorization are applied to layers in their native parameterization. In particular, the timm ViT implementations use fused QKV layers, and we regularize and factorize these fused matrices without splitting them. This follows LoRITa’s ViT implementation. Q3R’s experiments split QKV layers, but we note this is an experimental choice rather than a mechanism inherent to the method (we remark that, in any case, this choice must be consistent across all methods). Hence, for simplicity, we use the default fused QKV implementation. We did not explore split QKV layers. We keep non-regularizer hyperparameters fixed across methods whenever possible. For LoRITa, the factor weight decay is part of the method and is varied as its regularization strength. For Q3R, our AdamQ3R implementation supports the same decoupled weight decay used by the AdamW baselines and SLORR runs. Across methods, ordinary weight decay is applied only to weight matrices; normalization parameters and biases are not decayed. For continued training experiments, we choose learning rates so that an unregularized baseline largely preserves the accuracy of the pretrained checkpoint. On ViT-B/16, Q3R tended to perform poorly with the smaller learning rate, so we tested a larger one. LoRITa also improved slightly in that setting, so we include both learning rates. We did not explore this higher-learning-rate setting for other models or for ResNet-18 pretraining (for ResNet-18, the learning rate was again chosen based on unregularized performance). We remark that even though our sweeps are significantly more extensive than those of prior work, differences in training configurations might benefit different techniques. Accordingly, our comparisons 13 https://github.com/ThatE10/Q3R

25

should be interpreted as a broad best-effort study under mostly matched training recipes rather than as an exhaustive sweep, as mentioned in Section 5. All runs use bfloat16 automatic mixed precision (weights are stored in float32). Checkpoints. For runs starting from checkpoints, we use the default ones on timm, except for ResNet-50 for which we use the TorchVision [54] checkpoint, which is the most common one. Hyperparameters for ViT-B/16. For ViT-B/16, we use learning rates of 1 × 10−5 and 5 × 10−5 , a batch size of 1024, a weight decay of 0.1, and a cosine schedule with 1 epoch of linear warmup [55, 56] with an initial factor of 10−3 , followed by decay to 0.1 times the original learning rate. We use standard ViT-B/16 augmentation through the timm library. We detail the augmentation parameters in Table 4. Hyperparameters for ViT-L/16. We use the same setup as for ViT-B/16, but with a batch size of 896. With a batch size of 1024, LoRITa produced out-of-memory errors, even with our optimizations. We therefore use a batch size of 896 for all methods in this setting to ensure comparable conditions. We only used a learning rate of 1 × 10−5 . Hyperparameters for ResNet-50. For ResNet-50, we use a learning rate of 5 × 10−5 . We use the same scheduler as in the ViT experiments, a batch size of 1024, and a weight decay of 0.01. We apply standard data augmentation, consisting of random resizing and cropping, followed by horizontal flipping with probability 0.5, and normalization using the standard ImageNet-1K mean and standard deviation. Hyperparameters for ResNet-18 (training from scratch). We use a learning rate of 10−3 and train for 110 epochs. The rest is the same as for ResNet-50. All vision experiments use A100 GPUs, except ViT-L/16 ones, where we use B200 GPUs. Method hyperparameter selection. Method-specific hyperparameters were selected through exploratory runs and manual coarse-to-fine tuning. SLORR and LoRITa each have a single main regularization coefficient, whereas Q3R has two interacting hyperparameters, excluding the refresh interval, making exhaustive grid search impractical. For Q3R, we found that values larger than the authors’ recommended range of [0.001, 0.01] worked substantially better in our settings. For all methods, stronger regularization generally improved compressibility but could reduce uncompressed accuracy or cause training collapse. Conversely, too little regularization had little effect on compressibility. Runs were discarded only when they collapsed or produced unusable uncompressed accuracy, not based on post-compression performance. All included configurations are listed in Appendix G. E.4

Factorization Details

Our factorization implementation is adapted from the BALF [57] repository. For every model, we factorize all layers except the first and last, typically the classifier. As mentioned in the main text, we perform experiments with both uniform truncation and energy truncation. We describe both below; our implementation is included in our code. Uniform truncation. Given a target retention ratio c ∈ (0, 1], with c = 1 denoting no compression, each eligible layer is truncated to approximately retain a fraction c of its original parameters. Under our matrix interpretation, the same rank choice also gives approximately the same retained FLOPs fraction for both linear and convolutional layers. Thus, uniform truncation applies the same compression ratio across eligible layers, although different layers might still retain different absolute numbers of parameters and FLOPs. Energy truncation. is defined as

Given a single matrix, the retained energy metric when keeping up to rank P PP 2 c ∥2 ∥W − W F i=1 σi E(P ) = PR =1− , 2 2 ∥W ∥ σ F i=1 i 26

Table 4: Training augmentation for the ViT-B runs. Component

Parameter

Value

create_transform

input_size is_training color_jitter auto_augment interpolation re_prob re_mode re_count mean std

(3, 224, 224) True 0.3 "rand-m9-mstd0.5-inc1" "bicubic" 0.25 "pixel" 1 (0.5, 0.5, 0.5) (0.5, 0.5, 0.5)

Mixup

mixup_alpha cutmix_alpha prob switch_prob mode label_smoothing num_classes

0.8 1.0 1.0 0.5 "batch" 0.1 1000

c denotes the reconstruction where R = min{M, N }, M and N are the dimensions of W , and W of the compressed matrix. Similar formulations are used in Alvarez & Salzmann [5], Yang et al. [34], Liebenwein et al. [58]. Given a fixed retained energy threshold ρ, the energy criterion selects a per-layer rank P ∗ = min{P : E(P ) ≥ ρ}, i.e., the lowest rank that maintains a relative squared error of at most 1 − ρ. In particular, for each model, we sweep {0.7, 0.75, 0.8, 0.85, 0.9, 0.925, 0.95, 0.975, 0.99, 0.995, 0.999, 0.9995, 0.9999, 0.99995, 0.99996, 0.99997, 0.99998, 0.99999}.

F

Q3R Refresh Period Ablation

Throughout our experiments, we use a refresh period of 5 for Q3R, following their main experiments. Sweeping the period in addition to the target rank and regularization strength would have been impractical, so we use their recommended configuration. However, for completeness, we also swept the refresh interval over a small number of runs to probe whether one could use a larger period, which is more efficient, and what the corresponding accuracy tradeoff would be. We limit ourselves to ViT-B/16. Results are presented in Table 5. Q3R appears to be surprisingly robust in the cases we tested. Note, however, that this might differ in other models, scales, or training horizons.

G

Additional Vision Model Results

Here, we report the complete uniform compression results for the vision models. Results are shown for ViT-B/16 in Table 8, ResNet-50 in Table 9, ViT-L/16 in Table 6, and ResNet-18 in Table 7. For each table, the different columns indicate the retention ratio (and Full indicates the uncompressed model). Top results are bolded, and second-top results are underlined. For the ViT-B/16 table, where we swept learning rates, ⋄ is used for 10−5 , and † for 5 × 10−5 . Notation. For Q3R, the first argument denotes its regularization strength, and the second its target rank. For LoRITa, its only argument denotes the regularization strength (in the form of weight decay). For all SLORR variants, the only argument denotes regularization strength. 27

Table 5: Q3R refresh interval ablation on ViT-B/16. ImageNet-1K top-1 accuracy under uniform retained parameter ratios and different configurations. Setting

Interval

21%

31%

41%

51%

60%

70%

80%

90%

Full

λ = 8.0 target rank 0.05 learning rate 5 × 10−5

5 20 50 100 300

29.6 34.8 33.3 32.9 44.4

75.2 75.2 75.1 75.0 74.8

76.0 75.8 75.8 75.8 75.5

76.1 76.1 76.1 76.0 75.8

76.2 76.3 76.3 76.2 75.9

76.3 76.4 76.3 76.2 76.0

76.3 76.4 76.3 76.3 76.0

76.3 76.4 76.4 76.3 76.0

76.3 76.4 76.4 76.3 76.0

λ = 3.0 target rank 0.15 learning rate 5 × 10−5

5 20 50 100 300

0.1 0.1 0.1 0.1 0.1

0.2 0.2 0.1 0.2 0.2

21.2 19.3 20.1 18.7 22.1

71.9 72.0 72.2 72.0 72.2

80.7 80.7 80.8 80.8 80.6

82.5 82.5 82.5 82.6 82.6

82.7 82.7 82.8 82.8 82.7

82.8 82.8 82.8 82.8 82.8

82.8 82.8 82.9 82.9 82.9

λ = 5.0 target rank 0.10 learning rate 1 × 10−5

5 20 50 100 300

0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.2 0.1

9.4 10.0 8.7 9.1 8.6

53.5 53.5 54.1 47.5 48.8

73.0 73.0 72.8 72.6 70.9

77.3 77.3 77.2 77.4 77.2

78.7 78.7 78.7 78.7 78.5

81.0 81.1 81.0 81.1 81.0

Table 6: Complete uniform compression results for ViT-L/16 continued training. Method

10% 20% 30% 40% 50% 60% 70% 80% 90% 95% Full

Baseline

0.2

0.6

7.5

38.9 63.5 75.2 80.6 83.2 84.4 84.8 85.8

LoRITa(0.0001) LoRITa(0.001) LoRITa(0.01) LoRITa(0.05) LoRITa(0.3) LoRITa(1)

0.1 0.1 0.1 0.1 0.2 0.2

0.7 0.7 0.7 0.7 0.4 0.2

12.5 12.5 12.7 11.5 7.5 0.3

45.8 45.8 44.9 44.6 38.8 3.6

67.2 67.2 66.8 66.7 63.9 36.4

76.9 76.9 76.9 76.8 75.8 66.5

81.3 81.3 81.2 81.2 80.7 77.0

83.5 83.5 83.4 83.4 83.0 81.1

84.5 84.5 84.5 84.4 84.3 83.1

84.9 84.9 84.8 84.8 84.5 83.7

85.5 85.5 85.5 85.5 85.5 85.1

SLORR-Nuc(0.00002) SLORR-Nuc(0.0002) SLORR-Nuc(0.001) SLORR-Nuc(0.00005)

0.2 0.2 0.1 0.2

0.6 0.4 0.6 0.6

9.1 8.5 12.2 7.8

42.4 43.8 54.0 41.5

65.7 70.0 77.0 66.6

76.5 79.1 80.3 77.1

81.3 82.5 81.1 81.6

83.5 84.1 81.5 83.8

84.6 84.8 81.5 84.7

84.9 85.0 81.5 84.9

85.7 85.5 81.5 85.7

SLORR-Hoyer(0.00005) SLORR-Hoyer(0.0002) SLORR-Hoyer(0.0005) SLORR-Hoyer(0.001) SLORR-Hoyer(0.0015)

0.1 0.1 0.1 0.1 0.1

0.4 0.2 0.2 0.2 0.2

7.0 7.0 7.9 3.3 5.8

38.2 42.8 52.2 54.7 62.1

64.1 65.5 73.4 78.0 80.1

76.2 77.5 82.0 82.4 82.6

81.4 82.8 84.1 83.6 83.6

83.8 84.6 84.9 84.1 83.9

84.8 85.3 85.1 84.3 84.1

85.2 85.5 85.2 84.3 84.1

85.8 85.9 85.4 84.4 84.1

28

Table 7: Complete uniform compression results for ResNet-18 pretraining. Method

14% 23% 33% 42% 52% 62% 71% 81% 90% 95% Full

Baseline

0.1

0.1

0.2

0.2

1.5

4.4

19.1 25.7 45.8 48.7 66.2

Q3R(1, 0.0025) Q3R(1, 0.01) Q3R(5, 0.01) Q3R(1, 0.025) Q3R(5, 0.025) Q3R(1, 0.05) Q3R(2, 0.05) Q3R(5, 0.05) Q3R(2, 0.1) Q3R(0.01, 0.15) Q3R(2, 0.2)

0.1 0.1 0.3 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.2 0.3 5.3 0.6 0.2 0.3 0.2 0.2 0.1 0.1 0.1

0.9 1.6 18.7 30.8 26.2 1.2 0.9 0.5 0.4 0.3 0.1

4.6 15.7 34.3 53.2 62.1 17.4 18.0 3.5 1.7 1.2 0.4

12.8 32.6 33.2 57.0 63.3 42.5 46.3 19.4 8.4 3.3 0.9

26.9 42.4 47.0 60.0 63.3 64.0 61.4 50.0 24.3 6.0 2.0

39.4 48.6 50.2 62.8 63.3 65.6 64.9 63.7 41.4 21.6 15.9

48.4 54.0 52.7 63.5 63.3 65.9 65.6 65.4 51.2 35.6 28.0

52.2 56.0 53.4 63.6 63.3 65.8 65.7 65.8 59.8 49.4 40.2

55.4 57.8 54.2 63.9 63.3 65.9 65.7 65.9 62.8 55.6 46.2

60.0 62.1 55.2 64.0 63.3 65.9 65.7 65.9 66.3 66.5 59.8

LoRITa(0.0001) LoRITa(0.001) LoRITa(0.01) LoRITa(0.05) LoRITa(0.3) LoRITa(0.5) LoRITa(1)

0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.2 0.2 0.1 0.2 0.1 0.1 0.2

1.0 0.6 0.3 0.3 1.2 1.4 3.8

6.6 7.9 2.5 6.9 23.9 43.0 38.1

26.2 27.0 22.0 37.2 55.5 55.7 48.8

45.6 46.9 43.1 58.7 63.2 63.1 59.5

53.9 59.9 56.8 65.1 65.0 64.0 60.7

62.9 64.0 64.0 67.1 65.8 64.1 60.7

66.2 66.0 65.9 67.5 66.0 64.2 60.7

66.5 66.5 66.6 67.7 66.0 64.2 60.7

67.1 67.0 67.3 67.9 66.0 64.2 60.8

SLORR-Nuc(0.00001) SLORR-Nuc(0.00005) SLORR-Nuc(0.0001) SLORR-Nuc(0.0005) SLORR-Nuc(0.001)

0.1 0.1 0.1 0.1 0.1

0.2 0.3 0.1 0.1 0.1

0.5 0.4 0.3 0.2 0.2

1.5 1.6 1.9 0.2 0.1

5.6 16.0 25.8 51.9 59.1 62.1 17.6 34.8 43.5 62.2 65.8 66.6 11.1 48.3 58.7 63.9 66.0 66.8 0.7 4.9 13.6 43.9 54.5 55.4 0.4 2.2 37.6 52.2 54.4 55.5

66.6 67.9 67.8 60.3 56.2

SLORR-Hoyer(0.0002) SLORR-Hoyer(0.0005) SLORR-Hoyer(0.001) SLORR-Hoyer(0.002) SLORR-Hoyer(0.005) SLORR-Hoyer(0.007) SLORR-Hoyer(0.01)

0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.2 0.2 0.2 0.1 0.1 0.1 0.2

0.3 0.6 0.1 0.1 0.1 0.1 0.2

0.9 0.6 0.6 0.4 0.3 0.2 0.1

3.5 2.1 6.2 0.8 1.5 0.8 0.7

17.2 13.6 13.8 11.3 7.4 5.2 8.5

37.9 31.1 36.4 30.0 23.6 36.1 27.4

48.8 50.1 53.5 43.5 46.8 54.5 40.8

58.0 57.7 57.9 54.9 58.6 58.8 57.0

61.2 61.3 61.4 59.0 62.2 61.3 60.9

66.5 66.7 66.4 66.2 65.2 64.4 63.3

SLORR-Hoyer-D(0.05) SLORR-Hoyer-D(0.1) SLORR-Hoyer-D(0.3) SLORR-Hoyer-D(0.5) SLORR-Hoyer-D(1) SLORR-Hoyer-D(5) SLORR-Hoyer-D(10)

0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.2 0.1 0.5 0.2 0.2 0.6 8.8

0.5 1.0 1.6 3.0 0.3 1.5 3.8 14.6 1.3 4.2 10.6 32.9 1.2 3.3 19.8 46.9 3.9 26.7 45.5 54.9 28.3 53.0 58.9 61.1 38.8 54.9 58.7 59.3

8.3 31.3 49.8 55.6 60.1 61.6 59.4

23.8 49.3 58.2 60.8 62.4 61.9 59.5

44.6 58.2 62.5 63.6 63.9 61.9 59.6

53.9 61.0 63.7 64.3 64.5 61.9 59.6

66.3 66.2 65.7 65.9 65.1 61.9 59.6

29

Table 8: Complete uniform compression results for ViT-B/16 continued training. Method

11% 21% 31% 41% 51% 60% 70% 80% 90% 95% Full

Baseline⋄ Baseline†

0.1 0.1

0.1 0.1

0.1 0.2

0.7 1.1

12.1 46.5 67.0 76.8 80.9 81.9 84.9 17.8 52.4 69.4 77.3 80.7 81.6 84.1

Q3R(1, 0.05)⋄ Q3R(2, 0.05)⋄ Q3R(3, 0.05)⋄ Q3R(5, 0.05)⋄ Q3R(1, 0.1)⋄ Q3R(2, 0.1)⋄ Q3R(3, 0.1)⋄ Q3R(5, 0.1)⋄ Q3R(0.01, 0.15)⋄ Q3R(1, 0.15)⋄ Q3R(3, 0.15)⋄ Q3R(2, 0.2)⋄ Q3R(3, 0.2)⋄ Q3R(3, 0.05)† Q3R(5, 0.05)† Q3R(8, 0.05)† Q3R(10, 0.05)† Q3R(3, 0.1)† Q3R(5, 0.1)† Q3R(3, 0.15)† Q3R(5, 0.15)† Q3R(3, 0.2)† Q3R(5, 0.2)†

0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 2.3 29.6 44.5 0.1 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 70.8 75.1 75.2 75.0 0.3 3.3 0.2 0.3 0.2 0.3

0.1 0.2 0.1 0.1 0.1 0.2 0.2 0.1 0.6 0.1 0.2 0.1 0.2 76.0 76.3 76.0 75.5 65.4 71.9 21.4 34.5 4.9 8.7

0.2 0.1 0.1 0.7 0.9 0.4 0.4 9.3 12.3 3.6 1.7 3.4 3.1 77.4 76.9 76.1 75.7 80.9 81.0 71.9 74.6 57.8 61.7

5.1 0.1 0.1 3.8 25.9 9.2 2.2 53.4 47.0 41.1 41.8 44.8 45.4 77.9 77.2 76.2 75.8 81.3 81.2 80.6 81.1 75.6 76.7

36.0 0.1 0.1 17.0 57.4 34.2 1.5 73.3 67.3 65.9 51.5 69.0 70.7 78.2 77.3 76.3 75.9 81.5 81.3 82.5 82.5 80.7 81.1

65.6 0.1 0.1 45.1 73.4 50.8 10.2 77.3 76.7 76.5 70.6 78.4 78.8 78.4 77.3 76.3 75.9 81.6 81.4 82.7 82.7 82.3 82.6

75.5 0.1 0.1 53.8 79.0 66.1 37.1 78.7 80.8 80.7 77.2 81.7 81.9 78.5 77.3 76.3 75.9 81.7 81.4 82.7 82.7 83.1 83.2

78.0 0.2 0.1 58.9 80.5 70.9 58.2 79.1 81.9 81.8 79.8 82.5 82.6 78.5 77.4 76.3 75.9 81.8 81.4 82.8 82.8 83.2 83.3

84.4 83.4 81.8 76.6 84.5 83.9 83.1 81.1 84.9 84.7 83.8 84.5 84.2 78.6 77.4 76.3 75.9 81.8 81.4 82.8 82.8 83.3 83.3

LoRITa(0.0001)⋄ LoRITa(0.001)⋄ LoRITa(0.01)⋄ LoRITa(0.05)⋄ LoRITa(0.3)⋄ LoRITa(0.5)⋄ LoRITa(1)⋄ LoRITa(0.0001)† LoRITa(0.001)† LoRITa(0.01)† LoRITa(0.05)† LoRITa(0.3)† LoRITa(0.5)† LoRITa(1)†

0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.2 0.1 0.1 0.1 0.1 0.1 0.1

0.2 0.2 0.1 0.1 0.1 0.2 0.1 1.0 0.9 1.1 0.7 0.2 0.2 0.1

1.0 1.0 1.0 1.1 0.6 0.4 0.1 15.4 13.2 14.0 15.4 3.3 0.6 0.1

17.8 17.8 17.2 17.7 12.3 5.9 0.2 47.1 47.2 47.4 48.3 35.8 10.7 0.2

52.2 52.2 51.9 52.6 48.4 40.6 8.3 67.7 67.6 68.0 68.5 64.9 47.7 0.2

70.0 70.0 70.1 70.3 68.3 65.1 42.7 76.4 76.2 76.5 76.6 75.0 70.0 1.4

77.7 77.7 77.9 77.8 77.0 75.9 68.1 80.2 80.1 80.0 80.2 79.5 77.2 4.5

81.2 81.2 81.3 81.2 80.8 80.2 76.2 81.9 81.9 81.8 82.0 81.5 80.2 11.6

82.2 82.2 82.1 82.2 81.9 81.4 78.7 82.6 82.6 82.5 82.6 82.2 81.4 15.4

84.7 84.7 84.8 84.8 84.7 84.6 84.4 84.0 84.0 84.1 84.0 83.7 83.5 81.8

SLORR-Nuc(0.001)⋄ SLORR-Nuc(0.002)⋄ SLORR-Nuc(0.003)⋄ SLORR-Nuc(0.001)† SLORR-Nuc(0.0002)† SLORR-Nuc(0.0001)† SLORR-Nuc(0.0005)†

0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.5 0.2 0.1 0.2

0.2 1.2 30.6 64.9 75.5 0.2 1.1 36.9 68.8 75.9 0.1 6.7 47.8 69.2 73.4 66.4 73.7 74.9 75.3 75.3 0.3 16.7 55.4 74.8 79.4 0.2 7.6 52.1 72.1 78.5 4.0 63.7 77.2 79.7 80.4

79.9 78.0 74.6 75.3 81.8 81.3 80.6

81.7 78.9 75.0 75.4 82.5 82.6 80.7

82.1 79.1 75.1 75.3 82.8 83.0 80.7

82.8 79.2 75.1 75.3 83.2 83.7 80.7

SLORR-Hoyer(0.0002)⋄ SLORR-Hoyer(0.0005)⋄ SLORR-Hoyer(0.001)⋄ SLORR-Hoyer(0.002)⋄ SLORR-Hoyer(0.003)⋄ SLORR-Hoyer(0.005)⋄ SLORR-Hoyer(0.0002)† SLORR-Hoyer(0.0005)† SLORR-Hoyer(0.0007)† SLORR-Hoyer(0.001)† SLORR-Hoyer(0.00125)†

0.1 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.1 0.1 0.2 0.1 0.1 0.1 0.1 0.1

0.1 0.1 0.1 0.1 0.1 1.8 0.1 0.1 0.2 0.1 0.3

80.4 82.0 82.8 81.5 79.6 75.9 82.8 82.7 82.3 81.7 81.3

82.4 83.4 83.4 81.7 79.6 76.0 83.3 82.8 82.6 81.9 81.3

83.2 83.8 83.5 81.6 79.6 76.0 83.5 82.9 82.6 81.9 81.4

84.8 84.4 83.5 81.7 79.8 76.1 83.8 83.0 82.6 82.0 81.4

1.1 1.4 1.3 6.6 34.7 61.5 0.7 6.7 15.9 62.3 65.8

30

21.3 24.9 32.3 66.2 74.7 73.0 10.1 70.6 75.7 78.7 78.8

58.2 67.1 75.3 79.5 78.3 75.4 67.9 80.6 81.0 80.8 80.4

74.1 78.9 81.3 81.1 79.2 75.9 80.8 82.1 82.0 81.5 81.1

Table 9: Complete uniform compression results for ResNet-50 continued training. Method

17% 26% 35% 45% 54% 63% 72% 81% 90% 95% Full

Baseline

0.1

0.2

0.4

9.4

40.3 62.5 67.9 71.9 74.0 74.5 76.1

Q3R(8, 0.0025) Q3R(20, 0.0025) Q3R(8, 0.01) Q3R(20, 0.01) Q3R(5, 0.025) Q3R(8, 0.025) Q3R(1, 0.05) Q3R(2, 0.05) Q3R(3, 0.05) Q3R(5, 0.05) Q3R(8, 0.05) Q3R(12, 0.05) Q3R(20, 0.05) Q3R(1, 0.1) Q3R(2, 0.1) Q3R(0.01, 0.15) Q3R(1, 0.15) Q3R(2, 0.2)

0.1 0.2 0.1 0.1 0.1 0.2 0.1 0.2 0.2 0.1 0.2 0.1 0.1 0.1 0.1 0.2 0.1 0.1

0.3 0.6 1.0 5.3 0.4 0.8 0.3 0.3 0.4 0.3 0.5 0.4 0.4 0.2 0.2 0.2 0.2 0.1

1.4 11.3 3.2 41.9 7.5 29.0 0.7 4.4 8.1 11.7 8.5 11.3 9.1 1.0 3.1 0.3 1.2 0.5

14.3 46.7 36.3 59.1 49.7 63.1 25.8 38.4 49.9 60.8 63.7 65.6 63.7 32.5 40.5 10.4 18.6 19.2

53.8 60.2 60.7 63.2 65.6 67.4 58.0 67.2 69.9 69.5 72.5 72.7 72.7 61.9 65.1 43.2 54.2 50.6

64.6 63.1 68.0 64.6 70.2 70.0 69.4 71.8 73.0 73.0 73.2 72.9 72.8 70.8 72.4 62.2 68.5 68.0

68.6 64.0 69.4 65.1 71.3 70.9 72.5 73.3 73.5 73.4 73.4 73.1 72.8 73.4 74.0 67.7 72.4 71.8

69.5 64.6 70.1 65.4 71.9 71.2 73.7 73.5 73.7 73.6 73.5 73.2 72.9 74.6 74.9 71.8 74.5 74.1

70.2 64.8 70.4 65.6 72.2 71.4 74.4 74.1 74.2 73.6 73.5 73.3 72.9 75.0 75.2 73.9 75.0 75.1

70.5 65.0 70.5 65.6 72.5 71.5 74.6 74.3 74.2 73.7 73.6 73.3 72.9 75.3 75.4 74.5 75.2 75.3

71.4 65.6 71.1 65.9 73.1 71.9 75.3 74.6 74.5 74.1 73.7 73.4 73.0 75.5 75.4 76.0 75.6 75.6

LoRITa(0.00001) LoRITa(0.0001) LoRITa(0.001) LoRITa(0.01) LoRITa(0.05) LoRITa(0.3) LoRITa(0.5)

0.2 0.2 0.1 0.2 0.2 0.1 0.2

0.2 0.2 0.2 0.2 0.2 0.2 0.1

0.4 0.4 0.4 0.4 0.3 0.4 0.4

7.0 7.0 6.6 5.9 5.9 6.0 6.6

37.6 37.6 39.8 39.5 40.3 38.8 38.5

61.9 61.9 61.5 61.8 61.7 61.5 61.2

67.5 67.5 67.5 67.3 67.5 67.5 67.8

71.7 71.7 71.8 71.5 71.7 71.5 71.7

73.8 73.8 73.9 73.8 74.0 73.8 73.9

74.3 74.3 74.5 74.5 74.7 74.4 74.3

76.0 76.0 76.1 76.0 76.2 76.1 76.1

SLORR-Nuc(0.0002) SLORR-Nuc(0.0005) SLORR-Nuc(0.001) SLORR-Nuc(0.002) SLORR-Nuc(0.003) SLORR-Nuc(0.005)

0.1 0.2 0.1 0.1 0.1 0.1

0.2 0.1 0.1 0.1 0.2 0.2

0.5 0.2 0.2 0.1 1.0 1.9

13.0 20.5 37.1 58.1 62.0 66.2

52.8 64.0 71.6 72.9 71.8 69.4

67.0 72.1 74.0 73.2 72.0 69.6

71.5 74.3 74.5 73.3 72.0 69.6

74.0 74.9 74.6 73.3 72.0 69.6

75.0 75.1 74.6 73.3 72.0 69.6

75.2 75.2 74.6 73.3 72.0 69.6

75.7 75.3 74.7 73.3 72.0 69.6

SLORR-Hoyer(0.0002) SLORR-Hoyer(0.0005) SLORR-Hoyer(0.001) SLORR-Hoyer(0.002) SLORR-Hoyer(0.003) SLORR-Hoyer(0.005)

0.1 0.1 0.1 0.1 0.1 0.1

0.2 0.2 0.2 0.1 0.1 0.1

0.4 0.6 1.6 0.9 1.5 8.2

23.4 44.1 46.8 55.5 61.1 68.1

56.8 68.2 69.9 70.7 70.6 69.7

69.2 72.5 72.8 72.0 71.2 69.8

73.4 74.1 73.5 72.2 71.2 69.9

74.7 74.5 73.6 72.4 71.3 69.9

75.2 74.6 73.7 72.3 71.3 69.9

75.4 74.6 73.7 72.3 71.3 69.9

75.5 74.6 73.6 72.4 71.3 69.9

SLORR-Hoyer-D(0.1) SLORR-Hoyer-D(0.25) SLORR-Hoyer-D(0.5) SLORR-Hoyer-D(1) SLORR-Hoyer-D(1.5) SLORR-Hoyer-D(2)

0.2 0.1 0.1 0.2 0.1 0.1

0.2 0.3 0.4 0.6 0.7 1.1

0.6 0.6 0.7 16.6 32.4 46.5

17.5 29.0 45.0 68.1 69.2 68.2

49.8 59.3 69.5 72.4 70.9 68.5

67.7 71.7 73.7 73.0 70.9 68.5

71.9 74.1 74.3 73.0 70.9 68.5

74.1 74.9 74.4 73.0 70.9 68.6

75.1 75.2 74.5 73.0 70.9 68.6

75.4 75.4 74.4 73.0 70.9 68.5

75.9 75.5 74.5 73.0 70.9 68.6

31

H

Overhead on Vision Models

For our main experiments on ResNet-50 and ViT-B/16, we used NVIDIA A100-80GB GPUs. For the ViT-L/16 experiments, we used B200 GPUs. For the ResNet-18 experiments, we used NVIDIA A100-40GB GPUs. To obtain controlled overhead measurements, we do not rely on end-to-end times from full training runs. Instead, our procedure is as follows. For each model, we conduct 10 short benchmark runs on the same hardware as in the main experiments. For each run, we first perform 10 warmup training steps and then measure time and peak memory over 300 training steps under the same setup as in our training runs. The measurement takes every part of training into account: data loading and augmentation, transfer to the device, training steps, regularization, etc. In the case of Q3R, this includes periodic SVDs. This does not include, for example, evaluation or checkpointing. Each benchmark run repeats this for all methods, including the unregularized baseline. We also tried using autotuning during the Polar Express compilation step, but this always yielded worse performance, so we do not use it. For SLORR, we also vary the number of Polar Express steps; for Q3R, we vary the refresh interval and the target rank. For each run, we compute the normalized time as the time taken with a given method or regularizer divided by the time taken with no regularizer in the same run. More precisely, if Tm,i is the time taken by method m in run i, measured over the 300 timed training steps, and Tbase,i is the corresponding time for the unregularized baseline, then NormTimem,i =

Tm,i . Tbase,i

Thus, a normalized time of ×1.037 means that the method takes 3.7% more time than unregularized training in that setting. Normalized memory is measured similarly, using peak memory instead of time. We show the means and standard deviations of both the normalized and raw measurements in Table 10. H.1

Scaling Up with Vision Transformers

To support our scalability claims, we also measure overhead at different ViT scales. In addition to standard configurations (ViT-T/B/L/H), we include interpolated variants to obtain a smooth overhead curve across model sizes. We use the same settings as in our ViT overhead measurements, but vary the batch size so that all regularizers fit into a B200 GPU, with a maximum of 1024. These models span from the ViT-T scale (on the order of 6M parameters) to ViT-H (over 0.6B parameters). We detail the configurations used in Table 11. We then measure overhead following the protocol described in Appendix H: normalized time is computed by dividing the measured time of each method by the measured time of the corresponding unregularized run under the same model and benchmark setup, and normalized memory is computed analogously using peak memory. Results are shown in Figure 2. For Q3R, the target rank is fixed to 0.1, and different refresh intervals are shown for completeness. Table 11: Vision Transformer (ViT) model configurations and parameter counts used in our overhead measurements. Model Patch Size Embed Dim Depth # Heads MLP Ratio Batch Size Params (M) vit_ti vit_256 vit_s vit_512 vit_b vit_b24 vit_960_24 vit_l vit_1152_28 vit_h

16 16 16 16 16 16 16 16 16 16

192 256 384 512 768 768 960 1024 1152 1280

12 12 12 12 12 24 24 24 28 32

3 4 6 8 12 12 12 16 16 16

32

4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0

1024 1024 1024 1024 1024 1024 1024 896 720 512

5.72 9.98 22.05 38.84 86.57 171.62 267.61 304.33 448.60 632.20

Table 10: Training overhead. Average wall-clock time and peak memory per 300 optimization steps. Means and standard deviations over 10 jobs for each method are reported. Model

Method

Time/300 steps (s)

Peak Mem (GB)

Norm. Time

Norm. Mem

ViT-B/16

Baseline SLORR-Hoyer (steps=6) SLORR-Hoyer (steps=8) SLORR-Hoyer (steps=10) SLORR-Hoyer-D (steps=6) SLORR-Hoyer-D (steps=8) SLORR-Hoyer-D (steps=10) LoRITa Q3R (interval=5, rank=0.01) Q3R (interval=5, rank=0.05) Q3R (interval=5, rank=0.1) Q3R (interval=5, rank=0.15) Q3R (interval=20, rank=0.1) Q3R (interval=50, rank=0.1) Q3R (interval=100, rank=0.1) Q3R (interval=300, rank=0.1)

258.975 ± 6.500 268.447 ± 6.470 271.520 ± 6.112 274.345 ± 6.506 274.687 ± 6.255 277.429 ± 6.678 280.014 ± 6.140 260.670 ± 6.372 426.005 ± 6.311 426.984 ± 6.009 428.857 ± 6.109 431.551 ± 6.355 310.177 ± 6.220 286.478 ± 6.248 278.859 ± 6.291 273.238 ± 6.271

71.584 ± 0.001 71.923 ± 0.000 71.926 ± 0.000 71.924 ± 0.001 71.582 ± 0.002 71.582 ± 0.002 71.582 ± 0.002 72.153 ± 0.001 72.729 ± 0.001 72.728 ± 0.002 72.729 ± 0.001 72.727 ± 0.002 72.728 ± 0.002 72.726 ± 0.001 72.725 ± 0.001 72.724 ± 0.001

×1.000 ± 0.000 ×1.037 ± 0.003 ×1.049 ± 0.004 ×1.059 ± 0.003 ×1.061 ± 0.003 ×1.071 ± 0.003 ×1.081 ± 0.005 ×1.007 ± 0.002 ×1.645 ± 0.016 ×1.649 ± 0.019 ×1.656 ± 0.018 ×1.667 ± 0.018 ×1.198 ± 0.006 ×1.106 ± 0.004 ×1.077 ± 0.004 ×1.055 ± 0.003

×1.000 ± 0.000 ×1.005 ± 0.000 ×1.005 ± 0.000 ×1.005 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.008 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000 ×1.016 ± 0.000

ResNet-50

Baseline SLORR-Hoyer (steps=6) SLORR-Hoyer (steps=8) SLORR-Hoyer (steps=10) SLORR-Hoyer-D (steps=6) SLORR-Hoyer-D (steps=8) SLORR-Hoyer-D (steps=10) LoRITa Q3R (interval=5, rank=0.01) Q3R (interval=5, rank=0.05) Q3R (interval=5, rank=0.1) Q3R (interval=5, rank=0.15) Q3R (interval=20, rank=0.1) Q3R (interval=50, rank=0.1) Q3R (interval=100, rank=0.1) Q3R (interval=300, rank=0.1)

238.173 ± 5.457 249.511 ± 5.716 252.558 ± 5.616 256.021 ± 5.647 256.658 ± 5.717 260.724 ± 5.618 264.580 ± 6.477 239.302 ± 5.506 291.274 ± 5.665 291.162 ± 5.735 290.995 ± 5.594 290.084 ± 5.973 258.970 ± 5.592 253.010 ± 5.494 251.038 ± 5.510 249.735 ± 5.463

46.221 ± 0.001 46.316 ± 0.002 46.315 ± 0.001 46.315 ± 0.001 46.221 ± 0.001 46.221 ± 0.001 46.222 ± 0.001 46.350 ± 0.001 46.532 ± 0.003 46.533 ± 0.002 46.531 ± 0.002 46.531 ± 0.001 46.530 ± 0.002 46.530 ± 0.002 46.529 ± 0.001 46.527 ± 0.001

×1.000 ± 0.000 ×1.048 ± 0.003 ×1.060 ± 0.002 ×1.075 ± 0.004 ×1.078 ± 0.004 ×1.095 ± 0.006 ×1.111 ± 0.016 ×1.005 ± 0.006 ×1.223 ± 0.007 ×1.223 ± 0.008 ×1.222 ± 0.006 ×1.218 ± 0.012 ×1.087 ± 0.002 ×1.062 ± 0.002 ×1.054 ± 0.006 ×1.049 ± 0.005

×1.000 ± 0.000 ×1.002 ± 0.000 ×1.002 ± 0.000 ×1.002 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.003 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000

ViT-L/16

Baseline SLORR-Hoyer (steps=6) SLORR-Hoyer (steps=8) SLORR-Hoyer (steps=10) SLORR-Hoyer-D (steps=6) SLORR-Hoyer-D (steps=8) SLORR-Hoyer-D (steps=10) LoRITa Q3R (interval=5, rank=0.01) Q3R (interval=5, rank=0.05) Q3R (interval=5, rank=0.1) Q3R (interval=5, rank=0.15) Q3R (interval=20, rank=0.1) Q3R (interval=50, rank=0.1) Q3R (interval=100, rank=0.1) Q3R (interval=300, rank=0.1)

192.797 ± 1.644 203.345 ± 1.227 205.978 ± 1.196 208.406 ± 1.064 207.288 ± 1.125 210.050 ± 1.210 212.514 ± 1.081 194.632 ± 1.502 580.708 ± 1.323 585.406 ± 1.353 589.815 ± 1.282 594.638 ± 1.470 307.126 ± 1.312 250.827 ± 1.454 232.288 ± 1.213 219.497 ± 1.362

164.376 ± 0.000 165.582 ± 0.001 165.583 ± 0.001 165.582 ± 0.001 164.375 ± 0.001 164.375 ± 0.001 164.375 ± 0.001 166.391 ± 0.000 168.420 ± 0.001 168.422 ± 0.001 168.421 ± 0.002 168.421 ± 0.001 168.419 ± 0.001 168.418 ± 0.002 168.416 ± 0.000 168.415 ± 0.002

×1.000 ± 0.000 ×1.055 ± 0.003 ×1.068 ± 0.004 ×1.081 ± 0.005 ×1.075 ± 0.004 ×1.090 ± 0.004 ×1.102 ± 0.004 ×1.010 ± 0.001 ×3.012 ± 0.023 ×3.037 ± 0.022 ×3.059 ± 0.023 ×3.084 ± 0.022 ×1.593 ± 0.007 ×1.301 ± 0.004 ×1.205 ± 0.006 ×1.139 ± 0.003

×1.000 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.007 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.012 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000 ×1.025 ± 0.000

ResNet-18

Baseline SLORR-Hoyer (steps=6) SLORR-Hoyer (steps=8) SLORR-Hoyer (steps=10) SLORR-Hoyer-D (steps=6) SLORR-Hoyer-D (steps=8) SLORR-Hoyer-D (steps=10) LoRITa Q3R (interval=5, rank=0.01) Q3R (interval=5, rank=0.05) Q3R (interval=5, rank=0.1) Q3R (interval=5, rank=0.15) Q3R (interval=20, rank=0.1) Q3R (interval=50, rank=0.1) Q3R (interval=100, rank=0.1) Q3R (interval=300, rank=0.1)

169.288 ± 7.488 169.822 ± 6.864 171.127 ± 8.464 169.576 ± 6.405 168.660 ± 5.460 170.404 ± 9.644 170.196 ± 8.941 167.907 ± 4.230 174.902 ± 16.476 170.511 ± 9.189 169.531 ± 5.638 170.471 ± 6.208 171.572 ± 9.018 170.862 ± 7.709 171.271 ± 8.197 170.911 ± 7.846

13.146 ± 0.000 13.192 ± 0.000 13.191 ± 0.000 13.191 ± 0.000 13.144 ± 0.000 13.144 ± 0.000 13.144 ± 0.000 13.191 ± 0.000 13.288 ± 0.001 13.289 ± 0.001 13.289 ± 0.000 13.288 ± 0.000 13.289 ± 0.000 13.288 ± 0.001 13.288 ± 0.000 13.288 ± 0.000

×1.000 ± 0.000 ×1.003 ± 0.022 ×1.011 ± 0.025 ×1.002 ± 0.027 ×0.997 ± 0.028 ×1.006 ± 0.022 ×1.005 ± 0.031 ×0.993 ± 0.032 ×1.031 ± 0.051 ×1.008 ± 0.047 ×1.002 ± 0.022 ×1.008 ± 0.032 ×1.014 ± 0.038 ×1.010 ± 0.026 ×1.012 ± 0.034 ×1.010 ± 0.038

×1.000 ± 0.000 ×1.003 ± 0.000 ×1.003 ± 0.000 ×1.003 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.000 ± 0.000 ×1.003 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000 ×1.011 ± 0.000

33

I

LLM Pretraining Experimental and Implementation Details

Implementation. For our LLM pretraining experiments, we use the OLMo [59] codebase14 . For all training experiments, which use multiple GPUs, we use distributed data parallelism (DDP). Aside from adding the SLORR regularizer and benchmarking utilities, the main codebase remains mostly unchanged. SLORR is implemented as an additional loss term, similar to Listing 1. To distribute its computational cost, we partition the regularized layers across DDP ranks, so that each rank computes SLORR only for its assigned subset. This does not introduce any extra communication. Instead, SLORR reuses the gradient synchronization already performed by DDP. Since DDP averages gradients across ranks during its standard all-reduce, each rank scales its locally computed SLORR gradients by the world size before synchronization. After the DDP all-reduce, every regularized parameter therefore receives exactly the gradient it would have obtained if the full SLORR regularizer had been computed redundantly on every rank. In this way, the computational overhead of SLORR is divided across GPUs without any additional communication cost. Experimental details. We use a Llama-like architecture [40] and the T5 tokenizer [60], available at https://huggingface.co/google-t5/t5-base. The results reported use the original OLMoLLamaBlock, which computes attention manually. We note, however, that for the overhead measurements in Appendix L, we also include results for fused attention implementations to provide a practical overhead estimate. We use the AdamW optimizer with learning rate 10−3 , β1 = 0.9, β2 = 0.95, and weight decay set to 0.1. Gradients are clipped using a maximum global norm of 1.0. We use a cosine scheduler with 10% of the training steps dedicated to a linear warmup from 0 to the base learning rate, which is then decayed to 10% of the base learning rate. We chose these settings based on common practice for standard training and did not try other settings. Training is performed on a subsample of the FineWeb-Edu [41] 100BT sample15 ; evaluation and calibration sets (for SVD-LLM) are also selected from this sample but are not included in the training set. We summarize the size configurations used in Table 12. All models use a global batch size of 512 and a sequence length of 1024. The 135M and 560M rows correspond to compute-optimal training budgets. The multipliers indicate the amount of data on which the model was trained relative to compute optimality [42]. The 135M models and their overtrained versions were trained on 4 H100 GPUs, while the 560M models were trained on 8 H100 GPUs. We regularize (and factorize after training) every linear layer inside transformer blocks (we exclude the embedding matrix and the final projection layer). Table 12: LLM model configurations and training budgets. All models use a global batch size of 512 and sequence length 1024. Model Hidden Intermediate Heads Layers Training steps 135M 135M×4 135M×8 560M

768 768 768 1280

2048 2048 2048 3456

12 12 12 20

12 12 12 24

5,200 20,600 41,200 21,400

As in our vision experiments, all runs use a default seed. For transparency, we note that the 135M×8 Llama run with regularization strength 10−5 encountered a numerical explosion and terminated before completion. We monitored different metrics and did not observe preceding indicators of instability. We believe the crash may have been due to the particular trajectory taken by the model under this seed and hyperparameter setting, rather than due to the regularizer itself. We therefore reran this configuration with another seed and report the completed rerun. 14 https://github.com/allenai/olmo 15 https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/viewer/sample-100BT

34

Method hyperparameters. The range of hyperparameters was not exhaustively explored and was mainly based on early experimentation. With the chosen hyperparameters, training was mostly stable. Among the completed runs, the ×8 run with the highest regularization strength reported (see Appendix J) had a loss spike early in training. This can also destabilize compression, as can be seen in Table 17. I.1

Compression

For compression, we adapt the SVD-LLM [18] codebase16 to support OLMo checkpoints and the FineWeb dataset. We substantially simplified the codebase to specialize it for our setting, and also added support for plain SVD compression. For SVD-LLM, we use 256 samples of sequence length 1024 (same as in the training procedure) of held-out FineWeb-Edu data. For activation-aware (whitened) truncation as used in SVD-LLM, we observe that regularized models often yield rank-deficient activations, meaning that their second-moment matrices are singular. SVDLLM uses Cholesky-based whitening, which does not support rank-deficient matrices; in practice, this is addressed by adding diagonal shift. We instead modify the codebase to use BALF-like whitening [57], which admits rank-deficient activations in closed form. We did not observe notable end-to-end gains from this modification, though. I.2

Evaluation

After training and compressing each model, we evaluate them according to standard practice. For perplexity, we report results on a held-out set of approximately 2M tokens. For downstream evaluations, we use the lm-evaluation-harness library [43] throughout and report zero-shot metrics. When available, we use acc_norm; otherwise, we use the acc field. Tasks. We evaluate on ARC-Easy and ARC-Challenge [44], HellaSwag [45], LAMBADA [46], OpenBookQA [47], and PIQA [48]. We also initially explored WinoGrande [61], but accuracy appeared to be mostly random in all cases, so we did not end up using it.

J

Beyond Compute-Optimal LLM Pretraining

Here, we report results for 135M models trained for 4× and 8× the compute-optimal token budget [42]. As noted in the main text, we find that the compressibility of all models, including regularized and unregularized ones, appears to be reduced as they ingest more data. However, in all cases, regularized models are still more compressible.

Perplexity

150 100 50

Llama 135M x4 Plain SVD

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001 =0.00005

0 0.2

0.4

0.6

Parameters ratio

0.8

Llama 135M x4 SVD-LLM (whitened) SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001 =0.00005

1.0 0.2

0.4

0.6

Parameters ratio

0.8

Llama 135M x8 Plain SVD

SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001 =0.00005

1.0 0.2

0.4

0.6

Parameters ratio

0.8

Llama 135M x8 SVD-LLM (whitened) SLORR-Hoyer Unreg. ( =0) =0.000005 =0.00001 =0.00005

1.0 0.2

0.4

0.6

Parameters ratio

0.8

1.0

Figure 4: Llama models trained for 4× and 8× the compute-optimal token budget. Perplexity values are clipped to 150, as in our main text figures. For complete tabular results, see Appendix K.

K

Complete LLM Results

In this section, we include tabular results for the LLM figures in the main text. We include one table for perplexity and, for the 560M variants, one table for each downstream task. Each row corresponds to a regularization parameter λ, including the unregularized baseline λ = 0. The columns denote the retained parameter ratio in the transformer blocks. 16 https://github.com/AIoT-MLSys-Lab/SVD-LLM

35

K.1

Llama 135M Results

Table 13: Llama 135M – Plain SVD – FineWeb-Edu perplexity λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700 0.800 0.900 1.000

Unreg. (λ = 0) 148924.22 96542.02 94756.84 48167.34 6204.80 745.65 120.20 43.56 30.31 20.03 λ = 0.000005 82613.87 89934.46 121806.95 10169.25 620.10 62.31 30.94 24.71 22.56 20.29 λ = 0.00001 244959.88 126294.20 17418.12 2114.17 101.18 31.82 24.55 22.72 21.73 20.79 λ = 0.00005 19597.55 4195.52 144.97 30.38 25.98 24.73 24.17 23.85 23.68 23.51

Table 14: Llama 135M – SVD-LLM (whitened) – FineWeb-Edu perplexity λ Unreg. (λ = 0) λ = 0.000005 λ = 0.00001 λ = 0.00005

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

1860.53 4931.33 4820.77 1855.93

508.60 659.68 586.07 123.38

185.68 158.93 125.66 38.59

90.66 66.69 51.09 27.87

52.75 38.35 31.66 25.40

36.50 28.79 25.63 24.45

29.00 24.71 23.13 24.00

25.01 22.65 21.93 23.77

22.86 21.52 21.33 23.64

20.03 20.29 20.79 23.51

Table 15: Llama 135M×4 – Plain SVD – FineWeb-Edu perplexity λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800 0.900 1.000

Unreg. (λ = 0) 256446.09 163660.19 38166.82 15480.69 9899.26 2142.90 410.64 102.25 45.71 16.69 λ = 0.000005 120145.84 70002.43 173048.14 26394.15 664.70 86.99 38.58 25.66 20.70 16.98 λ = 0.00001 481274.56 260275.45 547987.81 12598.17 181.30 43.16 25.81 20.62 18.68 17.29 λ = 0.00005 31923.13 9960.79 1046.29 379.80 131.44 50.42 30.91 24.03 21.29 17.63

Table 16: Llama 135M×4 – SVD-LLM (whitened) – FineWeb-Edu perplexity λ Unreg. (λ = 0) λ = 0.000005 λ = 0.00001 λ = 0.00005

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

33472.52 10243.44 6497.80 5202.02

4275.64 1826.38 1064.54 1210.30

704.20 385.01 181.37 241.95

183.61 109.28 68.22 83.69

87.98 47.59 35.97 44.83

51.24 31.06 25.45 33.21

33.59 24.29 21.04 28.31

25.79 20.58 19.05 25.47

21.79 18.64 18.13 23.85

16.69 16.98 17.29 17.63

Table 17: Llama 135M×8 – Plain SVD – FineWeb-Edu perplexity λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800 0.900 1.000

Unreg. (λ = 0) 458317.91 145537.27 46363.39 91373.03 175450.86 96216.00 15849.05 1432.41 176.94 15.85 λ = 0.000005 61717.04 24633.72 83167.53 59771.51 21030.25 391.34 64.09 32.02 22.94 16.06 λ = 0.00001 146102.70 133060.38 37594.02 2437.90 573.78 110.50 45.29 27.67 21.48 16.17 λ = 0.00005 30970.36 35753.34 21145.99 10263.97 6197.59 1838.47 590.69 169.88 51.43 18.67

Table 18: Llama 135M×8 – SVD-LLM (whitened) – FineWeb-Edu perplexity λ Unreg. (λ = 0) λ = 0.000005 λ = 0.00001 λ = 0.00005

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

9130.10 11983.68 5442.37 3653.86

5885.35 2819.26 1502.64 457.59

781.80 587.96 328.19 151.60

223.97 178.03 109.19 79.93

106.42 69.14 48.44 50.61

60.41 38.42 30.83 36.68

39.07 27.14 23.87 29.26

28.43 21.92 20.34 24.77

22.96 19.04 18.45 22.12

15.85 16.06 16.17 18.67

36

K.2

Llama 560M Results Table 19: Llama 560M – Plain SVD – ARC-Challenge

λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

27.39 26.28 26.37

25.68 26.19 24.23

23.72 23.55 26.02

25.34 27.90 27.30

24.49 28.33 27.39

28.58 29.44 28.24

29.52 29.27 28.50

31.14 29.35 28.07

30.63 30.12 28.67

30.55 29.69 28.33

λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

26.09 26.43 24.92

25.72 26.22 27.61

26.14 39.02 45.50

28.79 50.08 49.45

34.64 52.40 52.02

42.21 53.41 52.78

47.52 54.04 52.95

52.78 54.12 53.32

54.08 54.38 53.49

56.82 54.55 53.54

λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

26.16 25.70 26.21

26.26 26.44 26.50

26.39 31.60 34.22

26.80 37.18 37.00

30.80 39.74 38.74

35.87 40.63 39.69

40.84 41.31 40.15

42.88 41.85 40.32

43.93 42.02 40.47

44.32 42.45 40.71

λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

0.00 0.00 0.00

0.00 0.00 0.02

0.00 9.37 22.69

0.37 23.91 29.40

3.28 30.39 31.96

16.51 34.14 33.13

26.14 35.09 33.48

31.19 35.07 32.58

33.67 34.80 32.78

36.43 35.94 33.40

Table 20: Llama 560M – Plain SVD – ARC-Easy

Table 21: Llama 560M – Plain SVD – HellaSwag

Table 22: Llama 560M – Plain SVD – LAMBADA

Table 23: Llama 560M – Plain SVD – OpenBookQA λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

28.20 29.00 26.20

25.40 26.60 24.60

23.00 27.80 31.20

25.60 31.00 31.20

27.60 33.80 32.20

33.60 34.00 33.20

35.80 35.00 34.20

37.20 34.40 34.80

35.20 34.60 34.60

34.80 34.40 35.20

Table 24: Llama 560M – Plain SVD – PIQA λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

50.54 50.05 49.40

50.16 48.20 51.74

50.54 58.16 61.32

53.54 62.79 64.09

56.64 65.18 65.72

60.07 66.49 65.89

64.15 67.52 66.54

66.43 67.85 67.03

68.12 67.79 66.97

69.04 68.23 66.92

Table 25: Llama 560M – Plain SVD – FineWeb-Edu perplexity λ Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

21391.07 42353.88 72488.98

40587.82 14071.56 2113.69

21837.16 93.47 31.16

30551.34 24.59 19.55

581.20 17.41 16.39

55.90 15.06 15.05

28.62 14.11 14.43

20.61 13.71 14.15

17.10 13.54 13.97

12.61 13.38 13.76

37

Table 26: Llama 560M – SVD-LLM (whitened) – ARC-Challenge λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

25.17 25.68 23.46

23.38 23.55 22.27

23.21 24.74 27.39

23.72 27.47 27.39

24.91 27.82 27.22

25.34 28.33 28.07

28.07 29.01 28.33

27.56 29.61 28.50

28.33 29.78 28.50

30.55 29.69 28.33

Table 27: Llama 560M – SVD-LLM (whitened) – ARC-Easy λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

25.25 25.88 27.69

27.36 31.99 37.67

31.65 41.25 46.42

35.52 50.08 49.96

42.34 50.97 52.10

47.39 53.16 52.82

51.35 53.83 53.62

54.08 54.17 53.24

55.60 54.59 53.28

56.82 54.55 53.54

λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

25.65 25.35 25.85

26.16 27.64 28.91

27.06 32.36 33.84

29.72 36.63 36.73

33.40 38.54 38.17

36.82 40.49 39.11

39.28 41.30 39.75

41.12 41.59 40.19

42.32 41.92 40.43

44.32 42.45 40.71

Table 28: Llama 560M – SVD-LLM (whitened) – HellaSwag

Table 29: Llama 560M – SVD-LLM (whitened) – LAMBADA λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

0.00 0.06 0.00

0.00 3.36 4.99

0.62 21.42 22.34

4.95 29.05 28.76

13.86 32.84 29.58

20.98 35.16 31.79

25.67 35.44 32.56

28.59 35.30 32.85

31.85 35.30 33.28

36.43 35.94 33.40

Table 30: Llama 560M – SVD-LLM (whitened) – OpenBookQA λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

29.60 25.80 27.20

28.60 26.80 24.40

25.60 28.20 29.60

27.20 30.60 31.80

27.40 34.40 31.00

31.60 34.20 33.00

33.00 33.80 33.80

34.60 34.60 34.40

34.60 34.80 34.00

34.80 34.40 35.20

Table 31: Llama 560M – SVD-LLM (whitened) – PIQA λ

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

50.49 50.54 50.38

50.82 54.30 55.77

52.83 60.01 61.64

54.79 63.87 63.93

59.96 65.61 64.64

63.28 67.03 66.27

65.40 66.81 66.59

66.38 67.63 66.59

67.46 68.01 66.92

69.04 68.23 66.92

Table 32: Llama 560M – SVD-LLM (whitened) – FineWeb-Edu perplexity λ Unreg. (λ = 0) λ = 0.000005 λ = 0.00001

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

89505.28 9702.74 1080.51

7759.18 187.39 76.44

475.48 35.58 25.46

82.92 20.72 18.30

36.62 16.35 15.93

23.30 14.67 14.87

18.17 13.97 14.37

15.68 13.65 14.10

14.33 13.52 13.93

12.61 13.38 13.76

38

L

Overhead on LLM training

As noted before, our main results use the original OLMo Llama-like implementation, which uses OLMoLlamaBlock and computes attention manually. To provide a more realistic benchmark, we also measure overhead under several implementation choices. Specifically, we benchmark OLMoLlamaBlock, which uses separate Q, K, and V projections, with both manual attention and fused attention; we add the fused attention variant by modifying the block. We also benchmark OLMoSequentialBlock, which uses a single shared QKV projection and only uses fused attention. Here, fused attention refers to torch.nn.functional.scaled_dot_product_attention. For each model/configuration, we run four matched pairs of jobs. Each pair consists of two independent training runs with identical model size, attention implementation, block type, microbatch size, data, and benchmark window. One baseline is run without regularization and one run with regularization. Within a pair, the two runs are performed on the exact same node, as we observed slight overall performance variations per node. For these, we use autotuning in the Polar Express compilation step, which we observed to yield slight gains in terms of overhead (unlike in the vision benchmarks). Each individual run measures the wall-clock time for steps 101–700. The 135M experiments use 4 H100 GPUs, and the 560M experiments use 8 H100 GPUs, following our main experiments. We aggregate results by first computing the overhead for each matched pair,   Treg,i 100 × −1 , Tbase,i where Treg,i and Tbase,i are the 600-step wall times for pair i. We report the mean and standard deviation of these paired overheads. Seconds and tokens/sec are reported as arithmetic means over the corresponding baseline or regularized runs. Peak memory is computed similarly, and was stable across runs. The reported quantities are per device, and the peak memory is taken as the maximum over all GPUs. Results are shown in Table 33. MBS denotes the microbatch size per GPU. Note that, for the sake of fairness, it is adjusted for fused attention runs because each sequence consumes less memory. On average, overhead remains below 1% across the different settings, although individual runs can slightly exceed 1%. Peak memory overhead is negligible in these benchmarks. Table 33: LLM overhead benchmark summary. Size

Attention

Block

MBS

Reg.

Seconds

Tokens/sec

135M 135M

Manual Manual

Llama Llama

32 32

no yes

735.031 738.614

106,992.99 0.000% 106,474.08 0.487% ± 0.042%

58.890 59.010

135M 135M

SDPA SDPA

Llama Llama

64 64

no yes

355.179 358.113

221,418.77 0.000% 219,606.97 0.826% ± 0.275%

56.810 56.930

135M 135M

SDPA SDPA

Sequential Sequential

64 64

no yes

345.173 347.225

227,837.61 0.000% 226,491.57 0.594% ± 0.122%

56.780 56.930

560M 560M

Manual Manual

Llama Llama

12 12

no yes

1,267.318 1,276.090

31,027.41 30,814.13

0.000% 0.692% ± 0.003%

72.880 72.890

560M 560M

SDPA SDPA

Llama Llama

24 24

no yes

616.515 622.627

63,780.88 63,154.74

0.000% 0.991% ± 0.014%

61.010 61.020

560M 560M

SDPA SDPA

Sequential Sequential

24 24

no yes

596.743 602.191

65,894.00 65,297.77

0.000% 0.913% ± 0.079%

61.030 61.040

39

Overhead

Peak mem. (GB)

All Llama 135M Singular Values

800

0

102

200

400

Index

600

800

blocks.4.k_proj.weight

0

200

400

Index

600

800

0

blocks.5.q_proj.weight

200

400

Index

600

800

Singular value

Singular value

Singular value

10 1

200

400

Index

600

800

blocks.6.q_proj.weight

100

200

400

Index

600

100

200

400

Index

600

800

blocks.7.q_proj.weight Singular value

Singular value

101 10 1

0

200

400

Index

600

100

200

400

Index

600

800

blocks.8.q_proj.weight Singular value

100

10 2

0

200

400

Index

600

100

400

Index

600

800

Singular value

Singular value

10 1

0

200

400

Index

600

100

0

200

400

Index

600

800

400

Index

600

10 5 200

400

Index

600

800

blocks.11.q_proj.weight Singular value

Singular value

10 1

0

200

400

Index

600

100

0

200

400

Index

600

800

0

200

400

Index

600

200

400

Index

600

200

400

Index

600

100

800

0

200

400

Index

600

400

Index

600

800

200

400

Index

600

10 5

400

Singular value Singular value Singular value 0

600

600

200

400

Index

600

0

200

400

Index

600

800

10 2 800 0

200

400

Index

600

800

blocks.5.ff_out.weight

101 100

10 2 800 0

200

400

Index

600

800

blocks.6.ff_out.weight 100

10 2 800 0

200

400

Index

600

800

blocks.7.ff_out.weight

101 100

10 2 800 0

200

400

Index

600

800

blocks.8.ff_out.weight 101

101 100

100

10 1

0

200

400

Index

600

10 2 800 0

blocks.9.ff_proj.weight

200

400

Index

600

800

blocks.9.ff_out.weight 101

101 100

100

10 1

0

200

400

Index

600

10 2 800 0

blocks.10.ff_proj.weight

102

800

600

10 1

200

400

Index

600

800

blocks.10.ff_out.weight 101 100

10 1

0 102

800

10 3

Index

400

Index

100

10 1 800 0

10 1

200

200

101

102

blocks.11.attn_out.weight

400

Index

10 1

100

0

200

100

blocks.8.ff_proj.weight

101

0

600

101

10 5

800

blocks.4.ff_out.weight

101

blocks.7.ff_proj.weight

800

600

101

10 2 800 0

blocks.10.attn_out.weight

101

10 3

200

400

Index

10 1

10 1

100

200

100

800

400

Index

100

blocks.6.ff_proj.weight

800

200

10 1

10 2 800 0

blocks.9.attn_out.weight

600

600

10 1

10 6 800 0

blocks.10.v_proj.weight

0

Singular value

Singular value 600

100

blocks.11.v_proj.weight

800

400

Index

10 4

0

800

ff_out.weight

wte.weight

Singular value

Singular value 400

Index

10 4

10 4

10 5

200

10 2

10 2

10 3

200

10 2

blocks.11.k_proj.weight

102

0

blocks.9.v_proj.weight

101

Singular value

0 101

0

400

Index

10 1

10 4

10 5

10 5

600

10 2

10 3

10 3

800

200

100

blocks.8.attn_out.weight

10 1

10 1

10 3

600

Singular value

Singular value

Singular value

10 1

400

Index

100

800

blocks.10.k_proj.weight

400

Index

10 5

200

10 5

200

200

10 3

10 3

0

0

800

blocks.3.ff_out.weight

10 2 800 0

blocks.5.ff_proj.weight

10 1

10 1

101

101

10 2 800 0

101

0

800

600

600

10 1

blocks.7.attn_out.weight

10 1

blocks.10.q_proj.weight 101

800

600

101

blocks.8.v_proj.weight

blocks.9.k_proj.weight

10 4

10 5

600

10 4

10 2

10 3

400

Index

10 2

102

101

200

100

800

400

Index

10 5

Singular value

200

blocks.9.q_proj.weight

200

10 3

0

400

Index

10 1

0

400

Index

100

101

101

10 1

blocks.8.k_proj.weight

10 4

0

800

200

100

blocks.6.attn_out.weight

10 4

10 2

10 4

600

10 2

102

600

100

blocks.7.v_proj.weight

Singular value

0 102

800

400

Index

200

blocks.2.ff_out.weight

10 2 800 0

blocks.4.ff_proj.weight

101

blocks.7.k_proj.weight

10 4

10 5

400

Index

10 5

10 2

10 3

200

10 3

102

200

blocks.6.v_proj.weight

Singular value

0

0

10 6

0

10 2 800 0

10 1

10 4

10 1

10 4

10 4

800

101

10 2

10 2

800

600

10 2

blocks.6.k_proj.weight

102

Singular value

102

100

10 6

0

400

Index

800

10 1

blocks.5.attn_out.weight

Singular value

0

600

10 4

10 5

10 5

400

Index

10 2

10 3

10 3

200

600

101

blocks.5.v_proj.weight

101

10 1

200

10 5

0

400

Index

100

blocks.4.attn_out.weight

10 3

blocks.5.k_proj.weight

101

0

10 1

10 5

10 5

10 6

800

200

blocks.3.ff_proj.weight

10 1

101

10 3

10 3

10 4

600

10 1

10 1

10 2

400

Index

10 2 800 0 101

10 5

200

blocks.4.v_proj.weight

Singular value

Singular value

100

600

10 3

101

101

400

Index

600

10 1

blocks.3.attn_out.weight

10 1

0

100

Singular value

600

200

400

Index

100

101

Singular value

400

Index

0 101

100

10 4

blocks.4.q_proj.weight

800

200

blocks.1.ff_out.weight

10 2 800 0

Singular value

200

600

600

Singular value

0

400

Index

10 2

10 4

10 5

200

400

Index

10 1

10 4

blocks.3.v_proj.weight

200

blocks.2.ff_proj.weight

Singular value

100

10 2

10 3

10 6 800 0

10 2 800 0

Singular value

600

100

0 101

101

10 2

blocks.3.k_proj.weight

800

Singular value

400

Index

600

Singular value

200

400

Index

Singular value

Singular value

Singular value

101 10 1

0 102

200

Singular value

800

0

100

10 1

blocks.2.attn_out.weight

100

600

100

Singular value

600

800

Singular value

400

Index

600

Singular value

200

blocks.3.q_proj.weight

400

Index

10 4

Singular value

0

200

blocks.2.v_proj.weight 10 2

10 4

10 4

0

400

Index

10 1

10 4

Singular value

100

10 2

10 2

800

100

10 2

blocks.2.k_proj.weight

200

blocks.1.ff_proj.weight 101

Singular value

600

600

Singular value

400

Index

400

Index

Singular value

200

200

Singular value

Singular value

100

0

0

101

10 1

blocks.1.attn_out.weight

10 6

102

800

Singular value

800

600

Singular value

600

400

Index

10 4

Singular value

400

Index

blocks.2.q_proj.weight

200

blocks.1.v_proj.weight 10 2

10 5

200

0 100

blocks.0.ff_out.weight

102

200

400

Index

600

800

blocks.11.ff_proj.weight

101 100

0

200

400

Index

600

800

blocks.11.ff_out.weight

102

Singular value

800

Singular value

600

10 3

0

Singular value

400

Index

10 1

102

Singular value

200

blocks.1.k_proj.weight Singular value

100

10 4

Singular value

0 101

10 2

Singular value

800

Singular value

600

Singular value

400

Index

Singular value

Singular value

200

blocks.1.q_proj.weight

100

10 1

10 5

Singular value

0 102

10 3

10 5

101

Singular value

10 5

blocks.0.ff_proj.weight

102

10 2 800 0

10 1

10 3

10 3

10 5

blocks.0.attn_out.weight

101

Singular value

Singular value

Singular value

10 3

blocks.0.v_proj.weight

101 10 1

Singular value

blocks.0.k_proj.weight 101 10 1

Singular value

blocks.0.q_proj.weight 101 10 1

Singular value

M

101 100

10 1

200

400

Index

600

800

0

200

400

Index

600

800

Singular value

Singular value

102 101

Unreg. ( =0)

101

0

200

400

Index

600

800

0

200

400

Index

600

SLORR-Hoyer

=0.000005

=0.00001

=0.00005

800

Figure 5: Singular value spectra for all layers of Llama 135M. The different curves correspond to different values of λ. As noted in the main text, only layers inside transformer blocks are regularized, but we also include weight embeddings and the final projection layer for completeness.

40

LLM Usage Disclosure We used LLM tools in limited supporting roles during this work. In particular, they helped identify potentially relevant related work, in addition to manual search. All cited works were selected and read by the authors; LLMs primarily assisted with search. We also used LLMs to discuss our research ideas, suggest possible proof steps, check the presentation of existing technical arguments, support code development, and proofread or help polish parts of the writing. All results presented were checked and validated by the authors.

41