SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Mahmoud Safari1
arXiv:2606.23568v1 [cs.LG] 22 Jun 2026
1
Frank Hutter2,3,1
University of Freiburg 2 Prior Labs 3 ELLIS Institute Tübingen {safarim,fh}@cs.uni-freiburg.de
Abstract Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-rank compression via singular value decomposition (SVD) is an effective remedy, but existing methods focus on how to factorize and which components to keep. We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis. Treating each singular value as a parameter, it computes a closed-form update of the retained singular values that compensates, to second order in the model loss, for those removed by truncation. The same analysis yields a saliency for choosing which values to prune. As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors. Applied to SVD-LLM, a leading SVD-based method, it improves the perplexity–compression trade-off on the OPT family and LLaMA 2-7B without any retraining.
1
Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language understanding and generation tasks. However, their deployment remains challenging due to substantial computational and memory requirements, with state-of-the-art models comprising billions of parameters that demand significant GPU resources both at inference and fine-tuning time. Reducing these costs without sacrificing quality has become a central problem for the practical use of LLMs. Model compression has emerged as a principled approach to address these demands, with pruning being among the most widely studied techniques. Structured pruning removes entire neurons, attention heads, or layers, yielding hardware-friendly sparse models but at the cost of coarse-grained approximations that often degrade performance significantly. Unstructured pruning operates at the individual weight level, achieving fine-grained sparsity but producing irregular patterns that are difficult to accelerate on modern hardware without specialized kernels. Semi-structured pruning (e.g. 2:4 sparsity) offers a compromise but remains constrained by fixed patterns that limit flexibility. Compression using low-rank decomposition such as SVD offers a qualitatively different approach: by decomposing each weight matrix as θ = U ΣV ⊤ and truncating to a target rank, one obtains a low-rank approximation that is naturally deployable via two successive matrix multiplications, without requiring specialized hardware support. Naive truncation of the smallest singular values is rarely optimal, and a growing body of work improves on it through importance reweighting and activation whitening that better align the decomposition with the model’s loss landscape (Section 2). We introduce SVD-Surgeon, which applies the Optimal Brain Surgeon (OBS) framework [5] directly in the basis of singular values. While prior SVD methods focus on how the low-rank approximation is formed and which components to discard, SVD-Surgeon also repairs those that remain: given the decomposition produced by any SVD-based compressor, it excises singular values and adjusts
the retained ones to absorb the induced loss. Concretely, treating each retained singular value as a parameter, it builds a second-order (Fisher) model of the loss in singular-value coordinates and derives a closed-form update of the kept singular values that compensates for those removed by truncation, recovering capacity in a single shot, with no gradient-based optimization and no posttraining fine-tuning. The same analysis yields an OBS saliency for each singular value (i.e. the loss incurred by removing it, accounting for the optimal correction of the others), which provides a principled, loss-aware alternative to magnitude for selecting which components to prune. Because the derivation assumes only a factorization θ = U ΣV ⊤ and never uses orthonormality of U, V , SVD-Surgeon can be applied as a corrective layer on top of a wide range of SVD-based compressors (Figure 1). We demonstrate this on SVD-LLM [12], a leading SVD-based method, where it improves perplexity across models and compression ratios, with the largest gains under aggressive compression. Post-training fine-tuning is orthogonal and can be layered on top of SVD-Surgeon, just as it can on any other compression method. Contributions. • We bring the Optimal Brain Surgeon framework to the singular-value basis, deriving a closed-form, training-free update of the retained singular values that compensates, to second order in the loss, for those removed by truncation. The same analysis yields a saliency that can be used to select which values to prune. • Since the derivation makes no orthonormality assumption, SVD-Surgeon applies on top of a broad class of SVD-based methods. Layered on SVD-LLM, a leading SVD-based method, it reduces perplexity across models and compression ratios, with the largest gains under aggressive compression.
2
Related Work
Second-order weight pruning. Using curvature to guide pruning dates back to Optimal Brain Damage [8], which scores weights by a diagonal Hessian approximation, and Optimal Brain Surgeon (OBS) [5], which uses the full inverse Hessian to derive both a saliency and a closed-form update of the surviving weights. Scaling this framework to modern networks motivated a line of layerwise approximations: the Optimal BERT Surgeon [7] extended it to transformers, while Optimal Brain Compression [2], GPTQ [4], and SparseGPT [3] apply OBS-style closed-form solutions in weight space for post-training quantization and unstructured pruning of LLMs, using the input Gram matrix as the layer-wise Hessian. LLM Surgeon [11] instead uses a Kronecker-factored approximate curvature (K-FAC) [10] for structured and unstructured weight pruning. All of these operate on the network weights (or their quantized values). In contrast, SVD-Surgeon applies the OBS framework in the basis of singular values, i.e. the parameters are the ℓ singular values of a layer weight matrix and the relevant Hessian is a small ℓ × ℓ matrix obtained by projecting the gradients onto the singular value space. SVD-based LLM compression. Truncated SVD is a hardware-friendly alternative to sparsity, and several methods improve on the naive truncation of the smallest singular values by making it data- or importance-aware. FWSVD [6] reweights the weight matrix by Fisher importance before decomposing. DRONE [1] minimizes the output rather than the weight approximation error using the input distribution, and ASVD [13] scales the weight by activation statistics. SVD-LLM [12], a leading SVD-based method, introduces a truncation-aware data whitening transform (a Cholesky factor of the input activation Gram matrix) that aligns singular-value magnitude with the reconstruction loss, and recovers accuracy through a LoRA-style fine-tuning of the decomposed factors. OBD-LLM [9] uses a K-FAC factorization of the task-loss Hessian as a metric, inducing a bidirectional (input- and output-aware) whitening under which truncated SVD yields a loss-optimal decomposition. All of these are concerned with how the decomposition is produced. SVD-Surgeon instead performs surgery in the OBS sense on a given decomposition, excising singular values and repairing the retained ones in closed form. Requiring no orthonormality of the factors, it composes on top of many such methods, which we demonstrate on SVD-LLM. 2
Pretrained layer weight θ
Host truncation (e.g. keep top-r σi )
Host decomposition θ = U ΣV ⊤
Compressed θ′ = Ur Σr Vr⊤
(U) Calibration grads ∂Ln Gn = ∂θ
Project & Assemble ḡ n = diag(U ⊤Gn V ) 1 H̄ = N
P
n ḡ
OBS compensation −1 δσS = H̄SS H̄SC σC
n n⊤
ḡ
Compressed θ′ = Ur Σ′r V ⊤ Σ′r = Σr +diag(δσS )
(S) Saliency selection σi2 /[H̄ −1 ]ii
Figure 1: Overview of SVD-Surgeon applied to a host SVD compressor (e.g. SVD-LLM). Top (blue): the host pipeline decomposes and truncates the weight matrix. Bottom (orange): SVD-Surgeon uses calibration gradients projected into the host’s singular-value basis to assemble a compact Hessian H̄ and compute a closed-form correction of the retained singular values. Vertical arrows show the information each SVD-Surgeon stage receives from the host. The update-only variant (U, solid) inherits the host’s pruned set, while the select-and-update variant (S, dashed) replaces it with a saliency-based selection.
3
SVD-Surgeon
3.1
Background: Optimal Brain Surgeon
Consider a pretrained model whose parameters θ have converged near a minimum of the loss L. Since the gradient is then negligible, the leading change in the loss under a perturbation δθ is second order, 1 δL ≈ δθHδθ (1) 2 where δθ H δθ denotes the quadratic form built from the Hessian H of L. Given a partition of the parameters into a retained set θS and a complementary set θC to be pruned, OBS uses (1) to answer the following question: if we prune (set to zero) the components θC , how do we shift the surviving entries δθS to optimally compensate for the induced increase in the loss? In the standard treatment θ is vectorized, so H is an ordinary square matrix and (1) a familiar quadratic form. We instead keep θ in its native matrix form, which makes the reduction to the SVD basis in the next subsection transparent. With θ a weight matrix, H is the fourth-order tensor Hij,kl =
3.2
∂2L , ∂θij ∂θkl
δθHδθ =
X
δθij Hij,kl δθkl .
(2)
i,j,k,l
Compression in the SVD basis
Classical OBS operates on the components, whether individual or entire rows or columns, of the matrix θ. We instead apply the same framework to singular values of θ. Write a general decomposition of the weight matrix θ ∈ Rm×n as θ=
ℓ X
σi ui vi⊤ = U ΣV ⊤
(3)
i=1
where ℓ = min(m, n), Σ = diag(σ1 , . . . , σℓ ), and ui , vi are the columns of U ∈ Rm×ℓ and V ∈ Rn×ℓ . We do not require U and V to be orthonormal, so the standard SVD (U ⊤ U = V ⊤ V = I) is a special case. For simplicity, and with a slight abuse of terminology, we continue to call the σi singular values and the u, vi singular directions even in this general, non-orthonormal case. 3
In this basis compression amounts to discarding singular triplets, i.e. setting σi = 0 removes the rank-one term σi ui vi⊤ . We partition the index set accordingly into the r retained values σS and the ℓ − r pruned values σC , with r the target rank. Pruning imposes δσC = −σC , and the OBS question becomes: what is the optimal update δσS of the retained singular values? In principle the directions ui , vi could also be relaxed, but here we update only the singular values, which is inexpensive and, as we show, already effective. 3.3
Reduction to singular-value space
We restrict attention to changes in the singular values alone, keeping the singular directions U, V fixed. This means the weight variation takes the form δθ = U δΣ V ⊤ with δΣ = diag(δσ). Substituting this into the quadratic model (1) and expanding in indices, X X 2 δL = δθij Hij,kl δθkl = (U δΣV ⊤ )ij Hij,kl (U δΣV ⊤ )kl (4) i,j,k,l
i,j,k,l
=
X
⊤ Uip δΣpp Vpj Hij,kl Ukq δΣqq Vql⊤
(5)
⊤ ⊤ δσp Upi Vjp Hij,kl Uqk Vlq δσq
(6)
δσp H̄pq δσq = δσ ⊤ H̄ δσ,
(7)
i,j,k,l p,q
=
X i,j,k,l p,q
=
X p,q
where in the third line we used that δΣ is diagonal, writing δΣpp = δσp , and collected the scalar factors δσp , δσq to the outside. Also, in the last line we defined X ⊤ ⊤ H̄pq ≡ Upi Vjp Hij,kl Uqk Vlq . (8) i,j,k,l
The fourth-order form in δθ then collapses to an ordinary quadratic form in the ℓ-vector δσ: 1 ⊤ δσ H̄ δσ. 2
δL ≈
(9)
Here σ stacks all ℓ singular values, comprising the retained σS and the pruned σC introduced above. The reduction (9) is the key practical simplification: H̄ is only ℓ × ℓ (at most min(m, n) in each dimension), whereas the weight-space Hessian in (2) is mn × mn. 3.4
Fisher approximation of the Hessian
We assume block-diagonality of the Hessian across layers, so that each layer’s projected Hessian H̄ can be estimated independently. Near convergence the per-layer Hessian is well approximated by the empirical Fisher information, a sum of outer products of per-sample gradients Gn = ∂Ln /∂θ, N
Hij,kl ≈
1 X n n G G , N n=1 ij kl
(10)
where N is the number of calibration samples. Inserting (10) into the definition of H̄ (8) and carrying out the contractions over i, j, k, l: 1 X X ⊤ ⊤ H̄pq = Upi Vjp Gnij Gnkl Uqk Vlq N n i,j,k,l
1 X ⊤ n = U G V pp U ⊤ Gn V qq N n 1 X n n = ḡ ḡ , N n p q 4
(11)
where in the second line, since p and q each appear in only one factor, the sum over all matrix indices collapses to the product of two diagonal entries of U ⊤ Gn V , leading to a further simplification of the projected Hessian. Defining the projected gradient ḡpn ≡ (U ⊤ Gn V )pp , this reads in matrix form N
1 X n n⊤ ḡ ḡ , H̄ = N n=1
ḡ n ≡ diag U ⊤ Gn V .
(12)
In fact, by the chain rule and ∂θij /∂σp = Uip Vjp , each component ḡpn is simply the singularvalue gradient ∂Ln /∂σp . These derivatives are not standard operations in automatic-differentiation frameworks. However, such frameworks readily provide the full gradient matrices Gn , from which ḡ n is obtained via a single matrix product U ⊤ Gn V and diagonal extraction. Similarly, H̄ coincides with the exact loss Hessian in singular-value coordinates, H̄pq = ∂ 2 L/∂σp ∂σq , but the Fisher outer-product form (12) avoids computing these second derivatives altogether, assembling the ℓ × ℓ Hessian entirely from first-order information. 3.5
Optimal singular-value update
In order to obtain the optimal update δσS of the retained singular values after pruning, we partition ⊤ H̄ conformally with (σS , σC ) into blocks H̄SS , H̄CC and H̄SC = H̄CS , so that ⊤ δL = 12 δσS⊤ H̄SS δσS + 12 δσC H̄CC δσC + δσS⊤ H̄SC δσC .
(13)
Pruning fixes δσC = −σC . Rather than enforcing this with a Lagrange multiplier, we equivalently substitute it directly and minimize the resulting unconstrained objective over δσS , δσS⋆ = argmin 12 δσS⊤ H̄SS δσS − δσS⊤ H̄SC σC . (14) δσS
⊤ H̄CC σC which is independent of δσS . Setting the Notice that we have dropped the term 12 σC derivative with respect to δσS to zero gives the optimal compensation in a simple closed form, −1 δσS⋆ = H̄SS H̄SC σC
(15)
which prescribes how the surviving singular values should shift to absorb, to second order, the effect of removing σC . 3.6
Saliency scores and pruning selection
Substituting the optimal update (15) back into (13) yields the loss increase induced by removing σC and optimally compensating σS . Expanding and simplifying, we get: ⊤ ⊤ δL = 12 δσS⊤ H̄SS δσS − σC H̄CS δσS + 21 σC H̄CC σC −1 −1 −1 ⊤ ⊤ ⊤ H̄SC σC + 12 σC H̄SC σC − σC H̄CS H̄SS = 12 σC H̄CS H̄SS H̄SS H̄SS H̄CC σC −1 ⊤ = 21 σC H̄CC − H̄CS H̄SS H̄SC σC .
(16) (17) (18)
The matrix in parentheses is the Schur complement of H̄SS in H̄. The first term H̄CC captures the naive cost of removing σC , while the second term accounts for the reduction afforded by optimally −1 adjusting σS . By the standard block-inversion identity, [H̄ −1 ]CC = (H̄CC − H̄CS H̄SS H̄SC )−1 , so the loss increase can equivalently be written as −1 ⊤ δL = 21 σC [H̄ −1 ]CC σC , (19) which recovers, in equivalent form, the OBS result obtained via Lagrange multipliers [7]. For the special case of removing a single singular value σi , this collapses to the familiar OBS saliency, δL =
σi2 , 2 [H̄ −1 ]ii
i ∈ C,
(20)
which scores each triplet by the loss it would incur if pruned, accounting for the optimal correction of the survivors. This leads to two variants of our method, both of which apply the same compensation (15) but differ only in how the pruned set C is chosen: 5
• Update-only (U). C is inherited from the host’s criterion (e.g. the smallest σi ), and we apply the compensation on top of it. • Select-and-update (S). We rank triplets by the saliency (20), take the lowest-scoring as C, and then apply the compensation. When H̄ ≈ I, the saliency reduces to σi2 /2, recovering magnitude-based selection. 3.7
Application to SVD-LLM
SVD-Surgeon assumes only a factorization θ = U ΣV ⊤ and does not use orthonormality of U, V , so it can be applied on top of any method that compresses by truncating such a factorization. We instantiate it on SVD-LLM, a natural host: conceptually simple yet among the strongest SVD-based compressors, and, since its whitening makes one of the factors non-orthonormal, a non-trivial test of this generality. SVD-LLM is built around truncation-aware data whitening. It takes the layer reconstruction error as the compression loss, min ∥θX − θ′ X∥F , (21) ′ θ
where X stacks the calibration activations, and whitens the inputs using a Cholesky factor S of the activation Gram matrix, XX ⊤ = SS ⊤ . Since (S −1 X)(S −1 X)⊤ = I, the loss equals ∥(θS − θ′ S)S −1 X∥F = ∥θS − θ′ S∥F , a plain Frobenius distance between the whitened weights, whose optimal rank-r solution is the truncated SVD of θS. Concretely, one computes θS = U ΣṼ ⊤ , truncates Σ to its top r values, and maps back through S −1 , θ′ = U Trunc(Σ) Ṽ ⊤ S −1 .
(22)
Cast in the form θ = U ΣV ⊤ of Eq. (3), the left factor U remains orthonormal while the right factor V ⊤ ≡ Ṽ ⊤ S −1 absorbs the inverse whitening transform and is no longer orthonormal. We apply SVD-Surgeon’s compensation (15) to this decomposition, inheriting SVD-LLM’s truncation set in the update-only variant or re-selecting it via (20) in select-and-update.
4
Experiments
4.1
Setup
Models and data. We evaluate on the OPT family (1.3B, 2.7B, 6.7B) and LLaMA-2-7B, reporting perplexity (↓) on WikiText-2 for all four models and on C4 for OPT-1.3B and OPT-2.7B. We sweep the compression ratio ρ (the fraction of parameters removed) from 20% to 80%, with emphasis on the high-compression regime, where truncation is most damaging. For an m × n weight, a rank-r factorization stores r(m + n) parameters, so ρ = 1 − r(m + n)/mn. A target ratio ρ thus corresponds to rank r = (1 − ρ) mn/(m + n), rounded to the nearest integer per layer. Calibration. SVD-Surgeon is single-shot: it estimates the Fisher information H̄ from a calibration set and produces the compressed model in one pass, with no gradient-based optimization. We draw this set from the same source as the host’s whitening data but keep it separate, since accurate Hessian estimation requires substantially more samples (N ) than whitening (Ncal ). Forming H̄ is the most expensive step of the pipeline, but it is a one-time, offline computation that parallelizes across batches. Implementation. Before the inversions in (15) and (20) we add a diagonal damping, dS and d, to H̄SS and H̄ respectively, and we scale the compensation update (15) by a factor λ to account for the approximate Hessian. For efficiency, we retain only the leading r + α(ℓ − r) block of H̄, corresponding to the top r singular values plus a fraction α of the remainder, and discard the rest. The hyperparameter settings for λ, N , the two damping coefficients dS , d, and α are reported in Appendix A. All experiments were conducted on a single NVIDIA H200 GPU. 1 1
Code: https://github.com/mahmoud-safari/SVD-Surgeon
6
Table 1: WikiText-2 perplexity (↓) vs. compression ratio across models. Dense perplexity: OPT1.3B = 14.62, OPT-2.7B = 12.47, OPT-6.7B = 10.86, LLaMA-2-7B = 5.47. Bold: best; underline: second best. Model
Method
0.2
0.3
0.4
0.5
0.6
0.7
0.8
OPT-1.3B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
17.82 17.50 17.49
20.69 19.62 19.59
27.28 23.37 23.31
47.79 31.29 31.39
140.82 53.63 53.46
654.89 168.48 163.41
4206.70 1728.88 1587.61
OPT-2.7B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
15.21 14.93 14.83
17.80 16.97 16.87
23.39 20.91 20.85
40.19 29.15 29.11
125.86 51.43 51.33
877.25 149.52 146.67
4584.67 1041.95 984.74
OPT-6.7B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
12.05 12.00 12.01
13.08 12.81 12.80
15.27 14.25 14.22
21.22 17.02 16.90
53.23 23.76 23.39
944.57 47.27 46.36
6777.57 316.47 279.88
LLaMA-2-7B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
8.38 8.34 8.20
10.67 10.52 10.36
16.15 15.71 15.59
33.28 31.49 31.14
89.97 84.04 82.50
253.40 241.71 232.76
570.44 549.14 531.62
Baseline. We layer SVD-Surgeon on SVD-LLM and compare against SVD-LLM with truncationaware whitening only: its LoRA-style fine-tuning recovery is orthogonal and can be applied on top of any method, including SVD-Surgeon, just as SVD-LLM applies it to its own decomposition. We report both variants of our method: update-only (U), which inherits the host’s truncation set and applies the compensation (15), and select-and-update (S), which additionally re-selects the pruned set via the saliency (20). 4.2
Results
Perplexity. Table 1 reports WikiText-2 perplexity for SVD-Surgeon layered on SVD-LLM (whitening only), across compression ratios for OPT-1.3B, OPT-2.7B, OPT-6.7B, and LLaMA-2-7B. the corresponding perplexity curves are plotted in Appendix B. C4 results (OPT-1.3B, OPT-2.7B) are given in Table 2. We report both variants, update-only (U) and select-and-update (S). Across all four models, SVD-Surgeon improves on SVD-LLM, and the gains grow with the compression ratio. under mild compression there is little to compensate, while under aggressive compression SVD-Surgeon prevents the steep degradation that SVD-LLM suffers (e.g. WikiText-2 perplexity on OPT-6.7B, 944.57 → 46.36 at ratio 0.7). Most of this improvement comes from the closed-form update. Re-selecting the pruned set via the saliency (S) adds a smaller, further gain in most settings. SVD-LLM is deterministic given fixed calibration data. The Fisher estimate in SVD-Surgeon introduces small variance through CUDA non-determinism in the gradient computation; for OPT models we report the mean over 3 seeds (standard deviations are given in Appendix B), while for LLaMA-2-7B the results were identical across seeds and we report a single value. Table 2: C4 perplexity (↓) vs. compression ratio. Dense perplexity: OPT-1.3B = 15.68, OPT2.7B = 14.06. Bold: best; underline: second best. Model
Method
0.2
0.3
0.4
0.5
0.6
0.7
0.8
OPT-1.3B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
20.08 18.99 18.98
24.88 21.68 21.69
37.83 27.01 26.89
84.20 38.72 38.54
329.17 74.61 75.41
1360.97 293.13 278.78
4220.29 3562.08 2915.40
OPT-2.7B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
17.60 16.73 16.70
21.48 18.99 18.91
32.02 23.72 23.56
63.00 34.19 34.00
211.21 63.18 63.28
1256.38 220.83 228.09
7113.83 4155.44 3664.16
Compression time. Figure 2 shows wall-clock compression time across ratios for LLaMA-2-7B and OPT-2.7B (the pattern is similar for the remaining models). This is defined as the time of the pruning algorithm itself, including loading the precomputed H̄ from disk for SVD-Surgeon, but excluding 7
(a) LLaMA-2-7B
(b) OPT-2.7B
Figure 2: Wall-clock compression time (seconds), measured on a single H200 GPU, across compression ratios. The pattern is similar for the remaining models. model and calibration-data loading, which are shared across all methods. The update-only variant adds modest overhead as it only solves a single linear system against H̄SS . The select-and-update variant is more expensive because it requires a pseudoinverse of the full (block-truncated) H̄ for the saliency scores. Assembling H̄ itself costs roughly 7.6 s per calibration sample for LLaMA-2-7B (2.6 s for OPT-2.7B). This is a one-time computation, parallelisable across samples and layers, whose result is reused across all compression ratios.
5
Conclusion
We introduced SVD-Surgeon, a method that brings the Optimal Brain Surgeon framework to the singular-value basis. By treating singular values as first-class parameters and building a compact, Fisher-estimated Hessian in that coordinate system, SVD-Surgeon derives a closed-form update of the retained singular values that compensates for those removed by truncation, as well as a compensationaware saliency that can replace magnitude-based selection. The entire procedure is training-free, requires no iterative optimization, and, because it assumes no orthonormality of the factors, composes directly on top of existing SVD compressors. Applied to SVD-LLM, SVD-Surgeon improves the perplexity-compression trade-off across models and compression ratios, with the largest gains in the aggressive regime where standard truncation is most damaging. Limitations. The current evaluation measures perplexity on two benchmarks (WikiText-2 and C4); validating on additional hosts, model families, and downstream tasks would strengthen the generality claim. By design, the method updates only the singular values while holding the directions U, V fixed. This is a deliberate simplification. The ℓ-dimensional σ-space is far smaller than the full parameter space, keeping both the derivation and the computation tractable, and the experiments show it can have a substantial effect, particularly under aggressive compression. Whether additionally varying U and V within the same second-order framework yields further gains that justify the added computational cost is an open question. Estimating the Fisher information requires a forward and backward pass over a calibration set for each layer, which can be expensive for large models, although in practice this is a one-time, offline computation that parallelises across both layers and samples. Finally, SVD-Surgeon introduces several hyperparameters (Appendix A). Default values proved robust across most configurations, but the interaction between these settings and factors such as model scale or Fisher accuracy is not yet fully understood. A principled selection scheme would make the method more plug-and-play. Future work. Several directions follow naturally. SVD-Surgeon could be layered on other SVDfamily compressors to test composability more broadly, and combined with post-hoc fine-tuning to measure whether the two recovery mechanisms stack, or paired with quantization of the corrected factors for additional compression. The per-component saliency is expressed in units of loss and is therefore comparable across layers, which could enable global, cross-layer rank allocation as a natural extension. More ambitiously, relaxing the fixed-direction assumption to allow joint updates of U , Σ, and V within the same second-order framework could close the gap left by freezing the singular directions. 8
Acknowledgements This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH_2698644) and the state of BadenWürttemberg.
Frank Hutter acknowledges the financial support of the Hector Foundation.
References [1] Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Drone: Data-aware lowrank compression for large nlp models. Advances in neural information processing systems, 34:29321–29334, 2021. [2] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate posttraining quantization and pruning. Advances in Neural Information Processing Systems, 35:4475– 4488, 2022. [3] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR, 2023. [4] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations. [5] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992. [6] Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022. [7] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 4163–4181, 2022. [8] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. [9] Yuhang Li, Donghyun Lee, Ruokai Yin, and Priyadarshini Panda. Optimal brain decomposition for accurate llm low-rank approximation. arXiv preprint arXiv:2604.00821, 2026. [10] James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417. PMLR, 2015. [11] Tycho van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The llm surgeon. In International Conference on Learning Representations, volume 2024, pages 13245–13255, 2024. [12] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations. [13] Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
9
A
Hyperparameter Settings
Tables 3 and 4 list, respectively, the hyperparameters introduced in Section 4.1. The compensation scaling λ, the damping coefficients dS and d (expressed as fractions of the mean diagonal of H̄SS and H̄, respectively), the block-truncation fraction α, and the number of Fisher samples N . For each model the same settings are used across all compression ratios, and only across models do they differ. The three hyperparameters λ, dS , d were chosen via light manual exploration starting from natural defaults (e.g. λ = 1) rather than aggressive tuning. The final values were held fixed across all compression ratios, which preserves the single-shot character of the method. Notice that the compensation scaling λ has been set to 1 for all OPT models and to 0.1 for Llama 2-7B. The block-truncation fraction α controls a trade-off between accuracy and cost: larger values retain a greater portion of H̄, improving the fidelity of the compensation update at the expense of higher computational overhead for Fisher collection and the inversion and matrix operations that follow. We fix α = 0.3 across all models and compression ratios, which we found to offer a good balance. The number of Fisher samples N was chosen large enough for the projected Hessian H̄ to be well converged but was not tuned as a hyperparameter. The hyperparameters of SVD-LLM itself, including the number of whitening calibration samples (Ncal = 256), are kept identical across both methods to ensure a fair comparison. Table 3: Hyperparameter settings for SVD-Surgeon on WikiText-2. Settings are fixed across all compression ratios for a given model. Model λ dS d α N 1 1 1 0.1
OPT-1.3B OPT-2.7B OPT-6.7B LLaMA-2-7B
10−5 10−5 10−5 10−5
0.3 0.3 0.3 0.3
1 1 10−7 0.1
16384 24576 32768 32768
Table 4: Hyperparameter settings for SVD-Surgeon on C4. Settings are fixed across all compression ratios for a given model. Model λ dS d α N OPT-1.3B OPT-2.7B
B
10−2 10−5
1 1
1 10−5
0.3 0.3
16384 24576
Extended WikiText-2 Results
Figure 3 plots WikiText-2 perplexity as a function of compression ratio for all four models. We restrict the range to 0.3–0.7 for visual clarity. The improvements are most apparent at aggressive compression, where the curves separate. Only the update-only variant (U) is shown. The variant (S) tracks it too closely to be distinguishable at this scale. Precise per-ratio values are given in Table 1. Table 5 additionally reports standard deviations over three seeds for the OPT family. Table 5: WikiText-2 perplexity (↓) vs. compression ratio (mean ± std over 3 seeds). Bold: best; underline: second best. Model
Method
0.3
0.4
0.5
0.6
0.7
0.8
OPT-1.3B
SVD-LLM 17.82 SVD-Surgeon (U) 17.50±0.01 SVD-Surgeon (S) 17.49±0.01
0.2
20.69 19.62±0.01 19.59±0.02
27.28 23.37±0.03 23.31±0.04
47.79 31.29±0.05 31.39±0.07
140.82 53.63±0.04 53.46±0.17
654.89 168.48±3.43 163.41±1.59
4206.70 1728.88±76.79 1587.61±111.53
OPT-2.7B
SVD-LLM 15.21 SVD-Surgeon (U) 14.93±0.02 SVD-Surgeon (S) 14.83±0.02
17.80 16.97±0.03 16.87±0.03
23.39 20.91±0.01 20.85±0.02
40.19 29.15±0.09 29.11±0.09
125.86 51.43±0.37 51.33±0.29
877.25 149.52±3.08 146.67±1.84
4584.67 1041.95±26.76 984.74±24.76
OPT-6.7B
SVD-LLM SVD-Surgeon (U) SVD-Surgeon (S)
13.08 12.81±0.01 12.80±0.01
15.27 14.25±0.01 14.22±0.01
21.22 17.02±0.01 16.90±0.02
53.23 23.76±0.08 23.39±0.07
944.57 47.27±0.10 46.36±0.05
6777.57 316.47±6.31 279.88±8.63
12.05 12.00±0.01 12.01±0.01
10
(a) OPT-1.3B
(b) OPT-2.7B
(c) OPT-6.7B
(d) LLaMA-2-7B
Figure 3: WikiText-2 perplexity vs. compression ratio for all four models. SVD-Surgeon (U) is shown, while variant (S) is omitted as it closely tracks (U) at this scale. See Table 1 for per-ratio values.
11