arXiv CS open access

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression Viacheslav Yusupov1∗ , Daria Cherniuk 2 , Evgeny Frolov2, 1 1

arXiv:2609.02451v1 [cs.LG] 2 Sep 2026

HSE University 2 AXXX v.yusupov.lab@gmail.com

Abstract In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analysis for billion-parameter networks where full computation is infeasible. Our approach reveals consistent vulnerability patterns: value projection layers exhibit the highest sensitivity and strongest cross-layer correlations across multiple model families, while other components exhibit architecture-specific behaviors. Through extensive experiments on quantization, sparsification, inter-layer corruption, and post-corruption finetuning, we demonstrate that our approximation strongly correlates with both performance degradation and recovery. Our framework provides a practical, theoretically grounded tool for identifying fragile components in large models, opening new avenues for guided compression and optimization strategies, such as mixed-precision allocation, layer-wise sparsity, and adaptive low-rank decomposition across layers and even individual weight groups.

Introduction

Understanding the curvature of the loss landscape is fundamental to deep learning, underpinning principled advances in optimization (Izmailov and Solodov 2014; Zhao et al. 2025), model compression (Frantar et al. 2023), fine-tuning (Liu et al. 2024), and interpretability (Wu et al. 2025). This curvature is captured by the Hessian matrix and its expectation, the Fisher information matrix, which indicate which parameters matter most and how perturbations propagate through the model. Due to complexity of computing and analyzing Hessian and Fisher matrix for large models, many practical models neglect cross-layer interactions and study and transform layers independently (Chekalina et al. 2025) or use only diagonal or block diagonal (Zhang et al. 2017) part of Hessian for further applications. Theoretical work that does study the full structure is instead restricted to small models on synthetic data (Dong et al. 2025) or small datasets (e.g. MNIST (LeCun and Cortes 1998)). Bridging this gap requires approximations that are both principled and scalable. We propose such an approximation, based on the Kronecker factorization framework. We show that the Fisher ∗

Corresponding author.

matrix admits an exact decomposition into a sum of Kronecker products, and that truncating it, together with an exact diagonal and a matrix-free eigensolver, reduces memory complexity from quadratic to linear in model size while retaining substantially richer structure than diagonal and blockdiagonal methods. Using this approximation, we provide direct empirical evidence of non-diagonal Hessian structure in LLMs, confirming theoretical predictions (Dong et al. 2025) at a scale where the full matrix cannot be computed. Our contributions are as follows: • We propose a scalable Kronecker-based approximation of the Fisher matrix that reduces memory complexity from quadratic to linear in model size. • We provide empirical evidence of non-diagonal Hessian structure in large language models, validated on four LLMs from 350M to 7B parameters. • We demonstrate a strong correlation between layer-wise Hessian values and layer sensitivity to quantization and sparsification. • We show that off-diagonal curvature predicts inter-layer effects: coupled layers suffer excess damage when compressed jointly, beyond the sum of their individual contributions, and fine-tuning the layers most strongly coupled to the corrupted ones recovers the most performance.

Related Work

Curvature approximation in deep learning. Exact second-order information is central to Newton-type optimization (Izmailov and Solodov 2014), and closed-form expressions for network Hessians are well understood (Naumov 2017; Botev, Ritter, and Barber 2017), but the full Hessian or Fisher information matrix (FIM) is quadratic in the parameter count and thus intractable beyond small models. Scalable methods therefore impose structure: Hessian-free optimization avoids materializing the matrix via Hessian–vector products (Martens 2010); K-FAC approximates the per-layer Fisher as a Kronecker product (Martens and Grosse 2015; Van Loan 2000); and block-diagonal schemes discard all inter-layer blocks (Collobert 2004; Zhang et al. 2017; Dangel, Harmeling, and Hennig 2020). At the largest scales, practice degrades further to diagonal curvature, either implicitly through adaptive first-order methods (Kingma and Ba 2015;

Das et al. 2024; Zhang et al. 2025) or explicitly through diagonal Hessian estimates (Yao et al. 2021; Liu et al. 2024), with low-rank preconditioners as a middle ground (Matveeva, Katrutsa, and Frolov 2025). Yet the structure these approximations discard is not negligible: transformer Hessians exhibit strong heterogeneity across parameter blocks (Zhang et al. 2024), and the prevailing near-block-diagonal picture is only approximate, with theory attributing it to architectural and output-dimension effects rather than to genuinely vanishing cross-layer terms (Dong et al. 2025). No existing estimator captures this cross-layer curvature at billion-parameter scale. Curvature-guided compression. Second-order sensitivity underlies much of modern model compression. The optimal brain compression framework and its LLM-scale successor use the Hessian of a layer-wise reconstruction loss to decide which weights to prune or how to round them during quantization (Frantar and Alistarh 2022; Frantar et al. 2023), and related sensitivity estimates drive layerwise mixed-precision bit allocation (Zhao et al. 2026). In lowrank compression, Fisher information reweights the decomposition toward task-sensitive directions, progressing from diagonal approximations (Hsu et al. 2022) to a Kroneckerfactored approximation of the observed FIM within each layer (Chekalina et al. 2025). Curvature further informs recovery after compression via Hessian-guided zeroth-order fine-tuning (Zhao et al. 2025). Crucially, all of these methods treat each layer independently, even though compression errors demonstrably propagate and interact across layers (Arai and Ichikawa 2026) - precisely the structure a layer-local curvature estimate cannot see. Our positioning. Across optimization and compression, a consistent pattern emerges: scalable methods estimate sensitivity independently per layer, and compression pipelines allocate precision, sparsity, or rank from layer-local proxies, even though errors demonstrably interact across layers (Arai and Ichikawa 2026). The closest work to ours, GFWSVD (Chekalina et al. 2025), captures intra-layer parameter correlations through a Kronecker-factored Fisher but remains a per-layer estimator tied to a single compression task. We extend Kronecker-based Fisher approximation to the crosslayer setting, yielding a tractable estimate of inter-layer curvature for billion-parameter models, and show that a single such estimate transfers across compression problems: quantization, sparsification, as well as predicting recovery under fine-tuning.

Hessian Kronecker Approximation

Let W1 , . . . , Wk be the weight matrices of a deep neural network. We collect their parameters into a single vector w = [w1 , w2 , . . . , wk ] ∈ Rd , where wi = vec(Wi ), vec(·) denotes the vectorization operation, and d is the total number of parameters. Let g(w; x) ∈ Rd denote the gradient of the loss with respect to w on a data sample x. We approximate the Hessian H(w) of the loss by the empirical Fisher matrix J(w), defined as the expected outer product of the gradients over the data distribution. This approximation holds under the assumption that the model weights lie near a local optimum

of the loss, a condition that is typically satisfied by converged pre-trained models H(w) ≈ J(w) = Ex g(w; x) g(w; x)⊤ ∈ Rd×d . (1) As can be observed, the number of parameters in both the Hessian and the Fisher information matrix scales quadratically with the number of model parameters. Consequently, storing and manipulating full matrices becomes infeasible even for moderately sized neural networks, due to prohibitive memory and computational requirements. To avoid full matrix construction we propose our Kronecker-based Fisher matrix approximation. We first vectorize the Fisher matrix (1) and express it in terms of Kronecker products: 2

vec(J(w)) = Ex [vec(g g ⊤ )] = Ex [g ⊗ g] ∈ Rd ,

(2)

where ⊗ denotes the Kronecker product and we write g = g(w; x) for brevity. Reshaping the gradient into a matrix G ∈ Rn×m with g = vec(G), equation (2) becomes Ex [g ⊗ g] = Ex [vec(G) ⊗ vec(G)]. The Kronecker product of vectorizations can be converted into the vectorization of a Kronecker product using the symmetric permutation 2 2 2 2 matrix P = In ⊗ Knm ⊗ Im = P ⊤ ∈ Rn m ×n m , where Knm is the commutation matrix: Ex [vec(G) ⊗ vec(G)] = Ex [P ⊤ vec(G ⊗ G)] R X σi ui vi⊤ , = P ⊤ vec(Ex [G ⊗ G]) = P ⊤ vec

(3)

i=1

where the last equality uses the full-rank SVD (Golub and Reinsch 1971) of Ex [G ⊗ G] with rank R, singular values 2 2 σi , and singular vectors ui ∈ Rn , vi ∈ Rm . Since ui and vi are themselves vectorizations of matrices Ui ∈ Rn×n and Vi ∈ Rm×m , we can rewrite (3) as P ⊤ vec

R X

R X σi ui vi⊤ = P ⊤ σi vec(Ui ) ⊗ vec(Vi )

i=1

R X

σi vec(Ui ⊗ Vi ) = vec(J(w)),

i=1

(4) where the second equality uses the identity vec(Ui ⊗ Vi ) = (In ⊗ Knm ⊗ Im )(vec(Ui ) ⊗ vec(Vi )). Equation (4) yields an exact Kronecker decomposition of PR the Fisher matrix, J(w) = i=1 σi Ui ⊗ Vi , which we approximate by truncating the sum to the r < R largest singular values: J(w) ≈ J(w) =

r X

σi Ui ⊗ Vi .

(5)

i=1

By the Eckart–Young theorem, this truncation is the best rank-r approximation of Ex [G ⊗ G] in the Frobenius norm. Computing the decomposition (4) directly is infeasible, 2 2 since Ex [G ⊗ G] ∈ Rn ×m is large. We therefore never form this matrix explicitly and instead obtain its leading singular values and vectors with an implicitly restarted Arnoldi

method (Lehoucq, Sorensen, and Yang 1998), which accesses the matrix only through products with vectors. Exploiting the identity (G ⊗ G) vec(V ) = vec(GV G⊤ ), these products reduce to small dense multiplications with the reshaped gradients: B

matvec(v) =

1 X vec(Gb V G⊤ b ), B b=1

√ which for a balanced factorization n ≈ m ≈ d becomes √ O(Bd(T + NA d)). The memory footprint consists of the gradient statistics, the diagonal, and the r pairs of Kronecker factors, O(d + r(n2 + m2 )) = O(rd), which is linear in the number of parameters for a fixed rank r, in contrast to the O(d2 ) required to store the Fisher matrix explicitly. We provide the end-to-end Hessian approximation construction times in Section 4.5.

(6)

1 X vec(G⊤ rmatvec(u) = b U Gb ), B b=1

where Gb ∈ Rn×m is the reshaped gradient accumulated over batch b, and u = vec(U ) with U ∈ Rn×n and v = vec(V ) with V ∈ Rm×m correspond to the two Kronecker factors. To improve the approximation further, we compute the diagonal of the Fisher matrix exactly, diag(J(w)) = Ex [g ⊙ g], where ⊙ denotes the element-wise product, and substitute it into the low-rank decomposition. Pr The final approximation is therefore J(w) = i=1 σi Ui ⊗ Vi with its diagonal replaced by Ex [g ⊙ g], which is exact. Table 1 shows that significantly improves the approximation. Compressed Visualization Storing every entry of J(w) for visualization would require O(d2 ) memory, as much as the full Fisher matrix. We therefore display a compressed form, in which each m×m block of the matrix is summarized by a single value, yielding an n × n image: r m 1 X X σi Ui · (V ) Jvis (w) = , i pq m2 p,q=1 i=1 (7) jm X 2 1 Jvis (w) jj = Ex gp . m t=(j−1)m+1

The off-diagonal entries of Jvis (w) are the mean values of the corresponding blocks, which follows from the block structure (Ui ⊗ Vi )jk = (Ui )jk Vi , while the diagonal entries average the exact diagonal of the Fisher matrix within each block. The resulting image requires only O(n2 ) memory, and the Kronecker factors it is√built from require O(r(n2 + m2 )), which for n ≈ m ≈ d is comparable to the size of the model itself for small r, as opposed to O(d2 ) for the full matrix. Computational Complexity We now estimate the time and memory requirements of our method for a model, or a part of it, with d = nm parameters, using B batches of T tokens each. A backward pass costs O(T d) per batch, so computing the gradients takes O(BT d); accumulating the exact diagonal from the per-batch gradients adds O(Bd). A single iteration of the Arnoldi method requires one product of the form (6) per batch, that is O(B(n2 m + nm2 )) = O(Bd(n + m)) operations, and assembling the compressed visualization from the resulting factors takes O(r(n2 +m2 )). With NA Arnoldi iterations, the total time complexity is O BT d + Bd + NA Bd(n + m) + r(n2 + m2 ) = O Bd (T + NA (n + m)) ,

4 4.1

Experiments

Validating the Hessian Approximation

Following Dong et al. (2025), we first validate our Hessian approximation on a small model where the true Hessian can be computed exactly: a two-layer perceptron trained with binary cross-entropy loss and the AdamW optimizer (Loshchilov and Hutter 2017) on a synthetic dataset of 500 Gaussian clusters. The input and output dimensions are 500 and the hidden dimension is 8, for a total of 8k parameters. Original

0.004

Approximated

0.004

2000

0.003 2000

0.003

4000

0.002 4000

0.002

6000

0.001 6000

0.001

2000

4000

6000

0.000

2000

4000

6000

0.000

Figure 1: The true Hessian H of the two-layer perceptron and its rank-16 Kronecker-based approximation J. Axes denote the parameter index. As Figure 1 shows, the Kronecker-based approximation J recovers the structure of the true Hessian H. To quantify the agreement, we compute the coefficient of determination ∥H−J∥2 R2 (H, J) = 1 − ∥H∥2 F , which reaches 42.3% with the F explicit diagonal computation from (7) and 29.9% without it. Table 1 details how R2 varies with the rank of our Fisher matrix approximation: the explicit diagonal consistently improves the fit, with the largest relative gains at low ranks. rank r

w/o diag w/ diag

3.2 17.8

9.8 26.3

15.6 32.7

24.5 36.1

29.9 42.3

Table 1: R2 score (%) between the true Hessian and our approximation for different ranks r, with and without explicit diagonal computation.

All subsequent experiments are conducted on four large language models spanning different families and sizes: OPT350M (Zhang et al. 2022), Qwen2-0.5B (Yang et al. 2024), OLMo2-1B (Team OLMo et al. 2024), and Qwen2.5-7B (Yang et al. 2024). For evaluation, we use text from the

1e−8

1e−7

3.5

v_proj

3.0

1e−7 q_proj

attn

k_proj

o_proj

q_proj

2.5

out_proj

k_proj v_proj o_proj

gate_proj

gate_proj 5

2.0 4

fc1

up_proj 3

1.0

(b) Qwen2.5-0.5B

pr oj

ro j

_p up

pr oj te _

p k_ roj p v_ roj p o_ roj pr oj

down_proj

ro j _p up

pr oj

(a) OPT-350M

pr oj

te _

0.0

down_proj

2 fc

1 fc

k_ pr o v_ j pr o q_ j p ou roj t_ pr oj

0.5

a o_ ttn pr oj

fc2

up_proj

1.5

Figure 2: Kronecker-Fisher approximation of the Hessian (Fisher Information) for one Transformer block of OPT-350M, Qwen2.5-0.5B, and OLMo2-1B. For the Qwen2.5 architecture, where the K- and V -projections are very small and most of the Transformer block parameters belong to the MLP linear layers, we group the entire attention mechanism (Q, K, V ) under a single label attn rather than labeling each projection individually. We also observed that the Hessian structure of Qwen2.5-7B is the same as Qwen2.5-0.5B. WikiText2 corpus (Merity et al. 2016). We first construct approximate Hessians over several batches of WikiText2, computing gradients over groups of full Transformer blocks with N parameters and visualizing the approximation in the com√ pressed form (7), where we set n ≈ N , n ∈ N. In the main paper, we restrict the Hessian visualizations to the level of individual Transformer blocks, as the full matrix is too large to display legibly even in the compressed form (7). Full-model and multi-block visualizations are provided in the Supplementary. As our next step, we test whether the approximate Hessian predicts layer sensitivity to compression: first within groups of layers of the same type (query, value, up-projection, etc.; Figure 3), and second across layer types, by compressing layers of different types (Figure 4).

4.2

Compression within a Single Layer Type

To connect our Kronecker-based Fisher matrix approximation to practical model compression, we study how sensitive individual layer types are to weight corruption. In this section, we compress only one layer type at a time (e.g., only the V -projections or only the upscale layers); interactions between different layer types within the same Transformer block are studied in Section 4.3. In both settings, we compress the corresponding layers in all Transformer blocks except the first and last three, and measure the resulting increase in perplexity on WikiText2. We exclude these boundary blocks because they often exhibit out-of-distribution behavior, and our Hessian approximations indicate weaker correspondence between their parameters (see Supplementary). Quantization. We first quantize the selected layers with 4-bit uniform quantization. As shown in the first row in Figure 3, the V -projection causes the highest perplexity increase in all models except OLMo2-1B, where the downscale projection is the most vulnerable. These results are consistent

with the corresponding Hessian approximations (Figure 2): the largest values appear on the V -projection for the OPT (Figure 2a) and Qwen (Figure 2b) models, and on the downscale projection for OLMo2 (Figure 2c). The remaining sensitive layers are model-specific, but in each case they match the layers our approximation highlights: quantizing the upscale layer causes a significant quality drop in OPT-350M, consistent with its high values in Figure 2a; the K-projection is vulnerable in Qwen2-0.5B and Qwen2.57B, in line with Figure 2b; and in OLMo2-1B, quantizing the V - and Q-projections and the downscale layer leads to a substantial perplexity increase, matching the largest values of our Kronecker-Fisher approximation in Figure 2c. Sparsification. We observe the same pattern under sparsification with ratio 0.5 (second row in Figure 3). The V projections exhibit the highest sensitivity in all models, in agreement with the Fisher matrix results in Figure 2. The only model whose vulnerability ranking differs between quantization and sparsification is OLMo2: the V -projection is the most sensitive to sparsification, with the downscale projection in second place, whereas for quantization their order is reversed. Both layers exhibit the highest values in the Hessian approximation for OLMo2 (Figure 2c), with the downscale projection being the more pronounced of the two. This discrepancy may stem from our normalization scheme: since we normalize the perplexity increase by the number of parameters in the compressed layer, and the V -projection is much smaller than the downscale projection, the per-parameter effect of the V -projection is amplified. Overall, across both compression schemes, the layers assigned high values by our Fisher matrix approximation are exactly those whose corruption degrades model quality the most, with the V -projection being the most vulnerable layer type throughout.

Qwen2-0.5B

1e−7

1.0

v_ pr o k_ j pr o o_ j pr ga oj te do _p w roj n_ up pro _p j r q_ oj pr oj

n_ v_pro pr j o q_ j pr o_ oj pr ga oj te _ up pro _p j r k_ oj pr oj

v_ pr o k_ j pr o o_ j pr up oj _p r q_ oj pr ga oj te do _p w roj n_ pr oj

0.0

1.5

Qwen2.5-7B

1e−8

2.5 0.0

OLMo2-1B

1e−7

5.0

0.5

7.5

1e−7

1e−8

1e−9

1.5 4

1.0

0.0

v_ pr o k_ j pr o o_ j pr up oj _ do pro w j n_ ga p r te oj _p q_ ro pr j oj

0.5

v_ pr oj w n_ p o_ ro pr j up oj _p ga ro j te _p q_ ro pr j o k_ j pr oj

v_ pr o k_ j pr o o_ j pr up oj _p r q_ oj do pro j w n ga _p r te oj _p ro j

0.5

v_ pr oj fc 1 q_ pr o k_ j pr oj ou t_ pr oj fc 2

ΔPPL / n_params

Sparsification

1e−6

OPT-350M

v_ pr oj fc 1 q_ pr o k_ j pr oj ou t_ pr oj fc 2

ΔPPL / n_params

Quantization

1e−7 1.0

Figure 3: Increase in Perplexity scores following 4-bit layer quantization (top row) and 50%-sparsification (bottom row) of various Transformer block components in the middle layers (excluding first 3 and last 3 transformer blocks) of the OPT-350M, Qwen2-0.5B, OLMo2-1B and Qwen 2.5-7B models. The scores are normalized by the number of parameters in the quantized layers to compare the relative effects of quantization across layers with different parameter counts. The layers are sorted in descending order based on the value of perplexity change.

Inter-Layer Quantization and Sparsification

opt-350m

qwen2-0.5b

2.5

n ow

Delta PPL

0.2

0.1

Down

n ow

at e

0.0

U p ow n D

at e G

0.0

0.3

Gate

Down

0.4

0.5

V O

0.6

Delta PPL

1.0

Gate

1.5

0.7

2.0

0.0

qwen2.5-7b

2.5

0.5

Down

olmo2-1b

1.0

FC2

Gate

FC1

1.5

2.0

Delta PPL

3 V

Delta PPL

To study inter-layer interactions in the Fisher matrices, we also quantize and sparsify pairs of layers, such as the V projection together with the upscale layer. As in Section 4.2, we apply 4-bit quantization and sparsification with ratio 0.5 to all Transformer blocks except the first and last three, and evaluate the corrupted models on WikiText2 (Figures 4 and 5). The greatest perplexity increase occurs for pairs involving the V -projection and for pairs combining the upscale and downscale layers. Note that Figures 4 and 5 are not normalized by the number of parameters, so pairs of large layers naturally appear brighter. What matters is which pairs stand out beyond this size effect, and these are precisely the pairs highlighted by our Hessian approximations: the V –FC1 pair for OPT, consistent with Figure 2a; the V –upscale pair for Qwen2.5, consistent with Figure 2b; and the V –downscale pair for OLMo2, consistent with Figure 2c. Therefore, the off-diagonal structure of our Kronecker-based Fisher matrix approximation predicts not only which individual layers are sensitive to compression, but also which pairs of layers interact most strongly when corrupted together. A natural concern is that the highlighted pairs in Figures 4 and 5, such as V –upscale, merely reflect the accumulation of two independent errors: if quantizing V and quantizing FC1 each degrade the model on their own, their combination would stand out even without any genuine interaction between the layers. To rule out this explanation, we compute the interaction difference DI for pairs of layers. Formally, for layers P and Q, DI (P, Q) = ∆({P, Q}) − ∆({P }) − ∆({Q}), where ∆(S) denotes the perplexity increase after corrupting the

4.3

Figure 4: The Perplexity growth following quantization in 4 bits the different pairs of layers in Transformer blocks in the middle of the LLM models. The value on crossing the row i and the column j is the result of PPL growth after quantization of layers i and j.

olmo2-1b

Delta PPL

0.1

Down

0.0

Gate Up −0.5

O at e U D p ow n G

Down

Delta PPL (interaction)

V O

O FC 1 FC 2

−1.0

0.5

ow n

Figure 6: Interaction difference DI for 4-bit quantization in OPT-350M and OLMo2-1B. The cell at row i and column j shows DI (i, j): the perplexity increase after jointly quantizing layer types i and j, minus the sum of the perplexity increases after quantizing each layer type individually. Positive values indicate that the joint corruption is more damaging than the two individual corruptions combined.

0.0

p ow n D

at G

0.0

Down

0.2

0.5

0.3

O Gate

1.0

1.5

O Gate

0.4

2.0

−0.5

FC1

olmo2-1b

0.5

Delta PPL

ow n

p U

2.5

0.0

FC2

qwen2.5-7b

0.0

0.5

Up Down

Gate

1.0

0.5

FC2

1.5

FC1

2.0

Delta PPL

K V

1.0

2.5

opt-350m

qwen2-0.5b Q

Delta PPL

opt-350m

Figure 5: The Perplexity growth following the sparsification with ratio 0.5 the different pairs of layers in Transformer blocks in the middle of the LLM models. The value on crossing the row i and the column j is the result of PPL growth after sparsification of layers i and j.

Quantization

qwen2-0.5b

olmo2-1b

+ Fine-tune o

+ Fine-tune k

+ Fine-tune q

+ Fine-tune v

Quantized MLP 2

qwen2-0.5b

Sparsification

layers in set S. By construction, DI removes the additive contribution of each layer and isolates the excess degradation caused by corrupting the two layers jointly. The largest interaction differences between attention and MLP layers occur precisely for the pairs with the highest values in our Hessian approximations (Figure 2), which are the pairs involving the V -projection. This confirms that the prominence of these pairs is a genuine interaction effect, captured by the off-diagonal structure of the Fisher matrix, rather than an artifact of cumulative independent errors. An exception to the otherwise strong agreement between our Fisher matrix approximation and the empirical results is the O-projection. While the corresponding regions of the approximated Hessians show comparatively low values in all models, pairs involving the O-projection, particularly O–upscale, cause a notable perplexity increase under both quantization and sparsification (Figures 4 and 5). We hypothesize that the discrepancy arises from the local nature of the Fisher approximation. The O-projection writes the attention output directly into the residual stream, which is known to carry a small number of high-magnitude outlier channels that the MLP layers rely on. Near the trained optimum, gradients through the O-projection are small, yielding low Fisher values; however, the finite perturbations introduced by 4-bit quantization or 50% sparsification disturb these outlier channels, causing damage that a local quadratic model cannot capture. We discuss the further in the Future Research (Section 5). We additionally evaluate the corrupted models on five zero-shot benchmarks (PIQA, WinoGrande, HellaSwag, ARC-Easy, ARC-Challenge), where the resulting layer rankings closely match those obtained from perplexity. Full pertask results are reported in Supplementary.

Quantized MLP

olmo2-1b

+ Fine-tune o

+ Fine-tune k

+ Fine-tune q

+ Fine-tune v

Sparsified MLP 0

Sparsified MLP 1

Figure 7: The Perplexity increase after corrupting all FFN layers, before and after fine-tuning individual attention layers. The top row shows results for 4-bit quantization and the bottom row for sparsification with ratio 0.5.

4.4

Finetuning

In modern model optimization pipelines, compressed models are typically fine-tuned to recover acceptable performance (Liao et al. 2024). We investigate whether our Fisher matrix approximation can predict which layers are most effective to fine-tune. To this end, we first corrupt all MLP layers (upscale, gate, and downscale) of each language model with 4-bit quantization or sparsification with ratio 0.5, and then fine-tune individual attention layers, one at a time (Figure 7). Fine-tuning the V -projection recovers the most performance, whereas fine-tuning the K- or Q-projections yields little improvement. This mirrors the off-diagonal structure of our Hessian approximations, where the V -projection exhibits the strongest coupling with the FFN layers. We additionally repeat this experiment with low-rank adapters (LoRA) (Hu et al. 2022) in place of full fine-tuning.

Quantization

qwen2-0.5b + LoRA o

+ LoRA k

+ LoRA q

+ LoRA v

Quantized MLP 2

qwen2-0.5b

Sparsification

Model OPT-125M OPT-350M Qwen2-0.5B OLMo2-1B

Quantized MLP

+ LoRA o

+ LoRA k

+ LoRA q

+ LoRA v

Sparsified MLP

Sparsified MLP 1

Figure 8: The Perplexity increase after FFN corruption and the its values after LoRA application to various Attention layers. The top rows demonstrates the experiments with 4-bit quantization and the bottom one with sparsification.

LoRA adds a trainable low-rank correction to a frozen weight matrix: Wnew = W + AB T , where W ∈ Rn×m is frozen and A ∈ Rn×r , B ∈ Rm×r are trainable matrices of rank r. As shown in Figure 8, applying the adapter to the V projection is again substantially more effective than applying it to the Q- or K-projections, in agreement with our Kronecker-based Hessian approximation of the Transformer block (Figures 2a, 2c, 2b). Overall, the fine-tuning experiments show that adapting layers with high off-diagonal Hessian values with respect to the corrupted layers recovers substantially more performance than adapting weakly coupled layers. This further confirms that the off-diagonal values of our approximation reflect genuine inter-layer dependencies in large language models. Our method (Section 3) therefore offers a practical criterion for selecting which layers to fine-tune in order to compensate for compression errors.

4.5

Total time (s) 157.64 1563.07 1090.61 2428.25

Gradient time (s) 50.73 272.94 200.09 218.64

olmo2-1b

+ LoRA o

Table 2: Full-model Hessian approximation construction time over all Transformer blocks, using a rank-1 approximation. Gradient computation time is included in the total.

olmo2-1b

+ LoRA o

Hessian Computation Time

To support our claim that the proposed approximation is tractable at scale, we report the wall-clock time required to construct it. We use a rank-1 approximation, already sufficient to reveal the Hessian structure, with 20 batches of 10 WikiText2 sequences on a single NVIDIA H100 GPU. As Table 2 shows, constructing the approximation for an entire model takes from a few minutes for OPT-125M to roughly 40 minutes for OLMo2-1B, so the full curvature structure of a billion-parameter model is obtainable within a single GPUhour. Gradient computation accounts for a decreasing share of the total as models grow, consistent with the complexity analysis of Section 3, where the Arnoldi iterations dominate the linear cost of backpropagation.

Conclusion

We introduced a scalable Kronecker-based Fisher approximation that enables practical Hessian analysis for billionparameter language models, reducing memory complexity from quadratic to linear in model size while preserving crosslayer interaction information. Our method bridges the gap between theoretical curvature studies and real-world model optimization, providing the first direct empirical evidence of non-diagonal Hessian structure in large LLMs. Across four model families, we consistently found that value projection layers exhibit the highest sensitivity and strongest cross-layer correlations, while other components show architecture-specific behavior. Extensive experiments on quantization, sparsification, inter-layer corruption, and fine-tuning demonstrate that our approximation strongly correlates with performance degradation and recovery, outperforming diagonal estimates and uncovering non-additive joint effects. These insights enable principled identification of fragile layers, guiding mixed-precision compression, targeted fine-tuning, and LoRA adaptation without the prohibitive cost of full Hessian computation. Our framework offers a practical, theoretically grounded tool for efficient model optimization, advancing both the understanding and compression of large-scale neural networks. Future Work A promising direction for future work is to test whether the observed inter-layer interactions are mediated by outlier channels in the residual stream. Outliermitigation techniques such as AWQ (Lin et al. 2024) and rotation-based methods such as QuaRot (Ashkboos et al. 2024) redistribute or remove these outliers. If the strong interactions we observe, such as the V–upscale pairs, arise because corruption in one layer damages the few highmagnitude channels the other depends on, then applying these techniques before compression should disproportionately reduce DI for exactly these pairs. The same experiment would clarify the O-projection discrepancy: a sharp drop in its vulnerability after outlier removal would indicate that this vulnerability stems from finite perturbations of outlier channels rather than from local curvature.

References Arai, Y.; and Ichikawa, Y. 2026. Quantization error propagation: Revisiting layer-wise post-training quantization. Advances in Neural Information Processing Systems, 38: 151916–151951. Ashkboos, S.; Mohtashami, A.; Croci, M. L.; Li, B.; Cameron, P.; Jaggi, M.; Alistarh, D.; Hoefler, T.; and Hens-

man, J. 2024. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Botev, A.; Ritter, H.; and Barber, D. 2017. Practical GaussNewton optimisation for deep learning. In International Conference on Machine Learning (ICML), 557–565. PMLR. Chekalina, V.; Moskovskiy, D.; Cherniuk, D.; Kurkin, M.; Kuznetsov, A.; and Frolov, E. 2025. Generalized Fisherweighted SVD: Scalable Kronecker-factored Fisher approximation for compressing large language models. arXiv preprint arXiv:2505.17974. Collobert, R. 2004. Large scale machine learning. Technical Report RR-04-42, IDIAP. Dangel, F.; Harmeling, S.; and Hennig, P. 2020. Modular block-diagonal curvature approximations for feedforward architectures. In International Conference on Artificial Intelligence and Statistics (AISTATS), 799–808. PMLR. Das, R.; Agarwal, N.; Sanghavi, S.; and Dhillon, I. S. 2024. Towards quantifying the preconditioning effect of Adam. arXiv preprint arXiv:2402.07114. Dong, Z.; Zhang, Y.; Yao, J.; and Sun, R. 2025. Towards quantifying the Hessian structure of neural networks. arXiv preprint arXiv:2505.02809. Frantar, E.; and Alistarh, D. 2022. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35: 4475–4488. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR). ArXiv:2210.17323. Golub, G. H.; and Reinsch, C. 1971. Singular value decomposition and least squares solutions. In Linear algebra, 134–151. Springer. Hsu, Y.-C.; Hua, T.; Chang, S.; Lou, Q.; Shen, Y.; and Jin, H. 2022. Language model compression with weighted lowrank factorization. In International Conference on Learning Representations. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of the 10th International Conference on Learning Representations (ICLR). Izmailov, A. F.; and Solodov, M. V. 2014. Newton-Type Methods for Optimization and Variational Problems. Springer. Kingma, D. P.; and Ba, J. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). ArXiv:1412.6980. LeCun, Y.; and Cortes, C. 1998. The MNIST database of handwritten digits. Available: http://yann. lecun. com/exdb/mnist/, 24. Lehoucq, R. B.; Sorensen, D. C.; and Yang, C. 1998. ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods. SIAM. Liao, B.; Herold, C.; Khadivi, S.; and Monz, C. 2024. ApiQ: Finetuning of 2-Bit Quantized Large Language Model. In AlOnaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings

of the 2024 Conference on Empirical Methods in Natural Language Processing, 20996–21020. Miami, Florida, USA: Association for Computational Linguistics. Lin, J.; Tang, J.; Tang, H.; Yang, S.; Chen, W.-M.; Wang, W.C.; Xiao, G.; Dang, X.; Gan, C.; and Han, S. 2024. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. In Gibbons, P.; Pekhimenko, G.; and Sa, C. D., eds., Proceedings of Machine Learning and Systems, volume 6, 87–100. Liu, H.; Li, Z.; Hall, D.; Liang, P.; and Ma, T. 2024. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations (ICLR). ArXiv:2305.14342. Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Martens, J. 2010. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning (ICML), 735–742. Martens, J.; and Grosse, R. 2015. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), 2408– 2417. PMLR. Matveeva, T.; Katrutsa, A.; and Frolov, E. 2025. Dynamic low-rank approximation of full-matrix preconditioner for training generalized linear models. arXiv preprint arXiv:2508.21106. Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Naumov, M. 2017. Feedforward and recurrent neural networks backward propagation and Hessian in matrix form. arXiv preprint arXiv:1709.06080. Team OLMo; Walsh, P.; Soldaini, L.; Groeneveld, D.; Lo, K.; Arora, S.; Bhagia, A.; Gu, Y.; Huang, S.; Jordan, M.; et al. 2024. 2 OLMo 2 Furious. arXiv:2501.00656. Van Loan, C. F. 2000. The ubiquitous Kronecker product. Journal of Computational and Applied Mathematics, 123(12): 85–100. Wu, Y.; Guo, W.; Liu, Z.; Ji, H.; Xu, Z.; and Zhang, D. 2025. How large language models encode theory-of-mind: A study on sparse parameter patterns. npj Artificial Intelligence, 1(1): 20. Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. Yao, Z.; Gholami, A.; Shen, S.; Mustafa, M.; Keutzer, K.; and Mahoney, M. 2021. AdaHessian: An adaptive second order optimizer for machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 10665–10673. Zhang, H.; Xiong, C.; Bradbury, J.; and Socher, R. 2017. Block-diagonal Hessian-free optimization for training neural networks. arXiv preprint arXiv:1712.07296. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al.

2022. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. Zhang, Y.; Chen, C.; Ding, T.; Li, Z.; Sun, R.; and Luo, Z.-Q. 2024. Why transformers need Adam: A Hessian perspective. Advances in Neural Information Processing Systems, 37: 131786–131823. Zhang, Y.; Chen, C.; Li, Z.; Ding, T.; Wu, C.; Kingma, D. P.; Ye, Y.; Luo, Z.-Q.; and Sun, R. 2025. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations (ICLR). ArXiv:2406.16793. Zhao, J.; Derakhshan, A.; Hyman, J.; Dong, J.; Jyothi, S. A.; and Harris, I. 2026. CoopQ: Cooperative game inspired layerwise mixed precision quantization for LLMs. In Findings of the Association for Computational Linguistics: ACL 2026, 7566–7578. Zhao, Y.; Dang, S.; Ye, H.; Dai, G.; Qian, Y.; and Tsang, I. 2025. Second-order fine-tuning without pain for LLMs: A Hessian informed zeroth-order optimizer. In International Conference on Learning Representations (ICLR).

Supplementary Materials A

Additional Hessian Visualizations

Figure 9 shows how the Hessian structure changes with depth. The earlier and middle groups contain more pronounced high-value regions and inter-layer correlations, whereas these values are substantially weaker in the final group of Transformer blocks. Figure 10 provides a global view of the Hessian across all OPT-125M Transformer blocks. Because OPT-125M is small enough for the complete approximation to remain legible, the figure exposes both the repeated block structure along the diagonal and the correlations between layers in different blocks. For larger models, a full-model plot at the same scale would make the layer boundaries indistinguishable, which is why the main paper presents block-level and multi-block views instead.

Pairwise Quantization with Concrete Values

Figure 11 contains the same pairwise-quantization experiment as Figure 4, but places the measured values directly into the heatmaps. The annotations make it possible to compare particular layer pairs quantitatively, while the color scale makes the overall interaction pattern easier to see. As in the main-paper figure, pairs involving the V -projection and pairs combining the upscale and downscale layers produce some of the largest perplexity increases.

C C.1

Additional Experimental Results

Zero-Shot Results on Downstream Tasks

We additionally evaluate whether the layer sensitivities observed through WikiText-2 perplexity transfer to downstream tasks. For each model, we apply 90% sparsification to one layer type at a time in the middle Transformer blocks, following the setup in the main paper, and evaluate the resulting model without task-specific fine-tuning on PIQA, WinoGrande, HellaSwag, ARC-Easy, and ARC-Challenge. Table 3 reports the accuracy decrease relative to the dense baseline in percentage points; thus, larger positive values indicate worse performance and greater sensitivity. The baseline rows report absolute accuracy, also in percent, and the Avg. column is the unweighted mean over the five tasks. The downstream results recover the principal sensitivity patterns found with perplexity. FC1 is the most damaging projection to sparsify in OPT-350M; the V -projection is the most damaging in Qwen2-0.5B; the upscale and downscale projections dominate in OLMo2-1B; and the gate projection is by far the most sensitive component in Qwen2.5-7B. The precise magnitude varies by task—in particular, HellaSwag and ARC-Easy often show the largest drops—but the most sensitive layer types remain consistent across evaluation criteria. Negative values for a few OPT-350M task– projection pairs denote small measured improvements and do not change the aggregate ranking.

Blocks 8-9-10

Blocks 20-21-22

1e 7 1.50 1.25 1.00 0.75 0.50 0.25 0.00

|Hessian|

L20 Attn L20 MLP L21 Attn L21 MLP L22 Attn L22 MLP

L8 At L8 tn M L9 LP At L9 tn L1 MLP 0A L1 ttn 0M LP

L8 Attn L8 MLP L9 Attn L9 MLP L10 Attn L10 MLP

L2 0A L2 ttn 0 L2 MLP 1A L2 ttn 1 L2 MLP 2A L2 ttn 2M LP

Blocks 4-5-6

L4 At L4 tn M L5 LP At L5 tn M L6 LP At L6 tn ML P

L4 Attn L4 MLP L5 Attn L5 MLP L6 Attn L6 MLP

Figure 9: Our Kronecker–Fisher approximations for groups of Transformer blocks 4–5–6 (left), 8–9–10 (middle), and 20–21–22 (right) of OPT-350M. Yellow denotes high Hessian values and blue denotes low values. While the Hessian values are high in the earlier Transformer blocks, they become significantly smaller in the later blocks. Table 3: Zero-shot accuracy under layer-type-wise 90% sparsification. The baseline rows give dense-model accuracy (%), while all projection rows give the accuracy decrease from that baseline in percentage points. Larger values indicate worse performance. Bold values mark the largest average decrease for each model. Model

Setting

PIQA

WinoGrande

HellaSwag

ARC-E

ARC-C

Avg.

OPT-350M

Baseline Q K V O FC1 FC2

64.25 4.68 2.34 0.60 0.05 10.72 3.59

52.33 2.13 -1.03 1.34 0.95 1.82 -0.08

36.80 3.33 3.51 2.88 2.68 10.38 3.25

40.11 8.21 6.86 1.14 1.73 10.52 4.17

23.72 -1.45 -0.09 0.09 0.17 0.51 0.43

43.44 3.38 2.32 1.21 1.12 6.79 2.27

Qwen2-0.5B

Baseline Q K V O Gate Up Down

69.37 3.05 4.08 11.15 3.21 2.77 7.24 6.31

57.54 4.66 4.58 7.81 4.42 4.89 5.13 3.63

49.10 7.68 9.26 17.90 7.28 7.72 14.00 9.60

50.42 4.17 7.03 14.44 4.29 2.78 9.97 7.95

28.75 4.01 3.24 7.25 3.67 1.54 5.46 4.52

51.04 4.71 5.64 11.71 4.57 3.94 8.36 6.40

OLMo2-1B

Baseline Q K V O Gate Up Down

74.81 2.67 2.12 2.01 3.10 2.77 8.76 7.73

63.85 4.81 3.95 2.29 4.50 5.45 10.89 4.97

66.61 10.11 4.85 8.32 13.14 10.11 22.11 19.20

72.60 11.83 9.47 8.71 15.28 13.17 26.89 25.17

40.96 7.34 6.57 5.38 9.30 6.14 12.37 13.05

63.77 7.35 5.39 5.34 9.06 7.53 16.20 14.02

Qwen2.5-7B

Baseline Q K V O Gate Up Down

79.38 1.80 1.09 1.09 1.25 25.35 5.66 3.92

70.56 4.89 4.81 1.82 4.66 21.70 7.81 5.13

78.21 3.87 6.84 5.93 7.71 49.16 15.26 10.71

75.93 1.26 7.37 3.32 5.85 47.01 7.03 5.43

49.66 1.79 3.16 3.75 4.01 27.39 8.11 7.42

70.75 2.72 4.65 3.18 4.70 34.12 8.77 6.52

−

1e 7 L0.k_proj L0.v_proj L0.q_proj L0.out_proj

1.0

L0.fc1 L0.fc2 L1.k_proj L1.v_proj L1.q_proj L1.out_proj L1.fc1 L1.fc2 L2.k_proj L2.v_proj L2.q_proj L2.out_proj

0.8

L2.fc1 L2.fc2 L3.k_proj L3.v_proj L3.q_proj L3.out_proj L3.fc1 L3.fc2 L4.k_proj L4.v_proj L4.q_proj L4.out_proj L4.fc1

0.6

L4.fc2 L5.k_proj L5.v_proj L5.q_proj L5.out_proj L5.fc1 L5.fc2 L6.k_proj L6.v_proj L6.q_proj L6.out_proj L6.fc1 L6.fc2 L7.k_proj L7.v_proj L7.q_proj L7.out_proj

0.4

L7.fc1 L7.fc2 L8.k_proj L8.v_proj L8.q_proj L8.out_proj L8.fc1 L8.fc2 L9.k_proj L9.v_proj L9.q_proj L9.out_proj

0.2

L9.fc1 L9.fc2 L10.k_proj L10.v_proj L10.q_proj L10.out_proj L10.fc1 L10.fc2 L11.k_proj L11.v_proj L11.q_proj L11.out_proj L11.fc1

L11.fc2

L11.fc1

L10.fc1

L10.fc2 L11.k_proj L11.v_proj L11.q_proj L11.out_proj

L9.fc1

L9.fc2 L10.k_proj L10.v_proj L10.q_proj L10.out_proj

L8.fc1

L8.fc2 L9.k_proj L9.v_proj L9.q_proj L9.out_proj

L7.fc2 L8.k_proj L8.v_proj L8.q_proj L8.out_proj

L7.fc1

L6.fc1

L6.fc2 L7.k_proj L7.v_proj L7.q_proj L7.out_proj

L5.fc1

L5.fc2 L6.k_proj L6.v_proj L6.q_proj L6.out_proj

L4.fc1

L4.fc2 L5.k_proj L5.v_proj L5.q_proj L5.out_proj

L3.fc2 L4.k_proj L4.v_proj L4.q_proj L4.out_proj

L3.fc1

L2.fc1

L2.fc2 L3.k_proj L3.v_proj L3.q_proj L3.out_proj

L1.fc1

L1.fc2 L2.k_proj L2.v_proj L2.q_proj L2.out_proj

L0.fc1

L0.fc2 L1.k_proj L1.v_proj L1.q_proj L1.out_proj

L0.k_proj L0.v_proj L0.q_proj L0.out_proj

L11.fc2

0.0

Figure 10: Full-model Kronecker–Fisher Hessian approximation for OPT-125M. This is one of the few language-model Hessians that can be presented in full while keeping the individual Transformer layers distinguishable.

OPT-350M

Qwen2-0.5B Q 0.253 0.867 1.209 0.878 1.154 1.861 1.142

K 0.945 0.510 2.045 0.828 2.992 1.667

2.0

FC1 3.051 2.992 4.743 3.028 2.373 4.423

O 0.878 1.221 1.595 0.597 1.560 2.216 1.664 Gate 1.154 1.618 1.873 1.560 0.856 2.801 1.990 Up 1.861 2.372 2.678 2.216 2.801 1.492 2.826

FC2 1.616 1.667 2.684 1.554 4.423 1.111

Up 1.107 1.069 1.727 1.483 2.018 0.880 2.906

1.0 0.5

Down 1.844 1.790 2.535 2.055 2.354 2.906 1.620

p ow n D

Q 0.050 0.119 0.209 0.215 0.343 0.386 0.442

2.0

K 0.119 0.068 0.236 0.236 0.355 0.420 0.463 V 0.209 0.236 0.160 0.346 0.426 0.520 0.542 ΔPPL

1.5

e at

Qwen2.5-7B

2.5

O 0.493 0.481 1.006 0.337 0.955 1.483 2.055 Gate 0.630 0.662 1.201 0.955 0.493 2.018 2.354

V 0.806 0.719 0.609 1.006 1.201 1.727 2.535

O 0.215 0.236 0.346 0.172 0.479 0.488 0.583 Gate 0.343 0.355 0.426 0.479 0.297 0.688 0.740 Up 0.386 0.420 0.520 0.488 0.688 0.343 0.724 Down 0.442 0.463 0.542 0.583 0.740 0.724 0.367

0.7 0.6 0.5 0.4 0.3 0.2 0.1

n ow

at G

0.0

n ow

p U

e at G

0.0

0.5 0.0

1 FC

K 0.263 0.134 0.719 0.481 0.662 1.069 1.790

1.0

Down 1.142 1.572 1.873 1.664 1.990 2.826 0.911 0

Q 0.111 0.263 0.806 0.493 0.630 1.107 1.844

1.5

ΔPPL

O 0.833 0.828 1.888 0.346 3.028 1.554

V 1.209 1.503 0.834 1.595 1.873 2.678 1.873 ΔPPL

V 1.983 2.045 1.283 1.888 4.743 2.684

OLMo2-1B

2.5

K 0.867 0.543 1.503 1.221 1.618 2.372 1.572

ΔPPL

Q 0.468 0.945 1.983 0.833 3.051 1.616

Figure 11: Pairwise 4-bit quantization results with the concrete perplexity increases shown in every cell. This is the annotated version of Figure 4 in the main paper. Each cell at row i and column j reports the perplexity increase after jointly quantizing layer types i and j in the middle Transformer blocks.

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

Analyze & explore related research in Conceptio

Related documents