arXiv:2606.07116v1 [cs.LG] 5 Jun 2026
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
Haoqi Wang1 , Lorenz K. Mueller2 , Jiawei Zhuang2 , Mathieu Salzmann13 , Lukas Cavigelli2∗ 1 School of Computer and Communication Sciences, EPFL, Switzerland 2 Huawei, Switzerland 3 Swiss Data Science Center, ETHZ & EPFL, Switzerland {haoqi.wang,mathieu.salzmann}@epfl.ch {lorenz.mueller,zhuangjiawei,lukas.cavigelli}@huawei.com
Abstract Low-bit quantization has been widely adopted to accelerate the inference of large language models (LLMs) by significantly reducing computational cost and memory usage. However, activation outliers pose a major challenge to effective quantization, often leading to notable performance degradation. In this paper, we introduce OffQ, a method designed to mitigate activation outliers in low-bit quantization through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a proposed top-1 PCA, and then concentrates high-magnitude activations into 1 channel via rotation. OffQ then absorbs this concentrated outlier channel by converting its magnitude into a shared offset, thereby reducing the standard deviation of the activations. This offsetting strategy enables effective W4A4KV4 quantization of LLMs using deployment-friendly uniform-grid and uniform-precision quantization. Extensive experiments across diverse LLM architectures and benchmarks demonstrate that OffQ outperforms state-of-the-art baselines, consistently improving model accuracy while preserving low-bit efficiency.
1
Introduction
Quantization [7, 21, 34, 46, 58] is a key technique for accelerating inference speed and reducing the memory footprint of large language models (LLMs). Its importance is further amplified by the increasing demand to deploy LLMs in resource-constrained environments, such as edge devices and cost-sensitive cloud platforms. By representing weights and activations with low-bit (e.g., 4-bit) integers rather than 16-bit floating-points, quantization significantly reduces model storage and improves computational efficiency. As LLM inference is often memory-bound, reducing memory traffic directly translates into higher throughput, lower latency, and lower serving cost [41, 89]. Despite this promise, robust 4-bit quantization of W4A4KV4 (Weight, Activation, and KV-cache all quantized in 4-bit) for LLMs remains challenging. A key difficulty stems from the activation outliers: sparse but extremely large values in a few channels that dominate the dynamic range [10, 18]. These outliers interact poorly with quantization because they force a coarse quantization step size that is suboptimal for the majority of the activations, leading to large rounding errors. A breakthrough in outlier mitigation is the use of rotation-based techniques, such as QuaRot [7] and SpinQuant [46], which apply orthonormal transformations to redistribute the activation energy to Gaussian-like distributions, significantly improving quantization quality. Yet, even equipped with the state-ofthe-art rotation-based methods, the resulting activation distribution still retain large variance and is suboptimal for quantization [60], rendering W4A4KV4 quantization unreliable. ∗ corresponding author
Preprint.
Several strategies have been explored to further improve quantization on top of rotation-based methods. The first approach either improves the rotation learning or uses more expressive equivalent transformations (e.g., OSTQuant [27], AffineQuant [47], DuQuant [39], FlatQuant [64], DartQuant [60], KurTail [2]) to transform activations into a quantization-friendly form, at the cost of additional parameters, computational overhead, and carefully tuned custom kernels. The second approach adopts mixed-precision schemes (e.g., SliM-LLM [29], Atom [89], ResQ [58]), preserving higher precision for outlier-heavy channels while aggressively quantizing the rest. However, it introduces heterogeneous execution, requiring extra kernels, casting, and memory movement, and often reducing the efficiency of streamlined low-bit computation. The third approach uses non-uniform quantization (e.g., RCP [14], NestQuant [57]), which adapts quantization levels to the weight or activation distribution, but requires more complex kernels and lacks broad hardware support. By contrast, here, we exploit recent progress in singular defects analysis [72], which unveils that the outliers in activation exhibit a pronounced low-dimensional structure: the high-norm tokens in an LLM share the same direction across layers, token ids, and locations in a sequence. Based on this observation, we introduce OffQ, a simple, efficient, and effective approach to mitigating activation outliers for 4-bit quantization without resorting to mixed-precision components, nonuniform quantization levels, or compute and memory consuming backpropagation-based rotation learning. The core idea is to eliminate the effect of outliers from the bulk of the activation distribution so that standard low-bit quantization can most effectively utilize its representational capacity. Specifically, OffQ first identifies the low-dimensional outlier subspace via a tailored top-1 PCA procedure and rotate the activations so that outliers concentrate on the least number of channels. It then reorders the channels into groups so that each group is assigned with one outlier channel. Finally, OffQ applies a specialized Hadamard rotation to convert the outlier energy into group-wise offsets, which will then be absorbed into the zero-point of standard low-bit quantization. In essence, the offset within each group eliminates the corresponding outlier channel, yielding quantities with substantially reduced standard deviation that can be effectively quantized. By suppressing outliers via offsets, OffQ enables accurate W4A4KV4 quantization while keeping all matrix multiplications in uniform 4-bit execution. We evaluate OffQ on a range of LLM architectures and benchmarks, focusing on the challenging W4A4KV4 quantization setting. Across these experiments, OffQ consistently improves the perplexity and accuracy compared to a range of state-of-the-art baselines, while preserving the simplicity and efficiency of end-to-end low-bit inference. Overall, OffQ offers a practical mechanism for making 4-bit quantization more robust, strengthening the feasibility of low-cost LLM deployments. Contributions 1. We identify and address the structured activation outliers that limit practical low-bit quantization. 2. We propose OffQ, a post-training quantization method that (i) rotates activations to concentrate outliers into a few channels, and (ii) absorbs them as group-wise offsets using Hadamard rotation, enabling W4A4KV4 quantization with uniform-precision matrix multiplication. 3. Extensive experiments across multiple LLM architectures and benchmarks demonstrate that OffQ significantly improves 4-bit quantization accuracy while maintaining low-bit efficiency.
2
Related Work
Quantization-Aware Training and Post-Training Quantization. Quantization in LLMs can be broadly categorized into Quantization-Aware Training (QAT) [8, 13, 20, 30, 31, 44, 52, 70, 78] and Post-Training Quantization (PTQ) [3, 18, 28, 32, 38, 43, 69, 73, 79]. QAT simulates quantization effects during training, allowing the model to adapt to low-precision representations. However, the retraining involved can be computationally and memory-wise prohibitive. In contrast, PTQ applies quantization to pre-trained models without further training, reducing its computational and memory requirements. In this work, we focus on PTQ due to its efficiency and practicality for LLMs. Weight-Only, Weight-Activation, and KV-Cache Quantization. In the context of PTQ for LLMs, weight-only quantization [1, 4, 12, 19, 21, 23, 26, 29, 33, 36, 40, 51, 67, 68, 85, 87, 88] focuses on quantizing the model weights while keeping activations in higher precision. It reduces memory traffic 2
during decoding phase of the inference, but requires the costly full-precision matrix multiplication and cannot fully leverage the emerging efficient low-bit hardware. By contrast, weight-activation quantization, which quantizes both weights and activations, not only reduces memory but also speeds up computation in compute-bound scenarios such as the prefill phase and batched inference [6, 35], as arithmetics can be performed in low-bit using specialized hardware such as Tensor Core [7, 40, 65, 89]. KV-cache quantization [11, 24, 25, 45, 61, 62, 80, 82] reduces the memory bandwidth overhead of autoregressive decoding by caching key/value in low-bit. It can be applied in conjunction with weightonly and weight-activation quantization. In this work, we focus on the challenging but rewarding W4A4KV4 setting: applying weight-activation quantization together with KV-cache quantization. Equivalent Transforms in Weight-Activation Quantization. A central challenge in weightactivation quantization is the presence of outliers in activations. While techniques such as mixedprecision [6, 18, 29, 89] are sometimes used, existing solutions mainly rely on the idea of equivalent transform: applying a linear transformation to the activations before quantization and applying the inverse transformation after dequantization, which does not change the model’s representational capacity but can redistribute the activation energy to be less spiky and more quantization-friendly. Prior work mainly uses the following types of equivalent transforms: (i) Permutation. RPTQ [81] permutes channels so that similar value ranges are clustered into groups and apply fine-grained quantization. (ii) Scaling and Shifting, either learnable or heuristic-based, are used in SmoothQuant [77], Outlier Suppressions [73, 74], OmniQuant [59], and ABQ-LLM [84]. (iii) Rotation. Rotation as an equivalent transform was initially used in SliceGPT [5] for model trimming, and later widely adopted in the quantization literature due to its effectiveness in outlier suppression. QuaRot [7] uses random Hadamard rotation; SpinQuant [46] trains rotation by Cayley SGD; DartQuant [60] optimizes the rotation using Whip loss; DFRot [76] learns rotation with a weighted loss; KurTail [2] learns rotation by a Kurtosis loss; DuQuant [39] combines two rotations with permutation; ResQ [58] uses rotation and computes 1/8 of the channels in higher precision; Both NestQuant [57] and RCP [14] use rotation combined with non-uniform grid. (iv) Affine Transform. AffineQuant [47] and FlatQuant [64] learn affine transforms using an MSE loss; QLLM [42] reallocates outlier magnitudes to other channels; OSTQuant [27] and QServe [41] combine rotation and scaling. Many of the aforementioned equivalent transform-based methods also combine mixed-precision [42, 58], and non-uniform grid [14, 57] to further enhance the quantization performance. In this work, we introduce a new equivalent transform: (v) Outlier2offset, that converts outliers into offsets. It is complementary to the existing techniques (backpropagation-based rotation optimization, mixed-precision, non-uniform quantization grid, etc.) and can be seamlessly combined with them for further improvement. Nevertheless, we demonstrate that our OffQ can achieve strong performance with the standard uniform-grid and uniform-precision setting, and leave the exploration of advanced combinations for future work.
3
Method
Quantization maps high-precision floating-point values to lower-bit discrete representations, reducing memory footprint and accelerating inference. We adopt a uniform quantizer [34], which transforms a floating-point value x into a b-bit integer xq via: j x m xq = clamp + z, 0, 2b − 1 , (1) s where ⌊ · ⌉ denotes the rounding operator, s is the scale factor defining the quantization step size, and z is the zero-point, an integer offset that enables asymmetric quantization. Omitting z gives symmetric quantization. The granularity of quantization parameters can be adjusted. Per-tensor quantization uses a single pair (s, z) for an entire weight or activation tensor; while memory-efficient, it suffers from quantization error in the presence of outliers. Per-channel quantization assigns separate parameters to each output channel, better capturing channel-wise distributional variances. Per-group quantization further partitions each channel into groups with dedicated parameters, offering finer-grained representational fidelity at the cost of a marginal increase in metadata storage. 3
(a) Token X ∈ R1×4096
(b) Concentrated Xrot
(c) Permute to 4 groups (d) Group-wise offset Xhad
Figure 1: Visualization of the outlier concentration of the top-1 PCA and the subsequent offsetting technique. We plot the activation values of an outlier token in Llama 3-8B before and after each step, where the x-axis is the channel index and the y-axis is the activation value. The top-1 PCA effectively concentrates the outlier into the first channel, while the regular PCA fails to do so due to interference from normal tokens. The permutation and group-wise offsetting then redistributes the outlier value across all channels, creating constant offsets that can be absorbed during quantization.
3.1
Concentrate Structured Outliers via Top-1 PCA
The emergence of structured outliers in LLMs has been widely documented in the quantization literature [10, 18] and in studies of massive activations [63]. These outliers are characterized by their sparsity and extremely large magnitudes. A key insight from recent work on singular defects in LLMs [72] is that such activation outliers exhibit a pronounced low-dimensional structure. Specifically, the outlier directions across the transformer layer boundary are dominated by a 1dimensional subspace and the subspace is shared across different layers, token IDs, and locations in a sequence. This observation suggests that the intrinsic dimensionality of the outlier subspace is substantially smaller than the number of affected channels. Consequently, as illustrated in Figure 1, concentrating outliers into fewer channels can enable more targeted suppression techniques. Top-1 PCA. To extract the low-dimensional outlier subspace, we propose a top-1 PCA procedure. Given a small calibration dataset of N sequences each of length L, we collect input activations before each linear layer with hidden dimension D, forming an N × L × D tensor. To focus on outlier tokens and suppress interference from normal tokens, we select the top-1 token per sequence with the largest L∞ norm, yielding a matrix X ∈ RN ×D . Assuming zero-mean activations, we compute the covariance matrix C and perform eigendecomposition: C = X ⊤X /N = UΛU ⊤ , (2) where the columns of U are the eigenvectors sorted by their corresponding eigenvalues in Λ in descending order. The leading eigenvector (first column of U ) captures the direction of maximal outlier variance. Concentrate Outliers. To align the outlier directions with individual channels, we rotate the activations X with the eigenvector matrix U , Xrot = XU .
(3)
The first channel of Xrot corresponds to the projection onto the leading eigenvector, thereby concentrating the dominant outlier into a single channel. More generally, the leading G channels capture the dominant outlier directions of the original activations, while the remaining channels are largely outlier-free and thus amenable to low-bit quantization. Why the L∞ Norm. We assume a pre-normalization architecture with RMSNorm, consistent with most contemporary LLMs. Following QuaRot [7] and SpinQuant [46], we fuse the scaling parameters of RMSNorm into the adjacent weight matrices, so that the input activations to the linear layers of WQ , WK , WV in the self-attention module and WU , WG in the feedforward module, lie on a scaled L2 sphere. On a sphere, tokens with a few large-magnitude entries and many small entries exhibit a large L∞ norm. For example, on the 3-dimensional unit sphere, the spiky token [1, 0, 0] has a large L∞ norm of 1, while the flat token [0.58, 0.58, 0.58] has a smaller L∞ norm of 0.58. The L∞ norm therefore serves as a proxy for identifying the extreme outlier tokens in each sequence. 4
Why Top-1 Selection. While prior work has applied PCA to identify the outlier subspace for quantization [58], our top-1 PCA is more effective at concentrating outliers by focusing exclusively on the most extreme tokens, which are the primary drivers of quantization error. Because activations are L2 -normalized, the covariance matrix computed over all tokens is distracted by the large number of normal tokens, causing the leading eigenvector to drift from the true outlier direction. Figure 1b validates this intuition, showing that top-1 PCA yields better outlier concentration than standard PCA. Types of Structured Outliers. We identify two primary types of structured outliers in LLMs. The first type arises from singular defects [63, 72], appearing at the boundary between the selfattention and feedforward modules. The second type occurs in LLMs that incorporate bias terms in self-attention layers (e.g., Qwen 2.5 [54]), where these biases skew the activation distribution and introduce outliers into the KV-cache. Both types are effectively captured by the Top-1 PCA. 3.2
Absorbing Outliers by Offsetting
Having concentrated the prominent outlier into the first channel of the rotated activations Xrot , we next introduce our offsetting technique to absorb this outlier prior to quantization. Offsetting Technique. We construct a partially random Hadamard matrix H ∈ RD×D , an orthogonal matrix with ±1 entries, and whose first row is constrained to be all ones, i.e., H1,: = [1, 1, . . . , 1]. We then apply the Hadamard rotation H to the outlier-concentrated activations Xrot , √ Xhad = Xrot H/ D. (4) This Hadamard rotation redistributes the value of the first channel uniformly across all channels, √ creating a constant offset of 1/ D times the original outlier value into every channel. To illustrate this, consider a 2-dimensional token Xrot =[x1 , x2 ]∈ R1×2 with an outlier in the first channel x1 . 1 1 Applying the Hadamard rotation with H = gives: 1 −1 √ x1 + x2 x1 − x2 x1 x2 x2 √ Xhad = Xrot H/ 2 = (5) , √ = √ + √ , −√ . 2 2 2 2 2 |{z} | {z } offset
without outlier x1
√ The outlier x1 is spread across all channels as a constant offset x1/ 2, which can be absorbed into the 2
zero-point under asymmetric quantization . The remaining activations exhibit a flatter distribution and can be quantized effectively. Note that we use the Hadamard rotation for its simplicity and efficiency, but the offsetting technique is compatible with any orthogonal matrix whose first row contains only one unique value. Future work may involve extending optimization frameworks like DartQuant [7], DFRot [76], or KurTail [2] to learn constrained rotations that further flatten the non-outlier channels.
Group-Wise Offsetting. To absorb multiple outlier channels, we extend the offsetting technique to a group-wise setting. We partition the D channels of Xrot into G groups of size D/G, and reorder the channels so that the first channel of the g-th group corresponds to the g-th largest outlier channel. The offsetting technique is then applied independently within each group, so that the top-G outlier channels are each absorbed as a zero-point under per-group asymmetric quantization. The remaining non-outlier channels are assigned to groups in sorted order similar to [81], which we found empirically to outperform the zigzag assignment of [39]. 3.3
Quantizing an LLM with Activation Offsetting
Building upon our insights into the structured nature of outliers and the offsetting technique, we introduce OffQ, a novel quantization method that effectively mitigates outlier-induced quantization error. The overall pipeline, illustrated in Figure 2, enhances the architecture of the rotation-based method SpinQuant [46] by incorporating our activation offsetting technique. Step 1. Collect Activation Statistics. We begin by fusing the RMSNorm scaling parameters into the adjacent weight matrices. Activation statistics are then collected at the following positions: 2 Asymmetric quantization needs not maintain the value 0 into the quantization range [34], since 0 is shifted by the offset.
5
𝑊embed
𝐻1𝑇
𝑃1𝑇
𝑈1𝑇
𝐻1𝑇
𝑃1𝑇
𝑈1𝑇
𝑈1
𝑃1
𝑊𝑄
𝐻1
RoPE
𝐻1𝑇
𝑃1𝑇
Attention Block
𝑈1𝑇
𝑈1
𝑃1
𝐻1𝑇
𝐻1
𝐻3 ⊙ Softmax
𝐻1𝑇
𝑃1𝑇
𝑈1𝑇
𝑊𝐾
𝑊𝑉
Quantization
RoPE
𝐻3
⊙
𝑅2𝑇
𝑊𝑂 𝑈1
𝑃1
FFN Block
𝑃1𝑇
𝑈1𝑇
𝐻1𝑇
𝑃1𝑇
𝑈1𝑇
𝑊𝑈 𝑃4
𝐻1𝑇
𝑃1𝑇
𝑈1𝑇
𝑊𝐺 𝑃4
𝑈1
𝑃1
𝐻1𝑇
𝐻1
⊙ × 𝐻4
𝐻1
𝑃1𝑇
𝐻4𝑇
𝑈1𝑇
𝑃4𝑇
𝑊head
𝑊𝐷 𝑈1
𝑃1
𝐻1
Swish
𝑅2
Merged weights
𝐻 Online Hadamard
𝐻
Merged Hadamard
𝑈 Top-1 PCA rotation
𝑃
Permutation
𝑅
Random rotation
Figure 2: The overall quantization pipeline of OffQ. U1 is the shared rotation from top-1 PCA on attention and feedforward inputs, P1 and P4 are the permutation matrices for grouping, R2 is a per-head random rotation, H1 is the group-wise Hadamard rotation for offsetting, and H3 , H4 with yellow background are the online Hadamard rotations. Most rotation matrices, except for the three online Hadamard rotations, are fused into the weight matrices to reduce computational overhead. (1) Attention and feedforward inputs. Input activations to WQ , WK , WV , WU , WG are stacked together over all layers. The rotation U1 is computed via Top-1 PCA on these activations and shared among layers. (2) Key activations. If the LLM contains bias-induced outliers in self-attention, we additionally collect the output activations of WK after RoPE encoding. The corresponding rotation U3 is computed via per-layer, per-head Top-1 PCA. (3) Down-projection inputs. We collect the per-channel absolute maxima of input activations to WD separately per layer. Step 2. Apply Rotation, Grouping, and Offsetting. These operations can be represented by orthonormal matrices and are fused into the weight matrices when possible to reduce the overhead. (1) Attention and feedforward inputs. We right-multiply WO and WD by U1 , apply the permutation P1 to reorder channels into groups, and apply the group-wise Hadamard rotation H1 for offsetting. The inverse transformation H1⊤ P1⊤ U1⊤ is left-multiplied onto WQ , WK , WV , WU , WG to preserve the original output space. The input embeddings are also rotated by U1 P1 H1 , with the inverse applied to the left of the output head Whead . (2) Value cache. A per-head random rotation R2 is fused into the right side of WV and the left side of WO , enabling quantization of the value cache. (3) Key cache. If no bias-induced outliers are present, we apply the per-head online Hadamard rotation H3 to the RoPE-encoded outputs of WQ and WK . Otherwise, H3 is replaced by U3 P3 H3 , which prepends a per-layer, per-head top-1 PCA rotation U3 and a permutation P3 . (4) Down-projection inputs. Because rotation cannot penetrate the element-wise multiplication preceding WD , top-1 PCA is not applicable here. Instead, outlier channels are identified via per-channel absolute maxima. Channels are permuted into groups by P4 , and then offset by online group-wise Hadamard rotation H4 . The permutation P4 is fused into the right side of WU , WG , and the inverses H4⊤ P4⊤ are fused into the left side of WD . Step 3. Quantize Weights. Following the practice of rotation-based quantization methods such as QuaRot [7], SpinQuant [46], DartQuant [60], Atom [89], and KurTail [2], we apply GPTQ [21] to quantize the weights in 4 bits after merging of transforms into the weight matrices. More advanced weight quantizers such as GPTAQ [38], QEP [3], LeanQuant [87], Qronos [86], and ResComp [37] are compatible with our pipeline, but we use GPTQ for all methods to ensure a fair comparison. 3.4
Discussion
Generality on Data Formats. Micro-scaling formats MXFP4 and NVFP4 employ block-wise shared scale factors and are increasingly supported by recent hardware. Applying the offsetting technique to these formats requires extending them to support asymmetric quantization, which introduces a zero-point parameter per block. With a suitable block size, the overhead of zero-points remains negligible while still effectively absorbing outliers. We leave this extension to future work. Application to ViTs. Vision Transformers (ViTs) are also known to exhibit structured outliers [17, 71], making OffQ a natural candidate for extension. We leave this exploration to future work. 6
Llama
Table 1: Compare the perplexity (PPL ↓) on WikiText [48] and the average 0-shot accuracy (0-shot8 ↑) on ARC-e/ARC-c [16], BoolQ [15], HellaSwag [83], OpenBookQA [50], PIQA [9], SIQA [56], and WinoGrande [55] under W4A4KV4 quantization. Results for 16-bit, GPTQ [21], QUIK [6], QuaRot [7], SpinQuant [46], and ResQ [58] are referenced from [58]; and results for DFRot [76], KurTail [2], and OSTQuant [27] are referenced from their respective papers. Missing results are marked with “–”. OffQ achieves the best performance. Detailed results are provided in the Appendix. 3-8B 3-70B 3.2-1B 3.2-3B 2-7B 2-13B Method 8 8 8 8 8 PPL 0-shot PPL 0-shot PPL 0-shot PPL 0-shot PPL 0-shot PPL 0-shot8 16-bit 6.1 67.09 2.9 73.09 GPTQ 166.3 39.79 11655 34.90 QUIK 14.2 51.60 8.0 58.15 QuaRot 7.8 62.10 5.7 67.56 SpinQuant 7.4 63.76 6.2 65.68 DFRot 7.91 62.35 5.03 68.98 KurTail 7.2 64.63 4.2 70.69 OSTQuant 7.29 64.70 4.01 71.16 ResQ 7.1 63.91 4.1 71.14 OffQ 6.98 65.49 3.88 70.63
Qwen 2.5
Method 16-bit GPTQ QUIK QuaRot ResQ OffQ
1.5B
3B 8
9.8 54.86 7.8 62.73 5.5 64.15 4.9 66.45 108.9 37.98 178.3 40.34 9600 38.89 3120 35.83 21.8 44.30 15.8 48.74 7.5 56.99 6.8 60.21 14.3 49.01 10.1 56.06 6.1 60.75 5.4 63.80 13.6 48.78 9.2 57.89 6.0 60.98 5.2 64.81 – – – – 6.25 60.97 5.43 63.83 12.9 50.11 9.0 59.04 5.9 61.31 5.2 65.18 – – – – 5.91 62.11 5.25 64.19 12.4 50.11 8.8 58.99 5.8 61.95 5.1 65.25 12.32 50.91 8.78 60.80 5.77 61.99 5.11 65.25 7B
8
PPL 0-shot PPL 0-shot
PPL 0-shot
14B 8
32B 8
PPL 0-shot
PPL 0-shot
72B 8
PPL 0-shot8
9.3 60.85 8.0 63.81 6.8 68.45 5.3 70.61 5.0 70.44 3.9 73.41 25770 35.21 9978 35.10 13594 34.85 5100 36.93 3891 38.53 37967 34.54 6614 35.81 15.5 51.19 260.3 41.48 10.5 57.66 9.6 59.08 8.3 61.90 6600 38.33 68.8 47.76 4036 38.36 6.8 67.14 6.1 67.90 4.9 70.28 12.5 55.26 9.0 61.13 8.2 65.29 6.2 69.16 5.6 69.55 4.6 71.98 11.35 57.53 8.98 61.47 7.66 66.16 6.07 69.20 5.52 69.59 4.29 72.68
Efficiency of Per-Group Activation Quantization. Per-group quantization is widely used in weight quantization [21], and recent work including Atom [89] and QServe [41] have applied per-group quantization to activations with custom kernel implementations. We expect advances in hardware support and kernel design to further reduce the computational overhead of group quantization. In terms of memory, the zero-point storage overhead per group is negligible relative to the savings from 4-bit quantization. For example, with a group size of 128 and scale/zero-point stored in 16 bits each, the effective bit-width per activation is (4×128+16×2)/128 = 4.25 bits, which is still a substantial reduction from the original 16 bits.
4
Experiments
Models and Metrics. We evaluate OffQ on several widely adopted LLM families, including Llama 2 (7B/13B) [66], Llama 3 (8B/70B) [22], Llama 3.2 (1B/3B) [49], and Qwen 2.5 (1.5B/3B/7B/ 14B/32B/72B) [54]. Model quality is assessed from two complementary perspectives: language modeling capability, measured by perplexity (PPL) on WikiText [48], and common-sense reasoning, measured by zero-shot accuracy on ARC-e/ARC-c [16], BoolQ [15], HellaSwag [83], OpenBookQA [50], PIQA [9], SIQA [56], and WinoGrande [55]. Baselines. We compare OffQ primarily against recent state-of-the-art rotation-based post-training quantization methods and their improvements, including QuaRot [7], SpinQuant [46], DFRot [76], KurTail [2], OSTQuant [27], and ResQ [58]. Specifically, SpinQuant, DFRot, and KurTail use learning based method for optimizing the rotation; OSTQuant augments rotation with channel-wise scaling, yielding an affine transformation; and ResQ retains high-precision for 1/8 of the channels, resulting in an effective bit-width of 4.5. For reference, we additionally include two representative PTQ baselines, the weight-only method GPTQ [21] and the mixed-precision method QUIK [6]. 7
Table 2: Ablation studies on Llama 3-8B. The table shows the impact of each design choice on WikiText perplexity (PPL ↓) and the zero-shot accuracy (↑) across 8 common-sense reasoning tasks. PPL 0-shot Ablation Wiki ARC-c ARC-e BoolQ HellaS OBQA PIQA SIQA WinoG Avg. OffQ 6.98 Without Top-1 Selection 8.27 Zigzag Grouping 7.03 Partial Random Rotation 7.00
50.68 47.78 49.40 47.53
77.44 73.99 75.38 74.33
80.43 77.95 79.72 79.33
76.96 74.24 76.30 76.40
43.80 41.20 43.20 42.00
78.89 45.96 77.58 43.09 79.00 45.29 79.60 45.80
69.77 69.22 71.27 70.24
65.49 63.13 64.95 64.40
Implementation and Settings. We implement OffQ using the HuggingFace Transformers [75] and PyTorch [53], building upon the open-source codebases of ResQ. For activations, OffQ adopts per-group asymmetric quantization with a group size of 128, whereas baselines follow their official implementations using per-token asymmetric quantization. Weights are quantized with per-channel symmetric quantization across all methods. Following the evaluation protocol of [58], all methods except OSTQuant additionally apply GPTQ for weight quantization. We focus on the W4A4KV4 setting, where weights, activations, and KV-cache are all quantized to 4 bits: a configuration that is both practically relevant and effective for reducing memory footprint and bandwidth consumption. 4.1
Quantization Results
Table 1 summarizes results across ten LLMs of various sizes from both the Llama and Qwen families. We make the following observations. (1) Weight-only methods, exemplified by GPTQ, degrades severely under the W4A4KV4 setting: PPL increases from 6.1 to 166.3 on Llama 3-8B and from 6.8 to 13,594 on Qwen 2.5-7B. (2) The mixed-precision method QUIK substantially improves over GPTQ, yet still suffers from significant degradation. (3) Rotation-based methods achieve considerably better performance overall. However, QuaRot fails on Qwen models smaller than 14B, suggesting that naive rotation alone is insufficient to suppress quantization error in the presence of severe outliers in small models. (4) Among rotation-based methods evaluated on the Llama family, learned-rotation approaches such as SpinQuant and KurTail improve over QuaRot but still exhibit a noticeable gap relative to the 16-bit baseline. OSTQuant, which augments rotation with an additional scaling transformation, and ResQ, which enhances rotation with mixed-precision, close this gap further. (5) Our proposed OffQ achieves the best performance across nearly all models and benchmarks, with PPL closer to the full-precision baseline and zero-shot accuracy on par with or exceeding that of the best prior methods, without relying on mixed-precision computation or learned rotation. 4.2
Ablation Study
We conduct ablation studies to analyze the contribution of each component in OffQ. Starting from the full OffQ configuration, we modify each design choice individually. The results are summarized in Table 2, and we discuss the key findings from each ablation below. Top-1 Selection in Activation Statistics. When computing activation statistics for outlier subspace identification, we select the token with the largest L∞ norm within each sequence (top-1 selection) rather than using all tokens as in standard PCA [58]. Comparing row 1 (with top-1 selection) and row 2 (without top-1 selection) of Table 2, top-1 PCA substantially outperforms standard PCA with all tokens (PPL: 6.98 vs. 8.27; 0-shot8 : 65.49 vs. 63.13). Together with the qualitative illustration in Figure 1, this confirms that accurately identifying the outlier subspace, which requires focusing on the most extreme outlier tokens, is critical for effective outlier suppression in the offsetting technique. Sorted vs. Zigzag Grouping. We compare the two strategies for grouping non-outlier channels. Suppose we have 6 values [1, 2, 3, 4, 50, 60] with 2 outliers and 2 groups, sorted grouping yields [60, 1, 2 | 50, 3, 4], while zigzag grouping yields [60, 1, 3 | 50, 2, 4]. We vary the construction of permutation matrices P1 and P4 according to the two grouping strategies and find that zigzag grouping (row 3 of Table 2) performs slightly worse than the sorted grouping in OffQ (PPL: 7.03 vs. 6.98; 0-shot8 : 64.95 vs. 65.49), likely because sorted grouping clusters channels of similar variance, thereby reducing quantization error. 8
32
64
128
256
512
1024
2048
4096
Figure 3: Effect of group size on quantization performance of OffQ on Llama 3-8B with a hidden dimension of 4096. Increasing the number of groups (i.e., reducing group size) leads to better quantization performance (lower perplexity) as more outlier channels can be absorbed by offsetting.
Hadamard vs. Partially Random Rotation. We constructed the partially random Hadamard matrix with all-one in the first row to achieve the offsetting effect, but it can alternatively be replaced √ by a general partially random rotation matrix whose first row is constant at 1/ D. On Llama 3-8B, substituting H1 with such a partially random rotation matrix (row 4 of Table 2) yields comparable perplexity but lower zero-shot accuracy compared with OffQ (PPL: 7.00 vs. 6.98; 0-shot8 : 64.40 vs. 65.49), suggesting the Hadamard structure provides a marginal benefit. Number of Groups. Figure 3 shows the effect of group size on quantization performance. We vary the group size for attention, feedforward, and down-projection inputs, while fixing the group size for the KV-cache to match the head dimension. As expected, more groups allow more outlier channels to be absorbed, reducing quantization error and perplexity; however, finer grouping also increases the number of quantization parameters (scales and zero-points) that must be stored. We select a group size of 128 for activations (average bit-width of 4.25) as it strikes a good balance between quantization performance and storage overhead.
5
Limitations
OffQ achieves strong quantization performance under the W4A4KV4 setting, yet limitations remain. First, while we focus on the effectiveness of OffQ in mitigating outlier-induced quantization error, we have not reported inference latency in real-world deployment scenarios. Second, we leave the exploration of combining OffQ with complementary quantization techniques, such as non-uniform weight quantization [14, 57, 82], advanced weight quantization methods [3, 37, 38, 86, 87], and rotation learning [2, 60, 76], as future work.
6
Conclusion
In this paper, we have presented OffQ, a post-training quantization method that addresses outlierinduced quantization error under the W4A4KV4 setting. The core of OffQ is an offsetting technique that converts outlier activation channels into a uniform offset, which is absorbed into the zero-point of asymmetric quantization. To effectively identify the outlier subspace and concentrate the outlier directions into the least number of channels, we have proposed a Top-1 PCA method, which efficiently captures the most significant variance in the activation distribution. Extensive experiments on Llama and Qwen 2.5 series of LLMs have demonstrated that OffQ consistently outperforms state-of-theart rotation-based PTQ methods across diverse model sizes and benchmarks, achieving superior perplexity and 0-shot accuracy while maintaining efficient memory usage. These results highlight the effectiveness of OffQ in enabling high-performance LLM inference on resource-constrained hardware, paving the way for broader deployment of LLMs in real-world applications. 9
Acknowledgments and Disclosure of Funding This work was supported in part by the Swiss National Science Foundation via the grant 200020_214878.
References [1] Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, and Vikas Singh. Framequant: Flexible low-bit quantization for transformers. In Proceedings of International Conference on Machine Learning (ICML), 2024. [2] Mohammad Sadegh Akhondzadeh, Aleksandar Bojchevski, Evangelos Eleftheriou, and Martino Dazzi. KurTail : Kurtosis-based LLM quantization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17404–17419, Suzhou, China, 2025. Association for Computational Linguistics. [3] Yamato Arai and Yuma Ichikawa. Quantization error propagation: Revisiting layer-wise post-training quantization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [4] Yamato Arai and Yuma Ichikawa. Quantization error propagation: Revisiting layer-wise post-training quantization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. [5] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024. [6] Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. QUIK: Towards end-to-end 4-bit inference on generative large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3355–3371, Miami, Florida, USA, 2024. Association for Computational Linguistics. [7] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024. [8] Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L. Castro, Torsten Hoefler, and Dan Alistarh. HALO: Hadamard-assisted lower-precision optimization for LLMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [9] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020. [10] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947–7969, 2021. [11] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In Second Conference on Language Modeling, 2025. [12] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees, 2024. [13] Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. EfficientQAT: Efficient quantization-aware training for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10081–10100, Vienna, Austria, 2025. Association for Computational Linguistics. [14] Euntae Choi, Sumin Song, Woosang Lim, and Sungjoo Yoo. Rotate, clip, and partition: Towards W2A4KV4 quantization by integrating rotation and learnable non-uniform quantizer. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 7568–7590, Suzhou, China, 2025. Association for Computational Linguistics. [15] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
10
[16] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. [17] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. [18] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318–30332, 2022. [19] Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for nearlossless LLM weight compression. In The Twelfth International Conference on Learning Representations, 2024. [20] DaYou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 102–116, Bangkok, Thailand, 2024. Association for Computational Linguistics. [21] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. [22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [23] Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. LQ-loRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations, 2024. [24] Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient KV cache quantization with salient token identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [25] Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [26] Chun Hu, Junhui He, Shangyu Wu, Yuxin He, Chun Jason Xue, and Qingan Li. Mlwq: Efficient small language model deployment via multi-level weight quantization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8078–8088, 2025. [27] Xing Hu, Yuan Cheng, Dawei Yang, Zhixuan Chen, Zukang Xu, JiangyongYu, XUCHEN, Zhihang Yuan, Zhe jiang, and Sifan Zhou. OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In The Thirteenth International Conference on Learning Representations, 2025. [28] Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291, 2024. [29] Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. SliM-LLM: Salience-driven mixed-precision quantization for large language models. In Proceedings of the 42nd International Conference on Machine Learning, pages 25672–25692. PMLR, 2025. [30] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integerarithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018. [31] Minsoo Kim, Sihwa Lee, Suk-Jin Hong, Du-Seong Chang, and Jungwook Choi. Understanding and improving knowledge distillation for quantization aware training of large transformer encoders. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6713–6725, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
11
[32] Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. SqueezeLLM: Dense and sparse quantization, 2024. [33] Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. SqueezeLLM: Dense-and-sparse quantization. In Proceedings of the 41st International Conference on Machine Learning, pages 23901–23923. PMLR, 2024. [34] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018. [35] Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. “give me BF16 or give me death”? accuracy-performance trade-offs in LLM quantization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26872–26886, Vienna, Austria, 2025. Association for Computational Linguistics. [36] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, pages 18913–18939. PMLR, 2023. [37] Shuaiting Li, Juncan Deng, Kedong Xu, Rongtao Deng, Hong Gu, Minghan Jiang, Haibin Shen, and Kejie Huang. Rethinking residual errors in compensation-based LLM quantization. In The Fourteenth International Conference on Learning Representations, 2026. [38] Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. GPTAQ: Efficient finetuning-free quantization for asymmetric calibration. In Forty-second International Conference on Machine Learning, 2025. [39] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Advances in Neural Information Processing Systems, 37:87766–87800, 2024. [40] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. In MLSys, 2024. [41] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. Proceedings of Machine Learning and Systems, 7, 2025. [42] Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. QLLM: Accurate and efficient low-bitwidth quantization for large language models. In International Conference on Learning Representations (ICLR), 2024. [43] Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024. [44] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 467–484, Bangkok, Thailand, 2024. Association for Computational Linguistics. [45] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In International Conference on Machine Learning, pages 32332–32344. PMLR, 2024. [46] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quantization with learned rotations. In The Thirteenth International Conference on Learning Representations, 2025. [47] Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. [48] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
12
[49] Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https: //ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, 2024. Accessed: 2026-05-05. [50] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium, 2018. Association for Computational Linguistics. [51] Lorenz K. Muller, Philippe Bich, Jiawei Zhuang, Ahmet Celik, Luca Benfenati, and Lukas Cavigelli. Sinq: Sinkhorn-normalized quantization for calibration-free low-precision llm weights, 2025. [52] James O’Neill and Sourav Dutta. Self-distilled quantization: Achieving high compression rates in transformer-based language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1329–1339, Toronto, Canada, 2023. Association for Computational Linguistics. [53] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. [54] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. [55] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019. [56] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, 2019. [57] Semyon Savkin, Eitan Porat, Or Ordentlich, and Yury Polyanskiy. Nestquant: nested lattice quantization for matrix products and LLMs. In Forty-second International Conference on Machine Learning, 2025. [58] Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, and Xin Wang. Resq: Mixed-precision quantization of large language models with low-rank residuals. In Forty-second International Conference on Machine Learning, 2025. [59] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. [60] Yuantian Shao, Yuanteng Chen, Peisong Wang, Jianlin Yu, Jing Lin, Yiwu Yao, Zhihui Wei, and Jian Cheng. Dartquant: Efficient rotational distribution calibration for LLM quantization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. [61] Donghyun Son, Euntae Choi, and Sungjoo Yoo. NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [62] Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. Accurate kv cache quantization with outlier tokens tracing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12895–12915, 2025. [63] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024. [64] Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, JiaxinHu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. Flatquant: Flatness matters for LLM quantization. In Forty-second International Conference on Machine Learning, 2025. [65] Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Aditya Atluri, Fengqi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS, 2023.
13
[66] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [67] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP$\#$: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Forty-first International Conference on Machine Learning, 2024. [68] Albert Tseng, Qingyao Sun, David Hou, and Christopher De. Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37:59597–59620, 2024. [69] Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality in llm quantization. arXiv preprint arXiv:2402.15319, 2024. [70] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models, 2023. [71] Haoqi Wang, Tong Zhang, and Mathieu Salzmann. Sinder: Repairing the singular defects of dinov2. In European Conference on Computer Vision, pages 20–35. Springer, 2024. [72] Haoqi Wang, Tong Zhang, and Mathieu Salzmann. Demystifying singular defects in large language models. In Forty-second International Conference on Machine Learning, 2025. [73] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022. [74] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1648–1665, Singapore, 2023. Association for Computational Linguistics. [75] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, 2020. Association for Computational Linguistics. [76] Jingyang Xiang and Sai Qian Zhang. DFRot: Achieving outlier-free and massive activation-free for rotated LLMs with refined rotation. In Second Conference on Language Modeling, 2025. [77] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023. [78] Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In Advances in Neural Information Processing Systems, pages 66357–66382, 2024. [79] Yuzhuang Xu, Shiyu Ji, Qingfu Zhu, and Wanxiang Che. CRVQ: Channel-relaxed vector quantization for extreme compression of LLMs. Transactions of the Association for Computational Linguistics (TACL), 13: 1488–1506, 2025. [80] Haoqi Yang, Yao Yao, Zuchao Li, Baoyuan Qi, Liu Guoming, and Hai Zhao. XQuant: Achieving ultra-low bit KV cache quantization with cross-layer compression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9785–9800, Suzhou, China, 2025. Association for Computational Linguistics. [81] Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models, 2023. [82] Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. Turboquant: Online vector quantization with near-optimal distortion rate. In The Fourteenth International Conference on Learning Representations, 2026.
14
[83] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [84] Chao Zeng, Songwei Liu, Yusheng Xie, Hong Liu, Xiaojian Wang, Miao Wei, Shu Yang, Fangmin Chen, and Xing Mei. Abq-llm: Arbitrary-bit quantized inference acceleration for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22299–22307, 2025. [85] Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, and Penghang Yin. Magr: Weight magnitude reduction for enhancing post-training quantization. Advances in neural information processing systems, 37:85109–85130, 2024. [86] Shihao Zhang, Haoyu Zhang, Ian Colbert, and Rayan Saab. Qronos: Correcting the past by shaping the future... in post-training quantization. In The Fourteenth International Conference on Learning Representations, 2026. [87] Tianyi Zhang and Anshumali Shrivastava. Leanquant: Accurate and scalable large language model quantization with loss-error-aware grid. In The Thirteenth International Conference on Learning Representations, 2025. [88] Yijia Zhang, Sicheng Zhang, Shijie Cao, Dayou Du, Jianyu Wei, Ting Cao, and Ningyi Xu. Afpq: Asymmetric floating point quantization for llms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 28–36, 2024. [89] Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. In Proceedings of Machine Learning and Systems, pages 196–209, 2024.
15
Appendix We provide detailed quantization results for Llama and Qwen 2.5 models under W4A4KV4 quantization in Table 4 and Table 3, respectively.
Qwen 2.5-1.5B
16-bit 9.3 GPTQ 25770 QUIK 6614 QuaRot 6600 ResQ 12.5 OffQ 11.35
45.1 23.9 21.8 23.6 38.7 40.87
72.1 26.9 31.9 37.3 64.1 69.74
72.9 43.9 40.9 46.2 65.7 70.73
67.7 26.1 27.9 28.6 61.4 63.27
40.2 27.6 27.4 27.0 37.8 37.80
76.3 48.8 49.7 32.1 52.8 35.2 56.3 35.2 71.6 42.7 73.12 44.93
63.7 51.5 48.6 52.4 60.1 59.75
60.85 35.21 35.81 38.33 55.26 57.53
Qwen 2.5-3B
16-bit 8.0 GPTQ 9978 QUIK 15.5 QuaRot 68.8 ResQ 9.0 OffQ 8.98
47.4 26.0 36.1 32.4 45.3 47.10
73.0 26.7 55.4 53.1 70.5 73.90
77.5 41.5 61.4 51.6 72.7 70.76
73.6 26.7 57.2 49.2 70.2 69.94
42.0 28.2 36.2 33.4 42.4 40.60
78.7 49.9 51.5 31.9 67.1 40.8 66.7 39.3 76.8 46.7 77.15 48.31
68.4 48.3 55.3 56.4 64.4 64.01
63.81 35.10 51.19 47.76 61.13 61.47
Qwen 2.5-7B
16-bit 6.8 GPTQ 13594 QUIK 260.3 QuaRot 4036 ResQ 8.2 OffQ 7.66
51.2 25.2 29.5 25.9 49.0 49.23
77.6 25.6 42.4 41.0 74.7 74.96
84.7 37.8 51.7 39.1 81.4 83.30
78.9 26.3 36.3 29.1 75.7 76.49
47.2 28.2 28.2 27.6 45.0 44.80
80.0 54.8 52.4 34.4 59.6 34.5 57.9 35.7 78.9 49.4 78.56 52.71
73.2 48.9 49.6 50.6 68.2 69.22
68.45 34.85 41.48 38.36 65.29 66.16
Qwen 2.5-14B
16-bit 5.3 GPTQ 5100 QUIK 10.5 QuaRot 6.8 ResQ 6.2 OffQ 6.07
58.8 23.8 45.0 54.8 57.6 56.40
79.4 29.1 67.1 79.6 82.1 80.26
85.4 47.7 64.7 79.9 84.9 85.17
82.9 30.1 68.9 78.7 81.1 81.00
45.4 27.6 37.6 44.0 44.8 43.60
81.9 55.3 51.3 34.6 74.8 43.9 79.5 49.9 80.5 51.7 80.79 52.46
75.8 51.2 59.3 70.7 70.6 73.95
70.61 36.93 57.66 67.14 69.16 69.20
Qwen 2.5-32B
16-bit GPTQ QUIK QuaRot ResQ OffQ
5.0 3891 9.6 6.1 5.6 5.52
55.7 25.4 41.0 54.5 55.1 55.12
78.0 35.4 64.6 76.1 78.4 78.07
87.4 48.5 74.9 85.1 86.0 87.03
84.1 31.8 72.0 81.5 82.5 82.58
44.4 27.0 39.6 44.2 45.4 45.00
82.3 56.4 53.8 35.8 75.8 44.5 80.1 51.3 81.1 53.9 81.94 52.05
75.2 50.5 60.2 70.4 74.0 74.90
70.44 38.53 59.08 67.90 69.55 69.59
Qwen 2.5-72B
Table 3: Detailed quantization results for Qwen 2.5 models under W4A4KV4 quantization. We report the perplexity (PPL ↓) on WikiText [48] and the 0-shot accuracy across 8 tasks. Results for 16-bit, GPTQ [21], QUIK [6], QuaRot [7], and ResQ [58] are referenced from [58]. Method PPL ARC-c ARC-e BoolQ HellaS OBQA PIQA SIQA WinoG Avg.
16-bit 3.9 GPTQ 37967 QUIK 8.3 QuaRot 4.9 ResQ 4.6 OffQ 4.29
62.6 25.4 45.1 55.8 58.4 61.95
83.2 25.8 68.1 81.1 80.9 81.99
89.2 38.1 77.2 87.5 88.4 88.93
86.0 25.6 77.2 84.0 84.9 85.28
46.6 26.6 39.0 45.2 48.2 46.80
83.6 58.4 51.2 34.2 77.4 45.6 81.7 52.5 82.6 55.5 83.46 56.50
77.7 49.4 65.6 74.5 77.0 76.56
73.41 34.54 61.90 70.28 71.98 72.68
16
Method
PPL
ARC-c
ARC-e
BoolQ
HellaS
OBQA
PIQA
SIQA
WinoG
Avg.
Llama 3-8B
16-bit GPTQ QUIK QuaRot SpinQuant DFRot KurTail OSTQuant ResQ OffQ
6.1 166.3 14.2 7.8 7.4 7.91 7.2 7.29 7.1 6.98
53.2 24.7 33.6 45.1 48.0 44.97 48.2 49.32 49.2 50.68
77.1 37.7 56.4 70.4 75.4 71.09 75.4 76.73 75.0 77.44
81.1 44.3 60.5 73.8 75.8 73.27 79.2 78.87 72.5 80.43
79.2 36.8 61.5 74.7 75.4 74.13 76.4 76.01 76.5 76.96
44.8 27.0 33.2 42.6 43.8 43.00 43.6 43.20 43.0 43.80
80.9 57.6 68.7 76.6 77.5 78.24 78.4 78.51 78.3 78.89
47.0 36.4 39.9 45.1 45.0 44.58 45.8 45.70 45.8 45.96
73.4 53.8 59.0 68.5 69.2 69.53 70.0 69.22 71.0 69.77
67.09 39.79 51.60 62.10 63.76 62.35 64.63 64.70 63.91 65.49
Llama 3-70B
16-bit GPTQ QUIK QuaRot SpinQuant DFRot KurTail OSTQuant ResQ OffQ
2.9 11655.0 8.0 5.7 6.2 5.03 4.2 4.01 4.1 3.88
64.2 25.9 44.5 53.7 52.0 58.02 59.2 61.29 61.4 59.22
85.9 26.0 68.9 74.5 77.3 81.1 82.7 82.39 84.3 79.12
85.3 37.9 60.7 81.6 81.7 81.13 83.9 83.43 83.9 86.12
84.9 26.2 75.0 81.1 75.6 81.59 83.3 83.25 83.5 84.21
48.6 28.6 36.4 46.6 43.8 47.4 46.6 48.93 46.0 46.80
84.4 50.4 76.1 81.0 78.8 81.83 83.5 81.73 83.1 82.43
50.8 34.3 43.2 46.8 43.4 46.57 49.7 51.24 48.6 48.82
80.6 49.9 60.4 75.2 72.8 74.19 76.6 77.01 78.3 78.30
73.09 34.90 58.15 67.56 65.68 68.98 70.69 71.16 71.14 70.63
Llama 3-2-1B
16-bit GPTQ QUIK QuaRot SpinQuant KurTail ResQ OffQ
9.8 108.9 21.8 14.3 13.6 12.9 12.4 12.32
36.5 24.7 27.4 30.0 32.3 31.1 34.0 30.80
60.6 32.7 46.0 51.4 51.8 52.9 54.2 55.22
63.4 52.3 55.0 59.1 59.3 60.7 57.0 62.26
63.6 30.7 46.0 54.0 55.4 56.4 57.3 57.55
37.4 23.6 26.4 34.2 30.4 36.4 31.2 33.2
74.5 54.3 62.4 66.7 67.7 68.6 69.4 70.95
42.8 34.4 38.6 39.6 38.6 40.5 41.0 40.89
60.1 51.1 52.6 57.1 54.7 54.3 56.8 56.43
54.86 37.98 44.30 49.01 48.78 50.11 50.11 50.91
Llama 3-2-3B
16-bit GPTQ QUIK QuaRot SpinQuant KurTail ResQ OffQ
7.8 178.3 15.8 10.1 9.2 9.0 8.8 8.78
46.2 27.0 32.9 38.6 38.9 42.2 43.1 44.8
71.7 27.0 50.1 59.0 64.8 66.7 65.6 70.41
73.1 48.8 52.6 65.9 68.0 69.8 68.8 71.62
73.7 44.4 59.1 66.5 69.1 68.8 70.5 71.49
43.4 27.8 33.2 35.8 39.4 39.8 38.4 39.60
77.4 59.1 68.7 74.4 74.9 75.6 75.1 75.52
47.2 37.1 40.3 43.1 45.1 44.8 45.6 45.65
69.1 51.5 53.0 65.2 62.9 64.6 64.8 67.32
62.73 40.34 48.74 56.06 57.89 59.04 58.99 60.80
Llama 2-7B
16-bit GPTQ QUIK QuaRot SpinQuant DFRot KurTail OSTQuant ResQ OffQ
5.5 9600 7.5 6.1 6.0 6.25 5.9 5.91 5.8 5.77
46.3 24.8 39.8 41.5 43.6 43.52 43.1 42.92 44.0 43.77
74.6 31.4 63.7 71.4 71.3 70.83 72.0 72.56 72.6 70.79
77.8 55.4 68.9 73.2 73.8 73.3 72.0 74.71 75.3 74.89
75.9 30.6 68.3 73.2 73.2 72.62 73.2 73.14 74.0 73.84
44.2 25.6 37.8 40.6 40.4 41.40 41.2 44.40 41.0 41.80
79.2 55.8 72.9 76.9 76.0 76.82 76.6 77.42 77.9 78.02
46.1 34.2 42.1 43.6 44.1 44.17 45.6 44.98 43.9 44.27
69.1 53.3 62.4 65.6 65.4 65.11 66.8 66.77 66.9 68.51
64.15 38.89 56.99 60.75 60.98 60.97 61.31 62.11 61.95 61.99
Llama 2-13B
Table 4: Detailed quantization results for Llama models under W4A4KV4 quantization. We report the perplexity (PPL ↓) on WikiText [48] and the 0-shot accuracy across 8 tasks. Results for 16-bit, GPTQ [21], QUIK [6], QuaRot [7], SpinQuant [46], and ResQ [58] are referenced from [58]; and results for DFRot [76], KurTail [2], and OSTQuant [27] are referenced from their respective papers.
16-bit GPTQ QUIK QuaRot SpinQuant DFRot KurTail OSTQuant ResQ OffQ
4.9 3120 6.8 5.4 5.2 5.43 5.2 5.25 5.1 5.11
49.1 23.6 43.7 46.9 49.0 46.50 48.1 47.10 49.1 48.38
77.4 31.1 68.0 74.9 76.3 73.48 75.4 75.21 76.1 76.30
80.5 38.7 71.3 76.6 78.2 76.67 79.7 77.46 79.7 79.36
79.4 27.2 73.3 75.8 77.1 76.83 77.4 76.71 77.9 77.96
45.2 26.8 40.0 42.6 42.8 43.00 45.0 44.60 43.6 43.6
80.7 53.6 75.7 79.1 79.3 79.27 79.0 78.67 79.1 79.05
47.2 35.8 45.1 45.5 46.3 45.55 45.6 45.75 46.6 46.52
72.1 49.8 64.6 69.0 69.5 69.30 71.2 68.03 69.9 70.8
66.45 35.83 60.21 63.80 64.81 63.83 65.18 64.19 65.25 65.25
17