GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation
2
Emre Neftci 2,3
1 Fakultät für Informatik, RWTH Aachen, Aachen, 52074, Germany Fakultät für Elektrotechnik und Informationstechnik, RWTH Aachen, Aachen, 52062, Germany 3 Peter Grünberg Institut, Forschungszentrum Jülich GmbH, Jülich, 52425, Germany {j.oberlaender, j.finkbeiner, c.schoefmann, e.neftci}@fz-juelich.de
Abstract Autoregressive decoding with LLMs is primarily bottlenecked by GPU memory bandwidth, especially in edge-computing settings. While quantization is essential for mitigating this bottleneck, most existing methods treat inference as a uniform process and fail to account for the asymmetry between the compute-bound prefill stage and the memory-bound decoding stage. We propose GRINQH (GRaded INput-based Quantization Hierarchy), a weight-only post-training quantization framework that accelerates decoding by unifying quantization and sparsification. GRINQH leverages activation magnitudes as a proxy for computational importance to dynamically assign weight channels to different precision levels, enabling flexible average bit widths during decoding. Evaluated on Llama3 and Qwen3 models, GRINQH outperforms state-of-the-art fixed- and mixed-precision baselines at comparable 3- and 4-bit settings, even enabling effective 2-bit generation. We experimentally verify theoretical speedups by leveraging a hierarchical nested memory layout for multi-precision storage in a custom GPU kernel. Ultimately, GRINQH establishes a new state-of-the-art Pareto frontier for LLM generation, enabling a dynamic trade-off between generation quality and inference speed.
1
Introduction
∗ These authors contributed equally to this work.
Preprint.
Effective Number of Bits
6 The primary operational focus for Large Lan90 bf16 baseline guage Models (LLMs) has shifted from train85 5x lower degradation ing to efficient inference at scale. Despite 5 advances in hardware throughput, inference GRINQH 25% faster 80 4bit Max 8bit remains constrained by the memory wall: a 4 GRINQH fundamental performance bottleneck where Max 6bit 75 off-chip DRAM transfer speeds cannot keep GPTQ 20x lower 3 pace with on-chip computation. In LLMs, AWQ 70 degradation this is driven by dense linear layers, which, RTN 3bit for models exceeding 7B parameters, account 65 2 0.8 1.0 1.2 1.4 for approximately 95% of total parameters VMM runtime [Normalized to 4-bit Marlin kernel] and 65–85% of total floating-point operations (FLOPs) [11]. We propose GRINQH, Figure 1: Pareto frontier of GSM8K accuracy vs. a weight-only quantization framework that di- VMM runtime for Qwen3-8B on an RTX 4090. rectly targets these linear layers to accelerate GPTQ, AWQ, RTN assume MARLIN execution.
GSM8K Exact Match Accuracy [%]
arXiv:2606.23419v1 [cs.LG] 22 Jun 2026
Jette Oberländer 1,3∗ Jan Finkbeiner 2,3∗ Catherine M. Schöfmann 2,3
inference by reducing the volume of weight data transferred from memory. By enabling fine-grained, channel-wise (row-by-row) dynamic precision adjustment, GRINQH facilitates structured sparse memory loads that significantly reduce weight data movement. This approach is motivated by the fact that LLM inference operates in two fundamentally different stages: prefill and decoding. In the prefill stage, the input prompt is processed in parallel, allowing model weights to be loaded once and reused across the batch, rendering this stage largely compute-bound [32]. However, the bottleneck shifts to a memory-bound stage during autoregressive decoding, where all weight matrices must be loaded from DRAM to generate a single token. Since these matrices comprise the vast majority of the model’s footprint, loading them repeatedly for each token is highly inefficient. Especially for edge applications with batch size one, GPU compute units remain underutilized while waiting for data from DRAM, resulting in latencies significantly higher than in the prefill stage [32, 23, 25]. Various compression techniques have been proposed to mitigate this memory-bound bottleneck, including quantization [23, 14, 43], pruning [36], and sparsification [27, 35, 24]. Activation sparsity, in particular, dynamically bypasses weight loading based on runtime activations [25, 35]. However, these methods often require costly retraining or fail to sufficiently address the memory wall [27, 24]. In contrast, post-training quantization (PTQ) [23, 14, 43] is a highly effective strategy for accelerating inference while maintaining competitive model accuracy without significant overhead. Standard quantization methods, however, typically apply a uniform strategy across both stages. Low precision during the compute-bound prefill stage provides limited speedup, as performance is bottlenecked by arithmetic throughput rather than memory transfer, yet it introduces quantization noise that unnecessarily degrades accuracy. This mismatch between algorithmic design and hardware behavior leaves substantial efficiency gains untapped. In this work, we argue that efficient LLM inference requires treating decoding as the primary optimization target. Our decoding-centric framework, GRINQH, realizes this by explicitly decoupling the prefill and decoding stages. Specifically, GRINQH preserves high precision during prefill to maintain model accuracy while dynamically adjusting weight precision (0–8 bits) during decoding based on activation magnitudes. A custom bit-planar memory layout ensures that only the required bits are fetched from DRAM for each row, directly alleviating the bandwidth bottleneck. Our custom GPU decoding kernel shows that latency scales smoothly with effective bits and achieves performance competitive with optimized fixed-precision kernels such as MARLIN [15]. Furthermore, assigning higher precision to weights associated with high-magnitude activations (outliers) [23], while aggressively reducing precision elsewhere, leads to significantly improved accuracy compared to quantizing to a single precision level. The inherent nested weight structure also provides GRINQH with elastic multi-precision capability, enabling real-time trade-offs between generation quality and speed within a single deployed model. Through synergies between our algorithmic framework and efficient GPU kernels, GRINQH not only outperforms existing methods at iso-bit precision but also defines a new Pareto frontier for the trade-off between generation quality and generation speed (Fig. 1). Our key contributions are as follows: • Dynamic PTQ Framework: We introduce GRINQH, a fine-grained, dynamic post-training quantization (PTQ) framework that targets the memory-bound decoding stage of LLM inference by loading weight channels in varying precisions depending on the magnitude of real-time input activations. • Superior Accuracy-Efficiency Trade-offs: We show that GRINQH consistently outperforms state-of-the-art static PTQ methods, such as AWQ and GPTQ, in terms of accuracy at comparable effective bit widths, particularly for 2-, 3- and 4-bit quantization. In doing so, our method establishes a new Pareto frontier in generation quality versus generation speed. • Optimized GPU Kernels: We develop a custom GPU decoding kernel that scales gracefully with the number of effective bits, achieving performance competitive with optimized fixedprecision quantization kernels.
2
Related Work
Quantization and Sparsification. Weight quantization and activation sparsification are two widely explored techniques for reducing memory traffic in LLMs. Fundamentally, sparsification can be viewed as an extreme case of quantization where the allocated bit width is reduced to zero (02
bit). Efficient hardware acceleration in this regime requires structured sparsity, such as zeroing out entire weight channels or blocks, to skip memory loads effectively. While activation sparsity naturally induces these patterns by allowing the hardware to bypass weight channels corresponding to zero-valued activations, training-free sparsity induction methods like magnitude-based thresholding [24] often yield insufficient memory savings or significant accuracy degradation. Consequently, sparsification alone is often inadequate to mitigate the memory bottleneck in modern, dense LLMs. Post-Training Quantization (PTQ). Unlike sparsification, quantization reduces memory traffic by representing weights in reduced bit widths rather than skipping them entirely. PTQ is particularly effective as it avoids costly training, but its accuracy is fundamentally challenged by high-magnitude outliers in LLM activations and weights. Early mixed-precision approaches attempt to protect critical weight outliers by keeping salient weights in high precision [11], yet they often struggle to translate theoretical savings into real-world speedups. Subsequent research demonstrated that managing activation outliers is often more crucial for maintaining model performance [23]. Consequently, fixedbit-width methods like GPTQ [14] and AWQ [23] emerged to protect outliers through second-order information or scaling, while techniques such as QuaRot [3] and QuIP [40] suppress outlier effects via basis transformations. However, these methods overlook the divergent hardware bottlenecks of prefill and decoding, and are constrained by applying a single, static precision level regardless of weight importance. To address this, recent mixed-precision research explores three primary paradigms: (i) nested weight representations [31, 28, 21], (ii) static allocation at the layer or block level [12, 28, 21, 19], and (iii) coarse-grained dynamic scheduling, such as per-token precision adjustment [6]. While nested frameworks like any-precision LLM [31] offer theoretical flexibility, they often lack the low-level loading logic required for mixed-precision execution. Similarly, static methods [19] cannot adapt to the runtime emergence of activation outliers, and existing dynamic approaches operate only on a coarse-grained level, e.g., at the token level [6], failing to isolate and protect fine-grained, channel-wise activation outliers. To address these limitations, GRINQH introduces a hardware-verified framework that pairs the efficiency of nested weights with stage-aware, row-wise precision adjustment. Positioning of GRINQH. GRINQH unifies sparsification and quantization within a decoding-centric, input-adaptive framework. Unlike prior static allocation strategies, GRINQH dynamically fetches mixed-precision weights at each decoding step based on real-time activation magnitudes. By assigning zero bits to negligible channels and higher precision to salient ones, GRINQH protects outliers while maximizing effective sparsity. By decoupling the memory-bound decoding stage from prefill, our framework resolves the algorithm-hardware mismatch prevalent in current literature. GRINQH serves as a flexible execution layer compatible with various optimization and quantization methods (e.g., GPTQ), translating theoretical bit-savings into measurable hardware acceleration.
3
GRINQH: Graded Input-based Quantization Hierarchy
We propose GRINQH, a weight-only post-training quantization (PTQ) framework designed to mitigate the memory-bandwidth bottleneck in sequential LLM decoding. GRINQH is built upon two synergistic pillars: (i) an input-dependent precision assignment that adaptively allocates bit widths to weight channels and (ii) a hierarchical bit-planar memory layout that enables low-overhead extraction of multi-precision weights (Fig. 2). To optimize end-to-end throughput, GRINQH differentiates between execution stages based on their primary hardware constraints. During the prefill stage, which is typically compute-bound, weights are loaded at a static, fixed precision. This design choice maintains the high arithmetic intensity required for high-throughput prompt processing. Conversely, during autoregressive decoding with a batch size of one, the system becomes primarily memory-bandwidth bottlenecked (see roofline analysis in Appendix C.2). In this stage, latency is dominated by the time required to transfer weight data from off-chip DRAM to on-chip SRAM. This “memory wall” arises as LLM weights far exceed the capacity of on-chip SRAM, necessitating a full transfer of the entire weight matrix for every generated token. To mitigate this constraint, GRINQH dynamically evaluates the magnitude of incoming activations at each linear layer to determine the optimal precision for the corresponding weight channels. By loading weight channels linked to high-magnitude “outlier” activations at high precision while reducing the precision of, or entirely bypassing the loading of channels with low-magnitude signals, GRINQH significantly reduces DRAM-to-SRAM traffic while preserving accuracy. This precision redistribution prioritizes the model’s most influential representational 3
stacked 6-bit weight
Threshold Logic
MSB
2-bit segment
din
LSB
Memory Bandwidth Bottleneck w0,0 w0,1 w0,2 MSB
4-bit
LSB
2-bit
MSB
input x
interpreted as 6-bit weight
MSB
x~
0-bit
0 bit 2 bit
4 bit 6 bit
6-bit
w3,0 w3,1 w3,2
1) Dynamic Precision Assignment
dout
2) Bit-Planar Weight Storage (Off-Chip DRAM)
Reduced Payload
3) Mixed-Precision Weights Loaded to On-Chip SRAM
Figure 2: Overview of GRINQH. During decoding, GRINQH mitigates the memory bandwidth bottleneck through a dynamic, channel-wise precision loading scheme. 1) Precision Assignment: Input activations xi are mapped to bit widths bi ∈ {0, 2, 4, 6} based on their magnitude via precomputed thresholds from a calibration set. 2) Bit-Stacked Storage: Weights are stored in DRAM using a bit-planar format. Each bmax = 6-bit weight is decomposed into bmax /2 2-bit planes which are stacked along the input-channel dimension (din ). Weights of the same input channel are contiguously aligned along the output-channel axis (dout ), ensuring efficient, coalesced loading of individual input channels. 3) On-Chip Reconstruction: Based on the assigned precision bi for each weight channel, only the most significant segments are fetched through the bandwidth-constrained DRAM interface. These segments are reconstructed on-chip into a mixed-precision weight matrix, significantly reducing DRAM-to-SRAM traffic while maintaining high model performance.
components, recovering quantization accuracy while providing fine-grained, per-channel control over total memory throughput. 3.1
Dynamic Precision Assignment
During the decoding stage, we assign a bit width bi ∈ {b0 , b1 , . . . , bn } (bn = bmax represents the maximum available precision) to each activation element xi along the input dimension (i ∈ {0, . . . , din − 1}) of the current token. This assignment is determined by the magnitude |xi | relative to calibrated thresholds, which dictates the precision of the corresponding weight channel wi,: (see color-coded precision distribution in Fig. 2). Motivated by LLM sensitivity to activation outliers [43] and the existence of “free sparsity” [24], GRINQH protects influential activations with high-precision weights while processing background channels with lower precision or bypassing them entirely. (ℓ)
(ℓ)
Formally, for each layer ℓ, the bit width bi is assigned as bi = bk if |xi | ∈ [θk , θk+1 ), where (ℓ) (ℓ) k ∈ {0, . . . , n} and the boundary thresholds are defined as θ0 = 0 and θn+1 = ∞. By allocating
high precision only to channels associated with large activations, GRINQH effectively implements dynamic activation outlier protection. Unlike static quantization or smoothing methods limited by fixed calibration statistics, GRINQH adapts to shifting outliers at each time step. The dynamic precision allocation induces dynamic structured sparsity that reduces memory traffic and accelerates inference without the accuracy trade-offs inherent to static pruning or quantization. 3.2
Threshold Calibration
To minimize runtime overhead, the thresholds Θ are derived offline via percentile mapping on a small, representative calibration dataset (see Appendix D.2 for an analysis of data robustness and resource consumption). We define a global target precision distribution P = (p0 , p1 , . . . , pn ), where each pk represents the desired proportion of weight channels to be processed at bit width bk . For each layer ℓ, we compute the empirical percentiles that satisfy P for every sample in the calibration set. These 4
(ℓ)
values are averaged to produce a single set of static thresholds Θ(ℓ) , where θk corresponds to the Pk−1 ( j=0 pj )-th percentile of the empirical activation magnitudes. By pre-calculating these thresholds, the runtime “grading” logic is reduced to a set of static scalar comparisons against the incoming activation vector. This ensures that dynamic precision assignment adds negligible latency to the inference pipeline, allowing our multi-precision extraction to maintain performance parity with highly optimized static kernels (see Fig. 4). Further discussion regarding the selection and impact of different precision distributions P is provided in Appendix A. 3.3
Hierarchical Bit-Slicing
GRINQH operates as a quantization-agnostic framework, optimizing bit-width distribution to surpass the baseline performance of the underlying quantization primitive. To realize dynamic multi-precision loading without the memory overhead of storing multiple weight copies, we employ a hierarchical bit-planar interleaved storage format. Storage Layout and Weight Loading. This format requires the underlying quantization to satisfy a nested property, which is naturally met by uniform methods and specific non-uniform techniques such as k-means quantization [31]. Under uniform quantization, each weight wij is quantized to a bmax -bit integer qbmax ; for non-uniform or inherently nested weights, the parent model must match bmax to ensure compatibility. The quantized weight qbmax is decomposed into discrete 2-bit segments representing successive refinements of the value (see Bit-Planar Weight Storage in Fig. 2). This organization enables the kernel to fetch weights in 2-bit increments, supporting an adaptive range of b ∈ {0, 2, . . . , bmax }. While Fig. 2 illustrates a configuration with bmax = 6, the framework is indifferent to the choice of maximum bit width; we evaluate bmax ∈ {4, 6, 8} to balance precision range against storage efficiency. To facilitate high-speed vector-matrix multiplication, weights are stored in a transposed, input-major layout. This ensures that for any element (or block) of the input vector x, the corresponding weight channels for all output dimensions dout are row-contiguous. To access a target precision b ≤ bmax , the kernel only loads the b/2 most significant segments from DRAM to on-chip SRAM, effectively reducing memory traffic while providing the bits required for reconstruction (see Appendix C.1 for more details). Weight Reconstruction and Bias Correction. For uniform quantization, let s denote the quantization scale factor and z the zero-point correction determined for the full bmax precision. The dequantized floating-point value ŵ is reconstructed from the truncated b-bit integer qb as: ŵ = s · (qb · 2bmax −b + ϕ) + z, where qb is the integer formed by the concatenated segments and ϕ = 2bmax −b−1 is a midpoint bias-correction term. Since bit-planar truncation acts as a floor operation, ϕ is essential to shift the reconstruction from the lower bound of the quantization bin to its center. Our ablation studies confirm that without this correction, cumulative quantization bias across deep layers leads to representational collapse. Uncorrected models in our testing degraded to 0% accuracy on GSM8K and exhibited WikiText-2 perplexity explosions exceeding 108 . Notably, for non-uniform nested weights utilizing Look-Up Tables (LUTs), this correction is unnecessary as the nested levels are inherently centered.
4
Experimental Setup
We evaluate GRINQH across the Llama3 (1B, 3B, and 8B Instruct) [17] and Qwen3 (0.6B, 1.7B, 4B, and 8B) [44] model families to demonstrate its robustness across scales and architectures. All accuracy-only evaluations are completed with the reference implementation of GRINQH, a PyTorchlevel simulation of the kernel logic that allows for larger-scale batching. End-to-end results and all reported speedups are obtained using the Triton kernel, which is functionally equivalent to the reference but utilizes an alternative prefill path. Models and Baselines. We compare GRINQH against several uniform quantization methods including Round-to-Nearest (RTN), AWQ [23], GPTQ [14], QuaRot with GPTQ [4], and AutoRound [7], targeting 3-bit and 4-bit symmetric configurations. To reflect modern hardware trends, we also include the 4-bit NVFP4 format [1]. For mixed-precision and non-uniform comparisons, we evaluate 5
against SliM-LLM [19] and any-precision LLM [31]. All methods employ a group size G = 128, except NVFP4 (G = 16) and any-precision LLM (per-channel). Further comparisons are detailed in Appendix B.2. GRINQH utilizes weights pre-quantized via RTN or GPTQ at bmax ∈ {4, 6, 8} bits, as well as any-precision LLM at bmax = 8. For all methods requiring calibration, we use 128 samples from The Pile (Uncopyrighted) dataset [16] with a 2048-token context window. We conducted a hyperparameter sweep over the target precision distributions P (see Appendix A for full selection). Notably, we constrain the 0-bit (sparsity) proportion p0 to a maximum of 30%. Our empirical testing indicated that exceeding this threshold often led to significant performance degradation. Implementation and Deployment. We perform RTN, AWQ, GPTQ, QuaRot, AutoRound, and NVFP4 quantization using the llm-compressor library [33]. For SliM-LLM, any-precision LLM, and PMPD [6], we use the official repositories. Baselines are deployed via vLLM [22] for 4-bit models and HuggingFace Transformers [42] for 3-bit configurations, as vLLM currently lacks native 3-bit kernel support. To deploy GRINQH, we extend the vLLM backend with custom functionality for dynamic precision assignment, selective bit-planar memory access, and weight reconstruction. Evaluation Benchmarks. We evaluate all models for accuracy using the EleutherAI LM Evaluation Harness[37] across seven benchmarks: WikiText-2 (Wiki2), LAMBADA (LMB), MMLU, BoolQ, HellaSwag, ARC-Challenge, and GSM8K (Chain-of-Thought, CoT) [26, 30, 18, 8, 45, 9, 10, 41]. All tasks utilize a 0-shot setting except for GSM8K, which uses 8-shot prompting. Specifically, we evaluate Llama3 using the gsm8k_cot_llama task applying multi-turn chat templates and few-shot CoT prompting to reach its expected performance. In contrast, Qwen3 is evaluated on the standard gsm8k_cot task with chat template disabled. For all other benchmarks, we omit chat templates. We report the average accuracy across all benchmarks excluding WikiText-2. Our end-to-end benchmarks additionally use IFEval [46] (leaderboard settings) and set GSM8K to 0-shot. We furthermore use variable-length prompts from a processed ShareGPT dataset [13] to show the prefill/decoding scaling across different token ratios. Prefill vs. Decoding Evaluation. Standard benchmarks are predominantly prefill-heavy; therefore, to rigorously assess representational fidelity, we apply GRINQH’s dynamic precision assignment to the prefill stage of all experiments performed with the reference implementation. Our kernel instead handles prefill by unpacking the weights at full (bmax ) precision and performing dense matrix-matrix multiplication. Additionally, we conduct specific evaluations on the decode-heavy GSM8K (CoT) 8-shot task to demonstrate GRINQH’s ability to decouple precision requirements between stages, effectively addressing the asymmetric computational demands of prefill and decoding (Fig. 3B-decode-only, Fig. 9B-decode-only). Setup for Kernel Benchmarking. Our kernel is implemented in Triton [38]. To obtain isolated decoding timings (see Fig. 4), we utilize the framework’s benchmarking suite with a standalone weight matrix of shape W ∈ R16,384×4,096 , representative of linear up-projection layers in modern 8B-parameter models. We compare against baselines taken from vLLM (MARLIN [15]) and native PyTorch implementations (TorchAO [39]). For our end-to-end benchmarks, we integrate GRINQH into vLLM, making it a selectable quantization method for any model given thresholds for the desired bit width.
5
Results
Quantization Performance and Pareto Frontiers. GRINQH demonstrates that near-lossless LLM inference is achievable at significantly reduced effective bit widths. As illustrated by the Pareto frontier for Qwen3 8B (Fig. 1), our framework outperforms 4-bit state-of-the-art methods on GSM8K while operating 25% faster on an RTX 4090 than optimized 4-bit MARLIN kernels. This speedup stems directly from our dynamic precision assignment, which enables the reduction or omission of non-critical weight channels. By prioritizing the protection of performance-critical weights, GRINQH drives down the average effective bit width without compromising model quality. Notably, GRINQH achieves up to 20× lower accuracy degradation compared to competitive 3-bit methods, effectively reconciling hardware efficiency with representational integrity. Pareto Frontier Across Scales. To demonstrate GRINQH’s robustness across model scales, Fig. 3 illustrates the emergence of a new SOTA Pareto frontier spanning the 2–6 bit range. For larger models such as Qwen3 4B and 8B (Fig. 3B (middle)) performance tracks the BF16 baseline closely down to 2.5 bits when using bmax ∈ {6, 8}, effectively closing the accuracy gap that is traditionally 6
GRINQH Max 8bit GRINQH Max 6bit GRINQH Max 4bit GPTQ AWQ RTN
Wikitext – Word Perplexity
30
20
Llama-3.2-1B Llama-3.2-3B 10
Llama-3.1-8B 2
3 4 5 6 Effective Number of Bits
GRINQH Max 4-bits
(B)
GRINQH Max 6/8-bits
GRINQH Max 6/8-bits Decode Only
Qwen3-8B Qwen3-4B
80 GSM8K – Exact Match Accuracy [%]
GRINQH Max 6/8-bits
(A)
Qwen3-1.7B Qwen3-1.7B
60
Qwen3-0.6B Qwen3-0.6B
40
20
0 2
3
4
2
3 4 5 6 Effective Number of Bits
2
3
4
5
6
Figure 3: GRINQH redefines the quantization Pareto frontier across model families and scales. Dashed lines indicate BF16 baselines. GRINQH precision distribution sweep is compared against iso-bit symmetric RTN, GPTQ, and AWQ baselines. (A) WikiText-2 perplexity vs. effective bit width for the Llama3 Instruct family using bmax ∈ {6, 8}. (B) GSM8K CoT accuracy vs. effective bit width for the Qwen3 family. (Left panel) GRINQH (bmax = 4) sweep on GPTQ baseline. (Middle panel) GRINQH (bmax ∈ {6, 8} on RTN baseline). (Right panel) Real-case scenario with decoupled inference strategy ( bmax ∈ {6, 8} on RTN baseline) where dynamic precision loading is enabled exclusively for the memory-bound decoding stage.
associated with 3-bit quantization. Similarly, for the Llama3 family (Fig. 3A), GRINQH maintains effective performance deep into the 2–3 bit regime, preventing the catastrophic perplexity explosions characteristic of low-bit quantization. In this regime, GRINQH significantly separates itself from widely adopted methods like GPTQ and AWQ at 3 bits. Across all evaluated scales and architectures, our framework consistently enables high-fidelity performance at fractional bit widths, establishing a superior Pareto frontier for low-bit LLM inference. Further results are provided in Appendix B.1.1. The GRINQH max-4-bit Baseline. Even when restricted to a 4-bit base (bmax = 4), GRINQH establishes a superior Pareto frontier (Fig. 3B (left panel)). The GRINQH data points seamlessly emerge from the 4-bit-baseline, with large Qwen3 models maintaining near-lossless performance for a significant range of bit width before declining. While GRINQH with bmax = 4 cannot exceed its own 4-bit source, it consistently outperforms specialized 3-bit RTN, AWQ and GPTQ models. This suggests that GRINQH’s dynamic bit-planar fetching is a more robust mechanism for bit-rate reduction than static low-bit calibration. At the same time, it highlights that GRINQH can also build on top of sophisticated quantization methods with any quantized base precision format. Consequently, GRINQH enables to seamlessly scale down effective precision, achieving accelerated inference without further architectural changes. Cross-Benchmark Superiority. Tab. 1 compares Llama3 8B and Qwen3 8B across the benchmark suite (full results for all scales are in Appendix B.1). At an effective bit width of ∼ 4 bits (bmax ∈ {6, 8}), GRINQH matches the BF16 baseline and consistently outperforms all uniform and nonuniform approaches. This stability extends into the sub-3-bit regime, where static PTQ typically suffers from catastrophic representational collapse. Additionally, GRINQH pushed to an effective width of 2.01–2.22 bits outperforms all tested SOTA methods operating at a full 3 bits, saving 1 bit per weight while simultaneously delivering superior accuracy. Notably, for Qwen3 8B, our 2.22-bit configuration even surpasses the average accuracy of the majority of 4-bit baselines. While competitive methods (exemplified by any-precision LLM and SliM-LLM) collapse toward chance-level accuracy at 2 bits, GRINQH maintains robustness. Although utilizing nested weights (any-precision LLM) provides a performance boost over simple bit-slicing in this ultra-low-bit regime, the fact that our RTN-based configuration also maintains high accuracy demonstrates that GRINQH’s superiority is driven primarily by its inherent precision allocation logic rather than the underlying quantization primitive. 7
Table 1: Benchmark results for the Llama3 8B and Qwen3 8B models. The average accuracy (Avg.) is calculated based on the accuracy metric of the following benchmarks: LAMBADA (LMB), ARC-C, HellaSwag, BoolQ, MMLU and GSM8K. Llama-3.1-8B-Instruct Eff. Bits
Wiki2 ppl ↓
BF16-Baseline
16
8.64
GPTQ [14] AWQ [23] RTN QuaRot+GPTQ [4] AutoRound [7] NVFP4-G16 [1] AnyPrecLLM [31] SliM-LLM [19] GRINQH-8b-RTN GRINQH-8b-AnyP GRINQH-6b-RTN
4 4 4 4 4 4 4 4 4.01 3.89 3.92
9.41 9.53 9.67 9.38 9.62 9.27 9.44 9.35 8.97 8.95 9.02
GPTQ [14] AWQ [23] RTN QuaRot+GPTQ [4] AutoRound [7] AnyPrecLLM [31] SliM-LLM [19] GRINQH-8b-RTN GRINQH-8b-AnyP GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
3 3 3 3 3 3 3 2.98 2.96 3.02 3.02 3.04
12.46 14.36 19.13 12.80 11.81 12.22 12.01 9.30 9.22 9.44 9.80 9.59
AnyPrecLLM [31] SliM-LLM [19] GRINQH-8b-RTN GRINQH-8b-AnyP
2 2 2.11 2.01
1953 535 12.87 10.47
LMB ppl ↓
Qwen3-8B
GSM8K acc ↑
Avg. acc ↑
Eff. Bits
Wiki2 ppl ↓
LMB ppl ↓
GSM8K acc ↑
Avg. acc ↑
3.40
85.14
74.35
16
12.20
4.59
88.32
74.02
3.64 3.91 3.90 3.55 3.89 3.41 3.71 3.46 3.38 3.36 3.41
80.82 80.59 79.83 81.88 80.89 82.49 81.88 82.64 85.14 84.53 84.31
72.85 71.98 71.44 72.86 72.25 73.24 72.84 72.82 74.34 74.19 73.91
4 4 4 4 4 4 4 4 3.97 3.95 3.94
12.68 12.70 12.89 12.58 13.30 12.38 12.82 12.61 12.29 12.23 12.27
5.17 5.09 5.40 4.81 5.94 5.07 4.70 5.20 4.58 4.44 4.58
82.79 82.41 74.75 86.28 84.08 86.13 87.64 87.41 87.57 87.72 87.19
72.18 71.77 69.89 72.99 71.79 73.36 73.36 72.50 73.70 73.91 73.81
5.86 8.04 7.43 5.77 5.24 4.83 4.85 3.49 3.37 3.40 3.80 3.56
60.35 34.50 6.60 57.54 49.58 56.18 68.46 81.50 84.23 82.41 79.68 79.91
63.27 56.05 50.64 63.40 63.00 64.59 66.35 73.19 73.69 72.75 71.34 72.34
3 3 3 3 3 3 3 3.07 3.06 3.06 3.03 3.03
14.89 16.33 22.84 14.28 15.08 15.23 14.88 12.61 12.40 12.58 13.02 12.84
6.96 14.24 41.29 7.30 7.31 6.50 5.58 4.51 4.35 4.53 5.30 5.10
67.78 59.67 15.54 68.84 67.78 79.30 77.63 88.02 87.41 85.60 79.38 82.56
64.67 59.19 44.79 65.03 65.45 69.28 67.64 73.33 73.74 73.14 70.94 71.75
1e4 3e4 5.94 3.72
1.52 2.96 53.53 71.65
22.22 19.59 61.37 69.06
2 2 2.25 2.22
110 101 13.36 12.95
669 212 5.12 4.21
0.61 2.12 84.00 87.04
33.80 31.44 71.60 73.27
Table 2: Tok/s performance over different effective bit widths and input:output ratios (prefill: (3000:1), decoding (1:3000)) of tokens on random data. Relative performance is calculated against GPTQ 4-bit MARLIN. Kernel
Prefill Abs. ( Rel. )
Decoding Abs. ( Rel. )
Eff. 2-bit GRINQH-8b Eff. 3-bit GRINQH-8b Eff. 4-bit GRINQH-8b
8187 ( 0.80× )
204.6 ( 1.28× ) 182.2 ( 1.14× ) 157.6 ( 0.98× )
Eff. 4-bit GRINQH-6b Eff. 3-bit GRINQH-4b
8241 ( 0.81× ) 8241 ( 0.81× )
165.1 ( 1.03× ) 165.1 ( 1.03× )
GPTQ 4-bit MARLIN RTN 8-bit MARLIN
10170 ( 1.00× ) 8927 ( 0.88× )
160.4 ( 1.00× ) 100.2 ( 0.62× )
Latency [Normalized to Marlin-4-bit]
Method
Marlin-4-bit Marlin-8-bit GRINQH TorchAO
1.8 1.6 1.4 1.2
GeForce RTX 5070 GeForce RTX 4090 RTX PRO 6000
1.0 0.8 2
4
6
8
Effective Number of Bits
Figure 4: Normalized isolated kernel runtimes over a range of target effective bit widths. Times are normalized w.r.t. 4-bit Marlin kernel on the same device.
Representational Fidelity in Decoupled Inference. We evaluate GRINQH in a realistic inference setting where prefill and decoding are treated separately. Since the prefill stage is primarily computebound, loading weights at bmax introduces negligible latency overhead. We therefore maintain maximum precision during prefill to preserve representational fidelity, while employing GRINQH’s dynamic mixed-precision logic exclusively during the memory-bound decoding stage. Fig. 3B (right panel) shows that this stage-aware approach yields a significant performance boost; the resulting Pareto frontiers are more densely packed and exhibit a delayed accuracy drop compared to the 8
(A)
(B)
(C)
QuaRot AutoRound AWQ
QuaRot AWQ GPTQ
GPTQ
AutoRound
Figure 5: End-to-End performance scaling and task evaluation. (A) Decoding throughput (tokens/s) over increasingly longer outputs relative to prompts selected from the ShareGPT dataset, averaged over 1000 samples across two seeds ( 225 avg. input tokens per prompt). While prefill initially weighs down the average throughput, GRINQH delivers SOTA performance even on tasks with considerable relative prefill. Vertical lines represent the average out/in ratio for GSM8K 0-shot (B) and IFEval (C) respectively. MARLIN 4-bit evaluated on GPTQ, 8-bit on RTN. (B, C) Speed vs. strict accuracy for GSM8K 0-shot with 4096 maximum output tokens per prompt and IFEval (leaderboard settings) with 256. In the low-bit regime, GRINQH consistently forms the Pareto front. Accuracy refers to “strict-match” for GSM8K and “inst_level_strict_acc” for IFEval. All measurements for NVIDIA RTX 4090 using Llama3.1-8B. stage-agnostic counterparts. This enables even smaller models to operate effectively below an average of 3 bits, demonstrating that by decoupling these stages, GRINQH bypasses the traditional accuracy-efficiency trade-off and protects initial context processing without sacrificing decoding throughput. Decoding Throughput and Bit-Width Scaling. To evaluate hardware efficiency, we benchmark our decoding kernel against state-of-the-art baselines. Our implementation achieves performance parity with optimized 4-bit and 8-bit kernels while exhibiting graceful linear scaling across intermediate effective precisions (Fig. 4). This confirms that the kernel’s runtime scales proportionally with the effective bit width, showing that non-uniform precision distributions incur no relevant overhead. We validate these results across a range of architectures, including consumer-grade (NVIDIA RTX 4090, NVIDIA RTX 5070) and professional-grade (NVIDIA RTX PRO 6000) GPUs. End-to-end Measurements. For our kernel we perform bmax dense prefill. Due to efficient unpacking and kernel fusion this is only 20% slower than optimized prefill for 4-bit weights, see Tab. 2. Limiting the maximum bits to < 8 allows for lower sustained memory usage and gives a slight boost to prefill. The slower prefill performance is outweighed by up to 28% faster decoding speed. As shown in Fig. 5A, the performance gains observed in both isolated kernel (Fig. 4) and full-model (Tab. 2) benchmarks translate well to end-to-end speedups (measured in output tokens/s) over the full inference duration, including prefill. This holds for all evaluated tasks, establishing Pareto-optimal points for all GRINQH configurations runs in Fig. 5B,C. Notably, our 3-bit effective configuration retains significantly higher accuracy than existing 4-bit SOTA methods while simultaneously offering a superior end-to-end speedup. This result effectively demonstrates that GRINQH can surpass the accuracy-efficiency trade-off of static 4-bit quantization, redefining the performance ceiling for low-bit LLM deployment.
6
Conclusion
In this work, we propose GRINQH, a decoding-centric quantization framework that addresses the memory-bandwidth bottleneck in LLM inference through input-dependent precision assignment. By unifying dynamic sparsity and quantization, GRINQH substantially reduces the volume of data 9
transferred from off-chip memory during decoding. This fine-grained control of precision enables high-throughput inference while surpassing the accuracy of state-of-the-art quantization methods at equivalent, or even significantly lower effective bit widths. GRINQH functions as a quantization-agnostic framework that enhances existing formats like RTN or GPTQ, while further benefiting from sophisticated nested weight techniques such as any-precision LLM in the sub-3-bit regime. Notably, the inherent structure of these nested weights provides elastic multi-precision profiles within a single set of parameters, offering a promising architectural foundation for future research into speculative decoding and adaptive precision scheduling. Yet, GRINQH targets batch-size-one scenarios and utilizes bit-planar storage, which necessitates a bmax -DRAM footprint. However, when restricted to a static 4-bit memory footprint (bmax = 4), GRINQH’s 3-bit inference closely matches the accuracy of 4-bit SOTA baselines for some models while significantly reducing DRAM-to-SRAM traffic. Furthermore, while prefill performance is slightly impacted by the requirement to unpack bit-planar representations, this is balanced by the advantages of stage-aware optimization. By decoupling inference stages, GRINQH achieves nearlossless context processing during prefill while leveraging dynamic precision during decoding to protect performance-critical outliers and adapt to activation distribution shifts. Additionally, GRINQH unlocks flexible bit widths that more fluidly adapt to model and hardware needs, unconstrained by standard fixed-precision grids. Ultimately, GRINQH translates theoretical gains into tangible reductions in memory traffic, redefining the Pareto frontier for low-bit LLM deployment.
Acknowledgments and Disclosure of Funding This work was sponsored by the Federal Ministry of Education, Germany BMBF under project NEUROTEC-II grants no. 16ME0398K and 16ME0399, and 01IS22094E WestAI - AI Service Center West; and Neurosys as part of the initiative "Cluster4Future" funded by the Federal Ministry of Education and Research BMBF (03ZU1106CB); and Phase II: NeuroSys as part of the initiative “Clusters4Future” funded by the Federal Ministry of Research, Technology and Space BMFTR (03ZU2106CB). The authors gratefully acknowledge computing time on the supercomputer JURECA [20] and JUWELS [2] at Forschungszentrum Jülich. They further thank Abigail Morrison for support during this work, and Viet Anh Khoa Tran and Matthias Oberländer for providing thoughtful comments on the manuscript.
Author Contributions JO helped define the trajectory of the GRINQH method, performed the empirical evaluation, including design of model calibration and quantization, accuracy experiments, and the implementation of baseline comparisons. JO implemented the initial functional prototype within vLLM to validate the empirical proof of concept and led the manuscript preparation. JF conceptualized the GRINQH approach, developed the main algorithmic formulation, designed and implemented the custom Triton decoding kernel, contributed to the prefill implementation strategy, and supervised the technical development and evaluation strategy of the project. CS developed the PyTorch-based prefill path, contributed to the Triton kernel development, and led the benchmarking, profiling, and final kernel integration into vLLM. EN provided project supervision and guidance. All authors contributed to the interpretation of results and the final revision of the manuscript.
References [1] F. Abecassis, A. Agrusa, D. Ahn, J. Alben, S. Alborghetti, M. Andersch, S. Arayandi, A. Bjorlin, A. Blakeman, E. Briones, et al. Pretraining large language models with nvfp4. arXiv preprint arXiv:2509.25149, 2025. [2] D. Alvarez. Juwels cluster and booster: exascale pathfinder with modular supercomputing architecture at juelich supercomputing centre. Journal of large-scale research facilities JLSRF, 7:A183–A183, 2021. [3] S. Ashkboos, I. Markov, E. Frantar, T. Zhong, X. Wang, J. Ren, T. Hoefler, and D. Alistarh. Quik: Towards end-to-end 4-bit inference on generative large language models, 2023. URL https://arxiv.org/abs/2310.09259. 10
[4] S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman. Quarot: Outlier-free 4-bit inference in rotated llms, 2024. URL https: //arxiv.org/abs/2404.00456. [5] Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. [6] H. M. Chen, F. Tan, A. Kouris, R. Lee, H. Fan, and S. I. Venieris. Progressive mixed-precision decoding for efficient llm inference, 2024. URL https://arxiv.org/abs/2410.13461. [7] W. Cheng, W. Zhang, H. Shen, Y. Cai, X. He, K. Lv, and Y. Liu. Optimize weight rounding via signed gradient descent for the quantization of llms, 2023. URL https://arxiv.org/abs/ 2309.05516. [8] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv. org/abs/1905.10044. [9] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: //arxiv.org/abs/1803.05457. [10] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. [11] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. URL https://arxiv.org/abs/2208.07339. [12] Devvrit, S. Kudugunta, A. Kusupati, T. Dettmers, K. Chen, I. Dhillon, Y. Tsvetkov, H. Hajishirzi, S. Kakade, A. Farhadi, and P. Jain. Matformer: Nested transformer for elastic inference, 2023. URL https://arxiv.org/abs/2310.07707. [13] D. Eccleston. sharegpt, 2022. URL https://github.com/domeccleston/sharegpt. Accessed 2026-03-12. [14] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2022. URL https://arxiv.org/abs/2210.17323. [15] E. Frantar, R. L. Castro, J. Chen, T. Hoefler, and D. Alistarh. Marlin: Mixed-precision autoregressive parallel inference on large language models, 2024. URL https://arxiv.org/ abs/2408.11743. [16] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL https://arxiv.org/abs/2101.00027. [17] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. [18] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2020. URL https://arxiv.org/abs/2009. 03300. [19] W. Huang, H. Qin, Y. Liu, Y. Li, Q. Liu, X. Liu, L. Benini, M. Magno, S. Zhang, and X. Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models, 2024. URL https://arxiv.org/abs/2405.14917. [20] Jülich Supercomputing Centre. JURECA: Data Centric and Booster Modules implementing the Modular Supercomputing Architecture at Jülich Supercomputing Centre. Journal of large-scale research facilities, 7(A182), 2021. doi: 10.17815/jlsrf-7-182. URL http://dx.doi.org/10. 17815/jlsrf-7-182. [21] M. Kleinegger, E. Crnčević, and D. Alistarh. Matgptq: Accurate and efficient post-training matryoshka quantization, 2026. URL https://arxiv.org/abs/2602.03537. [22] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 11
[23] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2023. URL https://arxiv.org/abs/2306.00978. [24] J. Liu, P. Ponnusamy, T. Cai, H. Guo, Y. Kim, and B. Athiwaratkun. Training-free activation sparsity in large language models, 2024. URL https://arxiv.org/abs/2408.14690. [25] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re, and B. Chen. Deja vu: Contextual sparsity for efficient llms at inference time. 2023. doi: 10.48550/ARXIV.2310.17157. URL https://arxiv.org/abs/2310.17157. [26] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843. [27] I. Mirzadeh, K. Alizadeh, S. Mehta, C. C. Del Mundo, O. Tuzel, G. Samei, M. Rastegari, and M. Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv.org/abs/2310.04564. [28] P. Nair, P. Datta, J. Dean, P. Jain, and A. Kusupati. Matryoshka quantization, 2025. URL https://arxiv.org/abs/2502.06786. [29] NVIDIA Corporation. NVIDIA Nsight Compute CLI. NVIDIA Corporation, 2026. URL https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html. Version 2025.4.1. [30] D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL https://arxiv.org/abs/1606.06031. [31] Y. Park, J. Hyun, S. Cho, B. Sim, and J. W. Lee. Any-precision llm: Low-cost deployment of multiple, different-sized llms, 2024. URL https://arxiv.org/abs/2402.10517. [32] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference, 2022. URL https: //arxiv.org/abs/2211.05102. [33] Red Hat AI and vLLM Project. LLM Compressor. https://github.com/vllm-project/ llm-compressor, Aug. 2024. [34] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [35] C. Song, X. Han, Z. Zhang, S. Hu, X. Shi, K. Li, C. Chen, Z. Liu, G. Li, T. Yang, and M. Sun. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models, 2024. URL https://arxiv.org/abs/2402.13516. [36] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models, 2023. URL https://arxiv.org/abs/2306.11695. [37] L. Sutawika, H. Schoelkopf, L. Gao, B. Abbasi, S. Biderman, J. Tow, B. Fattori, C. Lovering, J. Phang, A. Thite, T. Wang, sdtblck, gakada, nopperl, researcher2, tttyuntian, E. Julen, Chris, J. A. Michaelov, H. A. Lee, Janna, L. Sinev, Z. Kasner, K. Stokes, Khalid, and KonradSzafer. Eleutherai/lm-evaluation-harness: lm-eval v0.4.9.2 release notes, 2025. URL https://zenodo. org/doi/10.5281/zenodo.17728786. [38] P. Tillet, H. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. 2019. URL https://www.eecs.harvard.edu/~htk/ publication/2019-mapl-tillet-kung-cox.pdf. [39] torchao. Torchao: Pytorch-native training-to-serving model optimization, oct 2024. URL https://github.com/pytorch/ao. [40] A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024. URL https://arxiv.org/abs/ 2402.04396. [41] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2022. URL https: //arxiv.org/abs/2201.11903. 12
[42] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Huggingface’s transformers: State-of-the-art natural language processing, 2019. URL https://arxiv.org/abs/1910.03771. [43] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2022. URL https://arxiv. org/abs/2211.10438. [44] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. [45] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830. [46] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instructionfollowing evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
13
A
Appendix: Hyperparameter Selection and Tuning
This section details the formalization of our precision distribution (Section A.1), describes the random sweep used to validate the robustness of the framework (Section A.2), and provides general recommendations for selecting suitable hyperparameters (Section A.3). A central finding of this work is that GRINQH consistently outperforms state-of-the-art (SOTA) fixed-bit baselines across a wide range of precision allocations. This suggests that the performance gains are inherent to our dynamic execution layer rather than a result of exhaustive hyperparameter over-tuning. While we formulate the optimal distribution as a constrained optimization problem, we leave the development of automated search strategies or a training-based approach for future work. A.1
Precision Distribution Formalization
A core hyperparameter in our framework is the precision distribution, which defines the fraction of a model’s weights assigned to specific precision levels within a linear layer. Specifically, given a maximum bit width bmax , we define a set of available precisions B = (b0 , b1 , . . . , bn ), where bi = 2i and n = bmax /2. The precision distribution is characterized by a vectorPof fractions P = (p0 , p1 , . . . , pn ) ∈ ∆n+1 , where ∆n+1 denotes the probability simplex such that pi = 1 and pi ≥ 0. Each pi represents the portion of weights allocated to bit width bi . To ensure the resulting model meets the user-defined target effective bit width (b∗ ), the distribution is constrained such that the weighted average bit width approximates b∗ . We formalize the selection of P as a constrained optimization problem: ! n n X X max M(Model(P )) s.t. pi = 1, pi ≥ 0, pi · b i − b ∗ ≤ ϵ (1) P
i=0
i=0
where M represents a (task-specific) performance metric (e.g., accuracy or negative perplexity) and ϵ is a small tolerance parameter (e.g., 0.01) introduced to allow for a wider range of valid precision distributions. Implementation via GRINQH. While the optimization problem determines the ideal configuration P , the GRINQH framework realizes the precision allocation of the weights by applying thresholds to the magnitudes of their corresponding activations. These thresholds are calibrated based on P . Specifically, for every calibration step, we identify the separating magnitudes (thresholds) that partition the activations into the fractions defined by P by applying a top-p approach. For each linear layer ℓ, these values are logged and subsequently averaged across all forward passes to obtain the final vector of fixed thresholds: (ℓ) (ℓ) Θ(ℓ) = θ1 , . . . , θn−1 (2) Fig. 6B shows that these calibrated thresholds successfully maintain the target effective bit width b∗ during inference across a variety of benchmark tasks. A detailed robustness analysis of this calibration phase is provided in Appendix D. A.2
Search Space and Robustness
To generate the Pareto curves (Fig. 1, Fig. 3, Fig. 9), we conducted a random sweep over the configuration space. The sampled distributions are detailed in Tab. 3. These distributions were selected to ensure a representative spread across effective bit widths b∗ ∈ [2, 7]. It is important to note that in this work we did not explicitly solve the optimization problem defined in the previous section. Instead, we sampled valid precision distributions P that satisfy the bit-width constraints without applying any sophisticated methods to maximize the performance metric M. We observe that even with these randomly sampled configurations, the majority of our results consistently outperform state-of-the-art (SOTA) fixed-bit baselines. This suggests that the framework is highly robust to the specific choice of P and that accuracy performance gains are inherent to the flexible precision allocation rather than hyperparameter over-tuning. As shown by the Pareto frontiers, performance is significantly more tied to the target bit width b∗ than to the specific distribution used, particularly in the lower-bit regime. However, the observed 14
variance between different configurations at identical bit widths suggests that solving the optimization problem for P remains a promising direction for future research. In the following section, we provide a heuristic guide for identifying “good” hyperparameter sets that yield strong performance without requiring an exhaustive search. Table 3: Precision distribution (P) sweep for various bmax values, where (p0 , p1 , p2 , p3 , p4 ) represents the fractions of 0, 2, 4, 6, and 8-bit widths, respectively. The distributions are sampled with the constraint 0.15 ≤ p0 ≤ 0.30, ensuring a significant level of activation sparsity. To maintain focus on the low-to-mid bit-width regime, the 8-bit fraction (p4 ) is constrained to 0.0 for most samples. All distributions are filtered to ensure the resulting effective bit width (b) satisfies 2 ≤ b ≤ 7. # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
bmax = 8 (p0 , p1 , p2 , p3 , p4 ) 0.30 0.35 0.28 0.15 0.68 0.13 0.25 0.30 0.41 0.20 0.40 0.32 0.30 0.21 0.33 0.35 0.07 0.51 0.20 0.32 0.44 0.30 0.17 0.36 0.25 0.22 0.31 0.25 0.19 0.37 0.25 0.08 0.59 0.25 0.25 0.24 0.20 0.16 0.56 0.20 0.24 0.32 0.20 0.20 0.40 0.15 0.13 0.68 0.30 0.15 0.16 0.25 0.10 0.38 0.15 0.21 0.43 0.15 0.09 0.67 0.30 0.05 0.32 0.30 0.05 0.26 0.25 0.05 0.35 0.20 0.18 0.17 0.25 0.05 0.40 0.25 0.08 0.15 0.15 0.19 0.18 0.00 0.00 1.00 0.20 0.08 0.24 0.15 0.05 0.48 0.20 0.16 0.08 0.25 0.10 0.06 0.30 0.06 0.07 0.15 0.09 0.17 0.25 0.05 0.07 0.20 0.06 0.06 0.20 0.10 0.10 0.30 0.05 0.05 0.15 0.10 0.10 0.30 0.05 0.30 0.25 0.15 0.10 0.30 0.09 0.14
0.07 0.04 0.04 0.08 0.16 0.07 0.04 0.17 0.22 0.19 0.08 0.26 0.08 0.24 0.20 0.04 0.39 0.20 0.21 0.09 0.26 0.36 0.28 0.45 0.22 0.52 0.48 0.00 0.48 0.24 0.56 0.39 0.50 0.59 0.42 0.64 0.42 0.21 0.50 0.20 0.38 0.07
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.07 0.03 0.07 0.00 0.08 0.00 0.00 0.00 0.00 0.08 0.00 0.20 0.07 0.00 0.21 0.04 0.18 0.39 0.15 0.15 0.12 0.40
bmax = 6 (p0 , p1 , p2 , p3 ) 0.30 0.35 0.18 0.15 0.56 0.13 0.25 0.30 0.30 0.20 0.36 0.28 0.30 0.21 0.33 0.35 0.07 0.41 0.20 0.32 0.30 0.30 0.17 0.36 0.25 0.22 0.31 0.25 0.19 0.37 0.25 0.08 0.49 0.25 0.25 0.24 0.20 0.16 0.46 0.20 0.24 0.32 0.20 0.20 0.40 0.15 0.13 0.50 0.30 0.15 0.16 0.25 0.10 0.38 0.15 0.21 0.43 0.15 0.09 0.47 0.30 0.05 0.32 0.30 0.05 0.26 0.25 0.05 0.35 0.20 0.18 0.17 0.25 0.05 0.40 0.25 0.08 0.15 0.15 0.19 0.18 0.00 0.00 1.00 0.20 0.08 0.24 0.15 0.05 0.48 0.20 0.16 0.08 0.25 0.10 0.06 0.30 0.06 0.07 0.15 0.09 0.17 0.25 0.05 0.07 0.20 0.06 0.06 0.20 0.10 0.28 0.30 0.05 0.44 0.15 0.25 0.10 0.30 0.05 0.30 0.25 0.15 0.10 0.30 0.09 0.14
15
0.17 0.16 0.15 0.16 0.16 0.17 0.18 0.17 0.22 0.19 0.18 0.26 0.18 0.24 0.20 0.22 0.39 0.27 0.21 0.29 0.33 0.39 0.35 0.45 0.30 0.52 0.48 0.00 0.48 0.32 0.56 0.59 0.57 0.59 0.63 0.68 0.42 0.21 0.50 0.35 0.50 0.47
bmax = 4 (p0 , p1 , p2 ) 0.00 0.00 0.20 0.05 0.18 0.04 0.15 0.05 0.25 0.03 0.30 0.02 0.05 0.15 0.04 0.18 0.03 0.17 0.02 0.20 0.01 0.19 0.10 0.05 0.08 0.07 0.06 0.09 0.05 0.10 0.04 0.11 0.12 0.03 0.10 0.04 0.08 0.05 0.06 0.06 0.05 0.05 0.02 0.08 0.03 0.07 0.04 0.06 0.05 0.04 0.01 0.09 0.15 0.02 0.18 0.02 0.22 0.01 0.25 0.01 0.28 0.01 0.03 0.03 0.02 0.04 0.01 0.05 0.04 0.02 0.05 0.01 0.25 0.10 0.30 0.05 0.20 0.20 0.15 0.25 0.25 0.20 0.25 0.25
1.00 0.75 0.78 0.80 0.72 0.68 0.80 0.78 0.80 0.78 0.80 0.85 0.85 0.85 0.85 0.85 0.85 0.86 0.87 0.88 0.90 0.90 0.90 0.90 0.91 0.90 0.83 0.80 0.77 0.74 0.71 0.94 0.94 0.94 0.94 0.94 0.65 0.65 0.60 0.60 0.55 0.50
# 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
bmax = 8 (p0 , p1 , p2 , p3 , p4 ) 0.15 0.08 0.13 0.15 0.05 0.12 0.25 0.00 0.00 0.30 0.00 0.00 0.15 0.00 0.25 0.30 0.00 0.00 0.15 0.00 0.00 0.15 0.00 0.08 0.30 0.00 0.00 0.25 0.00 0.00 0.25 0.00 0.00 0.20 0.00 0.00 0.25 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.15 0.00 0.00 0.00 0.00 0.00
0.60 0.59 0.52 0.28 0.34 0.21 0.68 0.51 0.07 0.23 0.15 0.32 0.08 0.26 1.00 0.16 0.17 0.00
0.04 0.09 0.23 0.42 0.26 0.49 0.17 0.26 0.63 0.52 0.60 0.48 0.67 0.54 0.00 0.64 0.68 1.00
bmax = 6 (p0 , p1 , p2 , p3 ) 0.15 0.08 0.13 0.15 0.05 0.12 0.25 0.00 0.00 0.20 0.00 0.10 0.15 0.00 0.25 0.30 0.00 0.00 0.15 0.18 0.17 0.15 0.00 0.34 0.30 0.00 0.07 0.25 0.00 0.23 0.25 0.00 0.15 0.20 0.00 0.32 0.25 0.08 0.00 0.20 0.13 0.13 0.00 0.00 0.00 0.20 0.00 0.16 0.15 0.00 0.17 0.15 0.25 0.40
0.64 0.68 0.75 0.70 0.60 0.70 0.50 0.51 0.63 0.52 0.60 0.48 0.67 0.54 1.00 0.64 0.68 0.20
bmax = 4 (p0 , p1 , p2 ) 0.30 0.20 0.35 0.10 0.15 0.40 0.25 0.30 0.25 0.34 0.27 0.33 0.30 0.33 0.30 0.28 0.35 0.23 0.35 0.25
0.50 0.55 0.45 0.45 0.41 0.40 0.37 0.42 0.42 0.40
Table Model Selection Criteria. For the table comparisons in the main text and appendix (Tab. 1, Tab. 4, Tab. 5), we selected configurations based on their Wikitext-2 perplexity. In this context, Wikitext-2 perplexity serves as our proxy for the performance metric M that is aimed to be maximized. This selection criterion demonstrates that configurations chosen to optimize for a general language modeling objective generalize robustly to specific downstream accuracy-based tasks. A.3
Empirical Guidelines for Configuration
The flexibility of the GRINQH framework allows for a vast configuration space. Based on our sensitivity analysis, we provide the following observations and a streamlined workflow for identifying high-performance precision distributions. Regime-Dependent Sparsity Trade-offs. Our analysis (Fig. 6A) reveals that the relationship between sparsity and performance is highly dependent on the target bit-budget. In extreme low-bit regimes (2.0 ≤ b ≤ 3.0), higher sparsity levels (≥ 20%) act as a critical enabler. By aggressively exploiting “free sparsity” to prune non-salient channels (0-bit allocation), GRINQH reallocates the saved budget to high-precision outliers. Without this strategic reallocation, the model lacks the bit-depth necessary to preserve critical activation saliency. Conversely, in moderate bit-regimes (3.7 ≤ b ≤ 4.3), the budget is inherently sufficient for outlier protection. In these cases, exceeding the “free sparsity” threshold (∼ 20%) can degrade performance by omitting necessary weight information. Thus, GRINQH’s primary strength lies in its ability to navigate these regimes dynamically, operating within the free sparsity window by default while strategically exceeding it when bit-constraints become severe. Search Space Selection and Proxy Metrics To identify optimal fractions pi , we utilize calibration perplexity (PPL) as a proxy metric M. Fig. 7 shows that calibration PPL correlates strongly with downstream Wikitext2 PPL, which in turn serves as a reliable indicator of task-specific accuracy (see Section A.2). Using calibration PPL as a signal is computationally efficient, as it is obtained by the existing calibration pipeline. A representative sweep for Llama3-1B at a 4-bit target (Fig. 8) suggests that certain sub-spaces are significantly more performant. For instance, high-performing configurations often fall within the bounds: p0 < 0.17, p1 < 0.25, p2 > 0.3, 0.2 < p3 < 0.25 and p4 < 0.13. While these specific bounds shift with the target b∗ , they provide a useful prior for narrowing the search space. 16
-10 -20
2.0 ≤ #Bits < 3.0
-30 10
10
20
30
3.7 ≤ #Bits < 4.3
0 -10 10
20
30
Sparsity (0-bit) Fraction [%]
(C)
Ideal GRINQH Max 8bit GRINQH Max 6bit
7 6
Latency [Normalized to Marlin-4-bit]
GSM8K – Difference to Mean [%]
0
Measured Effective Number of Bits
(B) 8
(A) 10
5 4 3 2 2
4
6
8
Desired Effective Number of Bits
Marlin-4-bit Marlin-8-bit GRINQH TorchAO
1.8 1.6 1.4 1.2 1.0
GeForce RTX 5070 GeForce RTX 4090 RTX PRO 6000
0.8
2
4
6
8
Effective Number of Bits
Figure 6: (A) Sensitivity analysis of the 0-bit (sparsity) fraction (p0 ) for specific bit-width windows (Top: 2.0–3.0 bits; Bottom: 3.7–4.3 bits). Data points indicate the relative GSM8K performance change across varying sparsity levels with respect to the average performance in that window. Results show that high sparsity benefits low-bit regimes but penalizes higher bit regimes, suggesting that optimal p0 configuration is bit width dependent. (B) Comparison between the desired effective bit width and the measured effective bits. The GRINQH data points are obtained over all benchmark tasks and various models (bmax ∈ {6, 8}). The close adherence to the identity mapping (black line) confirms that the calibrated thresholds accurately achieve target bit-rates across diverse model distributions. (C) Kernel performance comparison on NVIDIA GPUs. Runtimes are measured in an isolated kernel setup over a range [2, 8] of target effective bit widths. Baselines use their respective state-of-the-art native kernels, which the GRINQH trend line closely rivals around the fixed-width points. For each device, latency numbers are normalized with respect to the best performing 4-bit MARLIN kernel run on the same device, measuring respectively [0.072, 0.057, 0.034]ms on the RTX5070/RTX4090/RTX PRO 6000. Recommended Configuration Workflow Based on these findings, we recommend the following four-step procedure to obtain a suitable hyperparameter instantiation: 1. Initialize: Define the target b∗ and a small tolerance ϵ (e.g., 0.05). 2. Sample: Use a constrained random sampler (e.g., based on a Dirichlet distribution) to generate ∼ 15 candidate vectors P that satisfy the simplex constraints and align with the sparsity regimes identified above. 3. Calibrate: Perform a single calibration forward pass for each P to determine the layer-wise thresholds and the resulting calibration PPL. 4. Select: Deploy the configuration that yields the lowest calibration PPL. This empirical approach effectively identifies high-performance distributions without the need for an exhaustive search. Future work may further automate this process through the development of learnable classifiers or direct threshold optimization.
17
Correlation: Calibration PPL vs. Wikitext2 PPL 175
9.6
150
Wikitext2 PPL
9.5
125
9.4
Trial Number
9.7
100
9.3
75 9.2 50 9.1
25
9.0 5.70
5.75
5.80
Calibration PPL
5.85
5.90
Figure 7: Correlation between calibration and downstream perplexity. Each data point represents a unique precision distribution configuration P sampled during our hyperparameter sweep. Calibration PPL is computed on The Pile (Uncopyrighted), while downstream PPL is evaluated on Wikitext-2. The strong linear correlation validates calibration PPL as a reliable and computationally efficient proxy metric for performance optimization during threshold selection.
B
Appendix: Extended Comparative Analysis
This section provides additional results comparing GRINQH with established state-of-the-art quantization methods (Section B.1) and existing multi-precision approaches (Section B.2). B.1
Extended Accuracy Benchmarks
B.1.1
Extended Pareto Frontier Plots
Figure 9 presents the complementary Pareto frontier analysis with model families reversed: the Qwen3 family evaluated on WikiText-2 (A) and the Llama3 family on GSM8K (B). These results demonstrate that GRINQH consistently redefines the Pareto frontier across model families, scales, and benchmarks. Notably, at an effective bit width of 3 bits, all GRINQH configurations (regardless of hyperparameter settings) outperform all static-precision SOTA counterparts on both WikiText-2 and GSM8K across all investigated peak memory settings (bmax ∈ {4, 6, 8}). For the decoding-heavy GSM8K CoT benchmark, where GRINQH’s mixed-precision logic is applied exclusively to the decoding stage (see Figure 9B, right panel), BF16 baseline performance is effectively recovered even at 3 effective bits for the larger Llama3 models (3B and 8B). The resulting frontiers extend significantly further along the baseline and exhibit higher density compared to runs where mixed precision is applied to both inference phases (Figure 9B, middle panel). This further underlines the benefits of stage-aware optimization, specifically treating the prefill and decoding stages as distinct computational regimes. B.1.2
Extended Table Model size comparison
Tab. 4 and Tab. 5 provide a comprehensive comparison across model scales ranging from 0.6B to 8B for the Qwen3 and Llama3 families at effective bit widths of 4, 3, and 2 bits. In the 2-bit regime, we report performance exclusively for GRINQH, as the perplexity of baseline SOTA methods diverges significantly at this level, and their performance on multiple-choice benchmarks falls to chance-level accuracy. Across all evaluated model sizes and families, GRINQH demonstrates superior performance 18
precision level 0 bit
precision level 2 bit
precision level 4 bit
Calibration PPL
8.3 8.2 8.1 8.0 7.9
0.05
0.10
0.15
0.20
0.25
bit-width fraction
0.30
0.10
0.15
0.20
precision level 6 bit
0.25
0.30
0.35
bit-width fraction
0.40
0.45
0.50
0.0
0.1
0.2
bit-width fraction
0.3
0.4
precision level 8 bit
Calibration PPL
8.3 8.2 8.1 8.0 7.9
0.15
0.20
0.25
0.30
bit-width fraction
3.92
0.35
0.075 0.100 0.125 0.150 0.175 0.200 0.225 0.250 0.275
bit-width fraction
3.94
3.96
3.98
Average Bit Width
4.00
4.02
4.04
4.06
Figure 8: Sensitivity of calibration perplexity to precision fractions pi . Each panel illustrates the impact of a specific bit-width allocation on Llama3-1B performance, constrained to a target effective bit width b∗ = 4.0 ± 0.1 (bmax = 8). We observe divergent scaling behaviors across the precision levels: boundary bit widths (b0 , b1 , and b4 ) show a positive correlation with perplexity, suggesting that excessive allocation to extremes, degrades performance. In contrast, the 4-bit fraction (b2 ) exhibits a strong inverse correlation. The 6-bit fraction (b3 ) displays non-monotonic behavior, indicating a localized optimal range. These empirical trends provide the basis for the reduced hyperparameter search space. over all baselines, consistently achieving lower perplexity on WikiText-2 and LAMBADA (LMB), as well as higher average accuracy across all evaluated bit widths. B.2
Comparisons to Multi-Precision Methods
Existing any-precision and mixed-precision frameworks generally target three distinct paradigms: (i) improved nested or multi-precision weight representations (e.g., any-precision LLM [31], MatQuant [28], MatGPTQ [21]), (ii) static mixed-precision allocation at the layer or block level (e.g., Mix’n’Match [12, 28, 21], SliM-LLM [19]), or (iii) coarse-grained dynamic scheduling (e.g., PMPD [6], where precision is token-dependent). GRINQH is complementary to paradigm (i), as its quantization-agnostic framework can directly benefit from more sophisticated weight-quantization methods (see Section E.2 for more details). Moreover, in contrast to (ii) and (iii), GRINQH performs activation-dependent, input-channel-wise precision allocation within each decoding step. This finer granularity is a key advantage, as it enables the system to adapt dynamically to token-specific outlier shifts within each vector-matrix multiplication (VMM), rather than relying on static saliency or coarse-grained decisions. Some methods, however, combine aspects of (i) and (ii). We therefore divide the methods into three categories for a more detailed comparison: Representation-Focused and Complementary Methods. Any-precision LLM focuses on improving nested multi-precision weight construction respectively by reformulating QAT for improved truncation robustness and iterative K-means based incremental upscaling. However, both methods do not provide a runtime mixed-precision scheduling logic. Consequently, these methods are complementary to GRINQH; while they improve the underlying base weights, GRINQH provides the dynamic allocation and efficient loading mechanisms required for inference. Tab. 1 and Section E.2 19
GRINQH Max 8bit GRINQH Max 6bit GRINQH Max 4bit GPTQ AWQ RTN
60
Wikitext – Word Perplexity
50 40
30
Qwen3-0.6B Qwen3-1.7B
20
Qwen3-4B Qwen3-8B 2
3 4 5 6 Effective Number of Bits
(B) GRINQH Max 4-bits 80 GSM8K – Exact Match Accuracy [%]
GRINQH Max 6/8-bits
(A)
GRINQH Max 6/8-bits
GRINQH Max 6/8-bits Decode Only
Llama-3.1-8B Llama-3.1-8B Llama-3.2-3B Llama-3.2-3B
60
Llama-3.2-1B Llama-3.2-1B 40
20
0 2
3
4 2
3 4 5 6 Effective Number of Bits
2
3
4
5
6
Figure 9: GRINQH redefines the quantization Pareto frontier across model families and scales. Dashed lines indicate BF16 baselines. GRINQH data points represent a sweep of precision distributions compared against iso-bit symmetric RTN, GPTQ, and AWQ baselines. (A) WikiText-2 perplexity vs. effective bit width for the Qwen3 family using bmax ∈ {6, 8}. GRINQH outperforms SOTA counterparts, maintaining representational stability deep into the 2–3 bit regime. (B) GSM8K CoT accuracy vs. effective bit width for the Llama3 family. (Left panel) GRINQH (bmax = 4) sweep on GPTQ baseline. GRINQH matches the GPTQ 4-bit baseline and dominates specialized 3-bit RTN, AWQ, and GPTQ baselines. (Middle panel) GRINQH with bmax ∈ {6, 8} on RTN baseline. Our framework effectively recovers BF16 performance for 3B and 8B model scales in low-bit regime. (Right panel) Decoupled inference strategy ( bmax ∈ {6, 8} on RTN baseline) where dynamic precision loading is enabled exclusively for the memory-bound decoding stage. This real-case scenario boosts accuracy even further, particularly for smaller models, demonstrating the significant benefit of stage-aware optimization. highlight the complementary nature by showing the performance increase of GRINQH when using any-precision LLM weights compared to uniform quantization techniques such as RTN and GPTQ. Nested Methods with Static Coarse-Mixed Precision Allocation. MatQuant [28] and MatGPTQ [21] optimize the base nested representation and employ Mix’n’Match-style [12] layer-wise static bit allocation. GRINQH remains complementary to these approaches by instead optimizing runtime channel-wise allocation (see Section E.2 for GRINQH based on MatGPTQ weights). Notably, MatQuant-style mixed-precision inference does not outperform optimized fixed-precision baselines in [28] (Fig. 1(b) and 2)), whereas GRINQH achieves significant gains over fixed-precision baselines (see Tab. 1). Tab. 6 shows that GRINQH outperforms both vanilla MatGPTQ and its optimized mixed-precision variants (MatGPTQ-EP-Mix’n’Match) across the full 2–4 bit range, with the gap widening significantly at 3 bits and below. For Llama-3.1-8B-Instruct at an effective bit width of 4 bits, GRINQH (73.79%) reduces accuracy degradation relative to the BF16 baseline (74.00%) by 6.3× compared to MatGPTQ (72.67%). At an effective bit width of 3 bits, this reduction in degradation is 16.9× (73.68% vs. 68.58%). Alternative Mixed-Precision Scheduling. SliM-LLM [19] utilizes static per-block precision allocation based on saliency, which precludes adaptation to runtime outlier shifts during generation. Tab. 1 shows that GRINQH substantially outperforms SliM-LLM, especially in the 3-bit (73.69% vs. 66.35% on Llama-3.1-8B-Instruct) and 2-bit (69.06% vs. 19.59% on Llama-3.1-8B-Instruct) regime. This confirms the advantage of fine-grained dynamic precision allocation during runtime. PMPD [6] is the most closely related prior work but utilizes coarser per-token dynamic scheduling. GRINQH’s input-channel-wise dynamic bit allocation is significantly more fine-grained and achieves substantially stronger results (e.g., 81.7% vs. 68.7% on GSM8K-0shot at 3 effective bits for Llama-3.1-8B-Instruct). We provide a detailed breakdown of these comparisons in Fig. 10.
20
Figure 10: Comparison between GRINQH’s fine-grained dynamic input-channel-wise bit allocation and PMPD’s per-token bit allocation (decreasing precision at later token positions), as well as PMPD with static fixed-precision bit allocation during decoding. PMPD datapoints originate from a sweep over various configurations of 8/6/4/3/2-bit allocations during decoding and were generated using the original PMPD codebase. The task is GSM8K 0-shot. The model is Llama3.1-8B-Instruct. All datapoints use 8-bit prefill.
C
Appendix: Kernel Implementation and Profiling
This section details the implementation of our Triton GEMV kernel, focusing on how it leverages the bit-planar layouts for sparse loading and bit-packing (Section C.1). We then validate these design choices via roofline analysis (Section C.2). Furthermore, we provide a version of Table 2 that includes standard deviations for all measurements: Tab. 7. C.1
Implementation Details
Sparse bit-loading: To optimize memory access for vector-matrix multiplication, we store the original weight matrix in a transposed (reduction-dimension major) layout. This ensures that for any element of the input vector x, the corresponding weights for all output channels are stored row-contiguously. The output dimension Nout is partitioned into bmax/2 precision zones, each storing a specific 2-bit significance slice of the quantized weights. Each zone spans 2Nout/bmax columns and stores all 2-bit segments of the weight values for a given precision level. Loading weights for higher precision, therefore, requires incremental loads offset from each zone, allowing for sparse loads. Zone sizing is static, since a bmax below 8-bit simply stops iterating at bmax/2. Bit-packing: We pack bit-segments of the same significance and of neighbouring weights within an input channel into uint32 format to maximize register occupancy and memory bandwidth utilization. This wrapper holds four 8-bit bytes, with each byte containing four spatially grouped 2-bit chunks. At runtime, this layout allows the GEMV kernel to load the equivalent of 16 partial weights of a single significance in a single 32-bit instruction. It also cuts the total amount of memory requests required by a factor of four. Grouping: To maintain writing locality, we pre-group weights into a 16-block layout. This mechanic aligns the internal structure of the uint32 packets with their logical destination in the output vector y. By offloading this structural realignment to a pre-processing operation, the GEMV kernel can utilize bitwise OR and SHIFT operations to reconstruct weights, without any runtime data-movement overhead. C.2
Roofline Analysis
Figure 11 utilizes a roofline analysis to visualize the memory-bound regime of our kernel. Reducing the effective bit width decreases memory transfers, lowering total runtime and increasing both arithmetic intensity and compute throughput. Because the analysis tracks only floating-point operations, it does not capture integer math such as shifts and masking. This static overhead weighs on the effective 2-bit configuration, imposing a soft cap on latency-hidden compute utilization.
21
FP32 roofline
Extrapolated roof
Effective bits:
Performance (FLOP / s)
FP32 Roofline · RTX4090 1014
2
3
4
FP32 Roofline · RTX5070
72.3 TFLOP/s 27.7 TFLOP/s
1013 1012
100
101
102
Arithmetic Intensity (FLOP / Byte)
103
100
101
102
Arithmetic Intensity (FLOP / Byte)
103
Figure 11: FP32 throughput over DRAM bytes loaded for different effective bit widths on consumergrade devices, RTX4090 and RTX5070. The kernel was profiled with NVIDIA NSight Compute[29] in isolation using a standalone weight matrix of W ∈ R16,384×4,096 , a bmax = 8 configuration for a target bit width of 4 and uniformly distributed thresholds. Identical kernel hyperparameters were used, matching those of our end-to-end result runs for the same target bit width. The maximum throughput values are affected by changing clock rates during execution, but are close to the base rate for both GPUs (73.24 TFLOP/s and 27.46 TFLOP/s respectively). All profiled kernels operate in the memory-bound regime.
D
Appendix: Calibration and Data Sensitivity
This section provides a detailed analysis of calibration resource requirements (Section D.1), the sensitivity of the method to calibration data composition (Section D.2), and the fidelity of the calibration mechanism across downstream tasks (Section D.3). To ensure a controlled and fair comparison with SOTA baselines, we use a calibration setup (128 samples, 2048 context length) that matches the standard configuration of the methods we compare against, rather than optimizing for the best samples for GRINQH.
D.1
Resource Consumption during Calibration
Table 8 shows that GRINQH’s calibration latency is significantly lower than data-dependent quantization methods like GPTQ and AWQ. For the Llama-3-8B-Instruct model, GRINQH achieves a ≈ 15.7× speedup over AWQ and ≈ 8.5× over GPTQ. Importantly, GRINQH is quantization-agnostic; it calibrates optimal thresholds for a target precision distribution on the quantized model, while the prior quantization step is independent of the threshold calibration itself. In our experiments, we mostly use Round-to-Nearest (RTN) for the quantization, which adds negligible overhead to the total setup time. Furthermore, since the perplexity on the calibration set is obtained as a byproduct of the calibration run, it can be utilized for zero-cost hyperparameter selection (see Section A.3). This allows for 10–15 iterative optimization runs while remaining competitive with the time required for a single AWQ calibration. 22
Finally, we note that our current calibration pipeline is entirely unoptimized, leaving significant room for further speedups through, e.g., sample-level parallelization. Table 8: Preparation speed and memory benchmarks for Llama-3.1-8B on a single NVIDIA RTX 6000 Blackwell. RTN, GPTQ, and AWQ are implemented in llm-compressor; GRINQH benchmarks only the threshold calibration phase within vLLM, where a single precision distribution (single datapoint) determines the thresholds. GRINQH enables near-instantaneous model preparation with significant speedups over SOTA baselines. Method
Prep Time (s)
Peak VRAM (GiB)
Relative Time
RTN 4.19 ± 0.08 GPTQ 363.42 ± 1.38 AWQ 673.14 ± 5.98 GRINQH (single datapoint, no quant) 42.82 ± 0.39
16.44 ± 0.08 16.37 ± 0.00 21.82 ± 0.00 40.94 ± 0.00
1.00x 86.83x 160.83x 10.23x
Table 9 demonstrates the scalability of GRINQH across the Llama3 and Qwen3 families. Both preparation time and peak VRAM scale sub-linearly with model size, confirming that the method remains viable for large-scale calibration without prohibitive hardware requirements. D.2
Representativeness of Calibration Set
We evaluate the robustness of the calibration process across varying sample sizes and subsets of the Pile (uncopyrighted) dataset. By sweeping calibration set sizes from 16 to 1024 samples, we observe that at a size of 128 samples (our chosen setup for baseline parity), the mean absolute relative error (MARE) of the calibrated thresholds compared to the 1024-sample reference drops below 1% (Fig. 12A). To assess subset sensitivity, we repeated the calibration across five random sets of 128 samples. The results demonstrate minimal variance, with threshold MARE consistently remaining below 1% (see crosses in Fig. 12A). The negligible variation across subsets (Fig. 12B) confirms that the method is not overly sensitive to the specific choice of calibration data.
Figure 12: Robustness of calibrated thresholds to sample size and subset variation for Llama-3.1-8BInstruct. Left (A): MARE of thresholds relative to a 1024-sample baseline. Crosses denote per-seed variation for N = 128. Right (B): MARE across random subsets relative to the reference. The method remains robust across both sample size and subset selection. D.3
Calibration Fidelity.
We evaluate calibration fidelity by comparing target bit widths against empirically measured effective bit widths across downstream tasks. Fig. 6B illustrates that the measured bit widths across all models and tasks evaluated for Fig. 3 and Fig. 9 (plus common-sense reasoning benchmarks) for 23
bmax ∈ {6, 8} totaling 3632 benchmark runs closely track the ideal identity mapping, demonstrating that thresholds optimized on the calibration set generalize effectively to real-world generation. This tight alignment across diverse models and target precisions confirms that our distribution-aware mechanism successfully regulates memory traffic as intended without the need for task-specific tuning or training.
E
Appendix: Weight Representation and Approximation Analysis
This section examines the technical design choices governing GRINQH’s weight representation and its approximation logic. Specifically, we provide a detailed comparison between hardware-efficient bit-truncation (slicing) and standard rounding (Section E.1), demonstrating how our framework compensates for approximation shifts. Furthermore, we analyze the compatibility and performance trade-offs of integrating various nested weight formats into the GRINQH pipeline (Section E.2). E.1
Truncation vs. Rounding
While standard uniform quantization typically employs rounding to minimize approximation error, GRINQH utilizes bit-truncation (slicing). True rounding would require loading an additional bit of precision to determine the rounding direction, negating the memory traffic savings achieved by reducing bit widths. Instead, GRINQH employs truncation and compensates for the resulting mean shift by applying a pre-computed truncation bias (see Section 3.3 for more details). As shown in Tab. 10, substituting rounding with truncation in the GRINQH framework results in negligible variations in average accuracy. This trend is further illustrated across varying effective bit widths in Fig. 13, where truncation and rounding curves largely overlap. While rounding remains slightly superior in the extreme 2-bit regime, the two methods achieve nearly identical average performance. Notably, this stability is unique to the mixed-precision setting. In fixed-precision configurations, truncation significantly degrades performance compared to rounding, to the point where data points fall outside the viable scale of our plots. This suggests that GRINQH’s input-adaptive precision allocation effectively mitigates the local errors introduced by truncation, allowing for hardwarefriendly memory loads without sacrificing model fidelity. Table 10: Performance difference between slicing from 8-bit versus rounding weights to lower precision (∆ = slice − round), averaged over 40 GRINQH configurations (different precision distributions) covering the 2.1–5.7 effective-bit range, across the Llama3 and Qwen3 families with RTN as the 8-bit baseline. Slicing from 8-bit achieves comparable performance to rounding, with differences remaining negligible on average across all tested models. ∆ (= slice − round) Model
Wikitext PPL ↓
Lambada PPL ↓
GSM8K Acc. (%) ↑
Average Acc. (%) ↑
Eff. Bits
Llama-3.2-1B-Instruct-RTN (Max 8-bit) Llama-3.2-3B-Instruct-RTN (Max 8-bit) Llama-3.1-8B-Instruct-RTN (Max 8-bit) Qwen3-0.6B-RTN (Max 8-bit) Qwen3-1.7B-RTN (Max 8-bit) Qwen3-4B-RTN (Max 8-bit) Qwen3-8B-RTN (Max 8-bit)
0.0196 0.1392 0.0531 1.2720 1.1128 0.1184 0.1313
0.6904 0.0060 -0.0028 5.9783 0.6365 -0.0879 0.0429
+0.24 -0.36 -0.20 -1.15 -0.99 -1.00 +0.32
-0.06 -0.32 +0.08 +0.12 -0.58 -0.03 -0.07
0.0028 0.0070 0.0104 0.0048 0.0223 0.0074 0.0117
E.2
Evaluation of Nested Weight Formats
Beyond uniform quantization, GRINQH is compatible with nested-weight representations such as any-precision LLM [31]. While MatGPTQ [21] also employs nested weights, its loading logic requires fetching an additional bit beyond the target bit width, making it incompatible with GRINQH’s objective of minimized memory traffic. 24
Wikitext
GSM8K
9.7 9.1 8.5
2
3
4
5
6
Average Accuracy
0.84 0.82 0.80 0.78 0.76
0.74
Accuracy
10
Accuracy
Perplexity
11
0.73 0.72 0.71
2
3
4
5
6
2
3
4
5
6
Effective Number Of Bits Effective Number Of Bits Effective Number Of Bits GRINQH-RTN (slice) GRINQH-AnyPrecisionLLM GRINQH-RTN (round) (single precision) GRINQH-RTN (round) GRINQH-MatGPTQ (single precision) GRINQH-AnyPrecisionLLM (single precision) GRINQH-MatGPTQ
Figure 13: Quantization method comparison across key metrics on Llama3 8B. Performance metrics (perplexity on wikitext, exact match accuracy on GSM8K, and average task accuracy) versus effective number of bits for four quantization approaches: GRINQH-RTN (slice), GRINQH-RTN (round), GRINQH-MatGPTQ, and GRINQH-AnyPrecisionLLM. Curves represent smoothed trends via Gaussian-kernel estimation over multiple precision distribution configurations. Single precision distributions are shown as diamonds; mixed-precision distributions are as circles. Fig. 13 shows that MatGPTQ, which utilizes a GPTQ-based nested structure, exhibits the lowest performance in our evaluation. This highlights a fundamental representation trade-off: nested formats often sacrifice high-bit fidelity to improve low-bit quantization performance. This characteristic undermines GRINQH’s outlier protection strategy, which relies on allocating maximum precision to weights associated with high-magnitude activations to minimize or eliminate quantization error in critical channels. If the underlying nested format introduces error even at its highest precision level, the benefit of dynamic allocation is diminished. In contrast, any-precision LLM demonstrates the potential of nested structures by outperforming baselines in the sub-4-bit regime while matching the performance of GRINQH-RTN (with slicing) at higher effective bit widths. This suggests that while nested weights can enhance efficiency, their effectiveness within a dynamic framework depends heavily on their ability to maintain high-fidelity representation for salient weights.
25
Qwen3-8B
Qwen3-4B
Qwen3-1.7B
Qwen3-0.6B
Table 4: Benchmark results for the Qwen3 model family. Eff. Bits
Method
WikiText2 ppl ↓
LMB ppl ↓
LMB acc ↑
ARC-C acc ↑
HellaSwag acc ↑
BoolQ acc ↑
MMLU acc ↑
GSM8K acc ↑
Average acc ↑
16 4 4 4 4.04 3.90
BF16-Baseline
26.194
24.705
40.91
34.47
47.39
64.16
39.90
41.93
44.79
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN
30.424 30.404 37.537 27.123 28.132
37.620 42.145 71.377 26.535 27.138
36.35 33.92 26.06 39.71 39.51
30.29 29.78 29.52 34.98 32.59
44.89 44.23 43.60 46.80 46.03
61.93 50.40 66.73 61.71 58.65
33.74 37.43 31.24 39.41 40.81
29.80 26.91 24.56 40.03 36.77
39.50 37.11 36.95 43.77 42.39
3 3 3 3.01 2.94 3.03 3.06
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
59.460 94.550 905.450 29.879 29.995 39.235 31.575
431.130 893.252 19216.238 32.596 31.322 76.691 40.674
14.52 8.21 2.19 37.88 37.01 25.42 35.55
26.96 24.06 22.78 32.51 31.57 29.86 28.50
37.33 34.23 29.37 45.37 44.73 42.86 44.34
52.51 59.57 42.32 67.03 55.44 64.04 61.10
24.28 23.53 24.28 38.99 36.38 30.17 32.52
1.52 0.83 0.00 32.60 36.01 22.44 26.46
26.19 25.07 20.16 42.40 40.19 35.80 38.08
16
BF16-Baseline
20.974
12.182
51.47
43.00
60.41
77.49
55.59
73.24
60.20
4 4 4 4.05 3.96
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN
22.162 21.748 25.778 20.942 21.021
13.456 16.740 23.280 12.773 12.420
50.57 45.92 39.90 50.94 51.87
39.76 39.68 39.85 43.52 42.32
58.64 58.61 57.91 60.17 59.44
77.89 72.72 76.09 77.13 77.37
51.35 52.31 52.05 55.08 55.61
61.33 62.09 54.97 71.87 68.99
56.59 55.22 53.46 59.78 59.27
3 3 3 3.02 3.07 2.90 3.05
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
33.027 40.370 88.595 22.415 21.739 25.544 22.415
57.929 229.731 1867.752 13.827 13.160 24.993 14.031
31.69 20.76 8.34 48.61 50.42 39.08 50.11
30.03 30.63 27.30 42.06 42.66 39.08 39.68
49.40 48.98 40.78 58.52 58.67 57.11 57.72
68.41 66.51 55.38 73.67 76.94 76.61 77.22
37.55 31.16 29.52 53.80 54.47 51.43 50.90
11.83 2.73 1.44 64.97 64.29 50.95 57.24
38.15 33.46 27.13 56.94 57.91 52.38 55.48
16
BF16-Baseline
16.427
7.308
60.26
53.84
68.48
84.95
68.37
83.40
69.88
4 4 4 4.05 4.08
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN
17.169 17.507 17.459 16.413 16.486
8.090 9.941 10.068 7.093 7.216
58.16 55.97 55.56 60.76 60.04
53.16 51.45 50.94 52.99 52.05
67.60 66.63 66.63 68.38 67.99
83.58 83.76 83.91 84.89 85.41
66.66 65.67 66.14 68.26 68.01
79.15 79.38 77.03 84.15 83.55
68.05 67.14 66.70 69.91 69.51
3 3 3 3.00 3.06 3.03 3.04
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
21.560 29.480 44.715 16.828 16.807 17.514 17.184
16.553 129.466 962.708 7.566 7.011 9.913 7.871
44.42 24.26 10.01 59.77 60.49 55.07 58.32
41.30 40.19 35.41 51.11 50.94 49.23 52.05
60.52 56.84 50.62 67.63 66.89 66.09 67.25
80.92 74.31 53.24 84.53 84.65 83.18 83.09
56.40 50.97 40.41 67.37 67.28 65.82 66.07
46.17 22.67 4.40 84.38 81.88 74.98 78.77
54.95 44.87 32.35 69.13 68.69 65.73 67.59
2.12 2.11 2.12
GRINQH-8b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
20.032 18.878 19.107
10.956 12.405 9.546
53.31 51.10 54.69
46.93 35.32 48.89
61.75 63.03 63.47
79.17 81.53 80.76
61.18 61.80 63.19
58.38 67.48 73.24
60.12 60.04 64.04
16
BF16-Baseline
12.200
4.594
65.11
56.31
74.82
86.64
72.90
88.32
74.02
4 4 4 4 4 4 4 4 3.97 3.95 3.94
GPTQ AWQ RTN QuaRot-GPTQ AutoRound NVFP4-G16 AnyPrecisionLLM SliM-LLM GRINQH-8b-RTN GRINQH-8b-AnyPrec GRINQH-6b-RTN
12.680 12.702 12.887 12.581 13.299 12.382 12.820 12.611 12.291 12.234 12.265
5.169 5.091 5.400 4.810 5.942 5.066 4.695 5.201 4.581 4.439 4.580
63.13 63.32 62.00 64.54 60.28 63.75 64.74 61.60 65.17 65.11 65.09
54.35 54.61 51.88 56.48 54.61 57.76 55.80 53.84 55.63 56.40 56.14
74.42 73.48 73.99 73.76 73.78 74.39 73.75 74.39 74.60 74.77 74.74
86.79 85.78 85.63 85.87 86.02 86.27 86.70 86.39 86.48 86.67 86.85
71.58 71.00 71.09 71.00 71.96 71.83 71.51 71.37 72.75 72.82 72.85
82.79 82.41 74.75 86.28 84.08 86.13 87.64 87.41 87.57 87.72 87.19
72.18 71.77 69.89 72.99 71.79 73.36 73.36 72.50 73.70 73.91 73.81
3 3 3 3 3 3 3 3.07 3.06 3.03 3.03 3.06
GPTQ AWQ RTN QuaRot-GPTQ AutoRound AnyPrecisionLLM SliM-LLM GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ GRINQH-8b-AnyPrec
14.886 16.327 22.844 14.282 15.079 15.231 14.884 12.610 12.581 13.020 12.841 12.403
6.957 14.239 41.288 7.297 7.307 6.499 5.582 4.512 4.527 5.304 5.101 4.348
56.26 43.68 28.26 55.13 54.49 59.62 60.35 65.42 65.65 62.41 63.21 65.48
47.01 43.34 38.82 48.81 49.57 52.56 46.93 55.38 55.63 53.50 53.16 56.14
68.87 66.05 61.39 69.03 69.99 69.96 70.45 73.63 73.65 73.81 73.90 74.06
83.76 80.67 70.95 83.82 83.91 86.09 84.19 85.87 86.61 85.75 86.57 86.76
64.31 61.70 53.79 64.53 66.96 68.15 66.27 71.66 71.71 70.80 71.12 72.61
67.78 59.67 15.54 68.84 67.78 79.30 77.63 88.02 85.60 79.38 82.56 87.41
64.67 59.19 44.79 65.03 65.45 69.28 67.64 73.33 73.14 70.94 71.75 73.74
2 2 2.25 2.22 2.11 2.11
AnyPrecisionLLM SliM-LLM GRINQH-8b-RTN GRINQH-8b-AnyPrec GRINQH-4b-RTN GRINQH-4b-GPTQ
109.867 100.974 13.356 12.947 14.365 14.086
669.490 212.912 5.117 4.209 5.885 5.969
14.24 25.05 63.50 66.14 61.11 61.77
34.22 26.19 53.41 56.14 49.74 52.22
47.28 40.83 72.35 72.53 70.39 70.53
67.86 63.88 86.21 86.70 84.53 85.20
38.61 30.57 70.11 71.10 67.04 68.60
0.61 2.12 84.00 87.04 75.82 77.86
33.80 31.44 71.60 73.27 68.11 69.36
26
Table 5: Benchmark results for the Llama3 model family.
Llama-3.1-8B-Instruct
Llama-3.2-3B-Instruct
Llama-3.2-1B-Instruct
Eff. Bits
Method
WikiText2 ppl ↓
LMB ppl ↓
LMB acc ↑
ARC-C acc ↑
HellaSwag acc ↑
BoolQ acc ↑
MMLU acc ↑
GSM8K acc ↑
Average acc ↑
16
BF16-Baseline
15.813
6.580
61.07
37.71
60.85
69.20
45.95
45.72
53.42
4 4 4 4.06 3.94
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN
18.685 19.337 18.958 16.333 16.443
8.854 8.499 10.183 6.744 6.932
55.48 56.05 53.21 60.10 59.89
37.88 37.63 34.98 36.35 37.80
59.40 58.98 56.39 59.94 59.24
69.02 66.57 66.39 69.60 69.27
45.16 44.74 41.79 46.40 45.54
37.91 32.60 23.43 41.55 37.23
50.81 49.43 46.03 52.32 51.49
3 3 3 3.04 2.97 3.04 2.95
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
32.059 62.497 61.012 17.372 17.766 19.430 18.641
33.060 61.082 393.156 7.852 8.923 10.797 11.009
35.51 27.25 12.61 56.86 54.65 52.42 52.14
31.74 30.38 26.62 37.03 35.92 34.56 36.60
52.08 49.28 39.28 58.26 56.79 55.66 55.76
63.79 56.15 60.80 68.38 68.04 65.84 66.88
35.84 29.13 25.95 44.38 42.94 40.99 41.50
4.55 1.97 0.15 34.12 28.81 21.76 24.64
37.25 32.36 27.57 49.84 47.86 45.20 46.25
2.12
GRINQH-8b-RTN
33.031
41.759
31.54
31.23
49.16
62.45
29.97
3.11
34.57
16
BF16-Baseline
12.276
4.821
67.18
46.25
70.44
78.41
60.29
78.24
66.80
4 4 4 3.92 3.89
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN
127.864 13.257 13.246 12.767 12.692
5.262 5.505 5.570 4.746 4.753
65.11 63.54 63.71 67.05 67.24
44.62 45.31 44.54 44.71 44.62
70.51 70.61 70.50 71.11 71.29
78.10 78.59 77.55 76.06 77.49
60.11 60.25 59.31 59.96 60.60
74.45 73.39 73.84 72.40 72.93
65.49 65.28 64.91 65.22 65.70
3 3 3 3.00 2.93 2.89 2.88
GPTQ AWQ RTN GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
6190.919 24.194 23.948 13.077 13.256 13.692 13.909
9.138 11.161 15.715 4.915 5.110 5.510 5.264
53.83 50.55 46.03 66.37 65.98 64.66 65.88
39.42 40.61 35.49 43.60 43.17 43.94 40.36
65.61 64.64 61.19 70.51 70.30 69.79 69.90
68.01 73.58 66.02 75.23 75.60 74.56 76.94
51.18 44.92 41.15 58.99 58.32 57.27 57.20
44.66 37.45 15.01 69.52 68.46 68.69 67.70
53.79 51.96 44.15 64.04 63.64 63.15 63.00
2.12
GRINQH-8b-RTN
18.303
8.427
57.56
38.05
65.67
63.88
49.32
42.84
52.89
16
BF16-Baseline
8.642
3.404
74.19
55.20
79.26
84.19
68.12
85.14
74.35
4 4 4 4 4 4 4 4 4.01 3.89 3.92
GPTQ AWQ RTN QuaRot-GPTQ AutoRound NVFP4-G16 AnyPrecisionLLM SliM-LLM GRINQH-8b-RTN GRINQH-8b-AnyPrec GRINQH-6b-RTN
9.409 9.525 9.665 9.377 9.615 9.272 9.444 9.354 8.971 8.945 9.020
3.644 3.911 3.903 3.554 3.894 3.408 3.714 3.463 3.377 3.358 3.409
72.85 70.02 70.11 72.73 70.46 73.86 69.63 72.19 73.78 73.18 73.26
54.52 51.37 50.77 52.90 54.18 53.84 55.20 52.99 54.95 54.86 54.18
78.59 78.63 78.62 78.39 78.42 78.75 78.49 78.45 79.27 79.18 78.87
84.19 84.59 84.65 85.20 84.28 84.13 85.14 84.56 85.02 85.29 85.41
66.16 66.69 64.65 66.04 65.28 66.36 66.69 66.07 67.88 68.07 67.43
80.82 80.59 79.83 81.88 80.89 82.49 81.88 82.64 85.14 84.53 84.31
72.85 71.98 71.44 72.86 72.25 73.24 72.84 72.82 74.34 74.19 73.91
3 3 3 3 3 3 3 2.98 2.96 3.02 3.02 3.04
GPTQ AWQ RTN QuaRot-GPTQ AutoRound AnyPrecisionLLM SliM-LLM GRINQH-8b-RTN GRINQH-8b-AnyPrec GRINQH-6b-RTN GRINQH-4b-RTN GRINQH-4b-GPTQ
12.461 14.363 19.133 12.798 11.812 12.217 12.010 9.304 9.220 9.435 9.802 9.593
5.857 8.039 7.427 5.768 5.239 4.832 4.853 3.485 3.368 3.402 3.799 3.563
60.88 54.82 60.10 63.07 64.43 63.19 65.53 73.03 72.52 73.01 70.68 73.06
44.28 45.39 46.25 47.70 49.15 50.60 50.00 54.44 55.12 53.24 51.02 53.50
74.08 72.46 68.09 73.24 75.61 75.32 73.65 78.50 78.41 77.88 77.88 78.24
81.83 78.62 73.76 80.70 81.41 82.72 82.35 85.20 84.86 84.31 84.77 84.13
58.18 50.52 49.05 58.13 57.83 59.52 58.09 66.50 67.02 65.62 64.02 65.23
60.35 34.50 6.60 57.54 49.58 56.18 68.46 81.50 84.23 82.41 79.68 79.91
63.27 56.05 50.64 63.40 63.00 64.59 66.35 73.19 73.69 72.75 71.34 72.34
2 2 2.11 2.01
SliM-LLM AnyPrecisionLLM GRINQH-8b-RTN GRINQH-8b-AnyPrec
535.131 1953.566 12.871 10.465
34205.477 11649.584 5.937 3.717
1.43 2.87 61.46 69.24
24.74 27.47 45.39 52.30
27.52 34.69 73.94 75.40
37.86 41.93 78.56 83.98
23.01 24.86 55.36 61.76
2.96 1.52 53.53 71.65
19.59 22.22 61.37 69.06
27
Table 6: Benchmark results comparing GRINQH and MatGPTQ on the Llama3.1-8B-Instruct and Qwen3-8B models. For MatGPTQ, results are presented for both the same precision across all layers (MatGPTQ) and the optimized mixed-precision Mix’n’Match version (MatGPTQ-EP-Mix’n’Match), where layers operate at varying precision. The MatGPTQ results are taken from the original paper (Tabs. 6, 9, 19 and 20). Notably, 4-bit results for Qwen3-8B are not included, as Mix’n’Match results are not reported in the original paper. Across both models and all bit widths, GRINQH’s dynamic input-channel-wise bit allocation consistently achieves superior average accuracy compared to both MatGPTQ’s static fixed- and mixed-precision variants.
Qwen3-8B
Llama-3.1-8B-Instruct
Eff. Bits
Method
ARC-C acc ↑
ARC-E acc ↑
HellaSwag acc ↑
PIQA [5] acc ↑
Winogrande [34] acc ↑
Average acc ↑
16
BF16-Baseline
55.89
79.80
79.52
81.34
73.48
74.00
4.00 4.00 3.96 3.881
MatGPTQ MatGPTQ-EP-Mix’n’Match GRINQH-8b-RTN GRINQH-6b-RTN
52.82 53.84 55.80 52.90
79.84 78.58 79.29 81.14
77.40 77.60 79.21 78.95
80.96 79.76 80.30 80.25
72.06 73.56 74.35 74.66
72.62 72.67 73.79 73.58
3.00 3.00 2.95 2.98 3.01
MatGPTQ MatGPTQ-EP-Mix’n’Match GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-GPTQ
46.25 47.53 55.03 53.58 54.27
70.92 71.46 79.67 78.49 78.45
72.92 73.53 78.68 77.75 78.36
76.44 77.86 80.25 79.54 80.47
71.27 72.53 73.01 73.16 72.53
67.56 68.58 73.68 72.50 72.82
2.08
GRINQH-8b-RTN
46.08
72.18
73.78
78.02
67.40
67.49
16
BF16-Baseline
56.57
80.93
74.94
77.69
67.56
71.54
3.00 3.00 3.06 3.05 3.02
MatGPTQ MatGPTQ-EP-Mix’n’Match GRINQH-8b-RTN GRINQH-6b-RTN GRINQH-4b-GPTQ
48.72 47.53 55.12 56.14 55.20
72.31 68.69 78.66 79.50 78.20
67.18 68.40 73.50 73.63 73.96
74.32 74.27 76.71 77.64 77.37
64.64 64.48 65.90 67.64 66.61
65.43 64.67 69.98 70.91 70.27
2.24 2.09
GRINQH-8b-RTN GRINQH-4b-GPTQ
52.99 51.71
78.45 76.94
72.21 70.94
77.31 76.77
67.72 66.93
69.74 68.66
Table 7: Tok/s performance over different effective bit widths and in:output ratios (prefill: (3000:1), decoding: (1:3000)) of tokens on random data. Relative performance is calculated against GPTQ 4-bit MARLIN. Kernel
Prefill Abs.
Prefill Rel.
Decoding Abs.
Decoding Rel.
Eff. 2-bit GRINQH (Max 8-bit) Eff. 3-bit GRINQH (Max 8-bit) Eff. 4-bit GRINQH (Max 8-bit)
8186.78 ± 17.92
0.80x
204.64 ± 0.74 182.18 ± 0.69 157.56 ± 0.06
1.28x 1.14x 0.98x
Eff. 4-bit GRINQH (Max 6-bit) Eff. 3-bit GRINQH (Max 4-bit)
8241.01 ± 28.23 8241.01 ± 28.23
0.81x 0.81x
165.10 ± 0.13 165.10 ± 0.13
1.03x 1.03x
GPTQ 4-bit Marlin RTN 8-bit Marlin
10170.38 ± 26.47 8927.07 ± 23.35
1.00x 0.88x
160.36 ± 0.19 100.21 ± 0.18
1.00x 0.62x
Table 9: Preparation speed and memory benchmarks for Llama3 and Qwen3 model families on a single NVIDIA RTX 6000 Blackwell. Benchmarks cover the threshold calibration phase within vLLM. Both preparation time and peak VRAM scale sub-linearly with model size, demonstrating the scalability of the calibration method. Model Size Prep Time (s) Relative Time Peak VRAM (GiB) Relative Peak VRAM
1B
Llama3 3B
8B
0.6B
1.7B
6.64 ± 0.02 1x 13.90 ± 0.00 1x
17.22 ± 0.26 2.59x 27.93 ± 0.00 2.01x
42.82 ± 0.39 6.45x 40.94 ± 0.00 2.95x
4.08 ± 0.12 1x 12.81 ± 0.00 1x
9.22 ± 0.12 2.26x 14.63 ± 0.00 1.14x
28
Qwen3
4B
8B
23.48 ± 0.25 5.75x 20.85 ± 0.00 1.63x
42.14 ± 0.77 10.32x 31.82 ± 0.00 2.48x