RoPE-Aware Bit Allocation for KV-Cache Quantization

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

RoPE-Aware Bit Allocation for KV-Cache Quantization

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

RoPE-Aware Bit Allocation for KV-Cache Quantization

Fengfeng Liang1

arXiv:2606.24033v1 [cs.LG] 23 Jun 2026

Yuechen Zhang2,3

Jiaya Jia1,∗

Hong Kong University of Science and Technology 2 The Chinese University of Hong Kong 3 MiMo, Xiaomi Corporation ∗ Corresponding author

Abstract Existing low-bit KV-cache quantizers typically treat each cached key as a flat vector. Under RoPE, however, the contribution of a cached key to a future attention logit decomposes into a position-dependent sum over two-dimensional frequency blocks. This makes key-cache quantization a block-wise bit-allocation problem: high-energy RoPE blocks are more sensitive to quantization error and should therefore receive more bits. We introduce Block-GTQ, a RoPE-aware bit allocator for key-cache quantization built on TurboQuant-MSE (TQ-MSE). For each layer and KV head, Block-GTQ computes a label-free energy score for each RoPE block and greedily allocates integer bit widths using marginal gains. Under matched K/V bit budgets, Block-GTQ better preserves RoPE query-key logits on a diverse ten-model diagnostic panel—at both 2 and 3 b/dim K-only, cutting per-layer MAE by 32–80% across models and winning all 367/367 layer comparisons at each budget against uniform TQ-MSE—and these fidelity gains translate to stronger downstream long-context retrieval, understanding, and reasoning. At K2V2 on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task NIAH average from 70.6 to 97.4, and the eight-task LongBench-EN average from 36.87 to 53.31, relative to the uniform-allocation TQ-MSE baseline. On AIME 2024/2025 with DeepSeekR1-Distill-Qwen-7B, and without relying on an fp16 recent-key buffer, Block-GTQ at K3V2 scores 51.7/37.5, close to fp16’s 54.2/37.9, whereas uniform-allocation TQ-MSE collapses to 0.0/0.0. We further implement a packed-cache serving path that avoids materializing an fp16 KV cache: on a single H800 GPU with Qwen2.53B-Instruct, the packed K3V3 path achieves 3.24× KV-cache compression with quality comparable to fp16, runs 1.34× faster than fp16 FlashAttention2 at 128K context, reduces peak memory from 56.31 GB to 19.85 GB, and remains feasible at 256K/512K where fp16 OOMs. Our code is available at https://github.com/ JIA-Lab-research/blockgtq.

Introduction

Long-context inference makes the KV cache the dominant sequence-dependent memory cost in autoregressive decoding. The cache stores one key and one value vector for every past token in every layer, and each decode step must access this growing state to attend over the context. For example, in a GQA-style 70B-class model with 80 layers, 8 KV heads, and 128-dimensional heads, an fp16 KV cache requires about 320 KiB per token, or roughly 40 GiB at a 128K-token context. This creates two coupled bottlenecks: capacity, because the resident cache must fit in memory, and bandwidth, because the attention kernel must stream the cached K/V at each decode step [27, 8, 7, 17, 28, 22]. KV-cache quantization mitigates this pressure by storing cached keys and values with fewer bits [24, 14, 13, 43, 41]. Most methods cast the problem as vector compression, choosing a quantization Preprint.

RoPE Block Structure

RoPE-Block energy Scores

𝑅! 𝑅"

…

𝑅! =

𝑅# 𝑞=

𝑞# 𝑞"

… 𝑞!

𝑘=

𝑘# 𝑘"

… 𝑘!

Block-GTQ Bit Allocation

High-energy block preserved

𝑞 $ 𝑅% 𝑘 = % 𝑞&$ 𝑅& 𝑘& &

Figure 1: RoPE-block allocation. (a) The RoPE attention logit q⊤ R∆ k decomposes into a sum over two-dimensional frequency blocks. (b) Per-block energy scores for one Qwen3-8B KV head (layer 10, head 4); scores are median-normalized for display and span orders of magnitude. (c) Under the same average bit width b̄ = 3 (dashed line), Block-GTQ reallocates bits from low-energy blocks to high-energy blocks instead of using a uniform 3-bit width. granularity over heads, channels, groups, or tokens so that dequantized vectors remain close to the originals. This view is natural for storage, but it does not capture how cached keys are used. A value error affects the post-softmax weighted sum, whereas a key error perturbs the pre-softmax logits seen by future queries and can change the attention distribution. For RoPE attention [30], this key-logit computation is block structured. Let ∆ be the relative position between a future query q ∈ Rdh and a cached key k ∈ Rdh . Up to the usual attention scaling, their logit is K∆ (q, k) = q⊤ R∆ k, where R∆ is block diagonal with 2 × 2 RoPE rotations. Hence the logit is a position-dependent sum of block terms q(i)⊤ R(∆θi )k(i) , for i = 1, . . . , dh /2, with θi the frequency of block i. A cached key is therefore not used through a flat-vector interface. Key-cache quantization should allocate precision across RoPE blocks according to their logit impact, rather than optimize a single flat-vector reconstruction objective over the whole key head. This block-wise view changes where bits should be spent. Uniform allocation within a key head is natural only if RoPE blocks have comparable influence on future logits. Empirically, block-energy profiles can be sharply uneven: a few frequency blocks carry most of the query-key signal, making future logits more sensitive to quantization error in those blocks than to comparable error elsewhere. RoPE-agnostic uniform allocation therefore spends the same precision on blocks with very different logit sensitivity, potentially over-protecting low-impact blocks and under-protecting high-impact ones. Figure 1 illustrates this allocation gap: one KV head from Qwen3-8B has a sharply non-uniform block-energy profile, and under the same average bit budget b̄ = 3, Block-GTQ shifts precision toward high-energy blocks instead of using the same bit width for every block. We propose Block-GTQ, a lightweight RoPE-aware allocator that spends key-cache precision where future logits are most sensitive. For each layer and KV head, Block-GTQ computes a label-free RoPEblock energy score from Q/K activations, combines it with the TurboQuant-MSE (TQ-MSE) [41] 4−b squared-error rate law, and greedily assigns integer bit widths under a fixed average-bit budget. Blocks assigned the same bit width are grouped and encoded by the original TQ-MSE local quantizer. Since values do not enter the RoPE key-logit computation, V is encoded with uniform-allocation TQ-MSE. All RoPE blocks are still stored; Block-GTQ changes only their bit widths. Our contributions are: 1. We formulate key-cache compression for RoPE models as a logit-preservation problem over two-dimensional frequency blocks, rather than a flat-vector reconstruction problem. 2. We derive a RoPE-block integer bit allocator that combines a label-free Q/K energy score with the TQ-MSE 4−b error law, and reuse the TQ-MSE encoder for same-bit-width block groups. 3. We validate the mechanism from RoPE-logit fidelity to downstream long-context retrieval, understanding, and reasoning tasks. On a diverse ten-model diagnostic panel, at both 2 and 3 b/dim 2

K-only, Block-GTQ cuts per-layer RoPE-logit MAE by 32–80% across models and wins all 367/367 layer comparisons at each budget against uniform TQ-MSE. At the K2V2 budget on Llama-3.1-8B-Instruct, it raises the six-task NIAH average from 70.6 to 97.4, and the eight-task LongBench-EN average from 36.87 to 53.31, relative to uniform-allocation TQ-MSE. On AIME 2024/2025 with DeepSeek-R1-Distill-Qwen-7B, and without relying on an fp16 recent-key buffer, Block-GTQ at K3V2 scores 51.7/37.5, close to fp16’s 54.2/37.9, whereas uniform-allocation TQ-MSE collapses to 0.0/0.0. 4. We implement a packed-cache serving path for compressed K/V codes and evaluate it with K3V3 Block-GTQ on Qwen2.5-3B-Instruct. Compared with fp16 FlashAttention2 using an uncompressed KV cache, at 128K tokens the packed path compresses the KV cache by 3.24×, reduces peak memory from 56.31 GB to 19.85 GB, and lowers single-request decode latency from 70.96 ms to 52.95 ms. At 256K and 512K tokens, the fp16 baseline runs out of memory on the same H800, while the packed path remains feasible with 33.42 GB and 60.56 GB peak memory.

RoPE-Structured Key-Cache Error

2.1 RoPE Block Notation For one query/key head, let q, k ∈ Rdh be split into L = dh /2 two-dimensional RoPE blocks q(i) , k(i) ∈ R2 , with block frequencies θi . At relative offset ∆, R∆ = diag(R(∆θ1 ), . . . , R(∆θL )), PL so the query-key logit is K∆ (q, k) = q⊤ R∆ k = i=1 q(i)⊤ R(∆θi )k(i) . (i)

Let k̂ be a decoded key in the same coordinate system as k, and define ek = k(i) − k̂(i) . The induced P (i) logit error is i q(i)⊤ R(∆θi )ek , and Cauchy–Schwarz together with rotation orthogonality gives PL (i) |K∆ (q, k) − K∆ (q, k̂)| ≤ i=1 ∥q(i) ∥2 ∥ek ∥2 . Each block thus contributes independently to the bound, with no cross-block terms. Since each RoPE block is an orthogonal 2 × 2 rotation, RoPE preserves the ℓ2 norm of every query/key block, so norm-based block statistics are RoPE-invariant— we can therefore compute the energy score in pre-RoPE coordinates. 2.2 TQ-MSE Rate Law Block-GTQ reuses the local encoder from TQ-MSE [41]: after normalizing a nonzero vector x, TQ-MSE applies a shared orthogonal rotation, scalar-quantizes the rotated coordinates, and restores the radius. For allocation, only its rate law is needed:√at b bits per coordinate, the decoded vector x̂ satisfies E∥x − x̂∥22 ≤ ∥x∥22 CTQ 4−b with CTQ = 3π/2. Each additional bit quarters the local MSE bound; Block-GTQ uses this 4−b rate law to allocate bits across RoPE blocks. 2.3 Key-Cache Logit Error R Let qR n = Rn qn and km = Rm km denote the post-RoPE query and key at positions n, m, with Rt the absolute RoPE rotation at position t. If the deployed cache decodes the post-RoPE key as k̂R m , the √ R ⊤ R R ⊤ R key-cache logit error En,m := |(qn ) km − (qn ) k̂m | is, up to the usual 1/ dh scaling, the logit perturbation induced by key-cache compression. We focus on keys because queries are computed on the fly and values are mixed only after softmax weights are computed. Although deployment stores post-RoPE keys, we analyze En,m in pre-RoPE coordinates with relative offset ∆ = m − n; the per-block bound from Subsection 2.1 applies: block i’s contribution depends only on the query norm and key-error norm in the same block (Appendices A.1 and A.2). This block-wise structure motivates a per-block bit allocation: forP each layer and KV head, choose integer bit widths b = (b1 , . . . , bL ) with bmin ≤ bi ≤ bmax and i bi = B, while keeping every RoPE block cached. In expectation, block i’s contribution to the bound is the ideal block weight s⋆i := E[∥q(i) ∥2 ∥k(i) ∥2 ] times the local quantizer’s bit-dependent rate (Appendix A.3). The optimal allocation therefore gives more bits to blocks with higher s⋆i .

Block-GTQ: RoPE-Block Bit Allocation

3.1

Block-Energy Score

Directly estimating s⋆i requires paired query-key products, which can be noisy on a short calibration prefix. Block-GTQ instead uses an AM-GM-based energy score that depends only on marginal Q/K second moments: si := 21 E[∥q(i) ∥22 + ∥k(i) ∥22 ]. By AM-GM, s⋆i ≤ si in expectation: si may overestimate s⋆i but never underestimates it. 3

Instantiating si for layer ℓ and KV head h, the empirical energy score is i i h h 1 (i) (i) Et,g∈G(h) ∥qℓ,g,t ∥22 + Et ∥kℓ,h,t ∥22 . sℓ,h,i = 2

(1)

Here G(h) is the set of query heads that read KV head h, and expectations are averaged over a short unlabeled calibration prefix; Appendix C expands these expectations into explicit sums. 3.2

Budgeted RoPE-Block Allocation

For each layer ℓ and KV head h with head-level integer budget B ∈ [Lbmin , Lbmax ], Block-GTQ PL −bi chooses an integer bit schedule P b = (b1 , . . . , bL ) that minimizes Jℓ,h (b) = i=1 sℓ,h,i 4 , subject to bmin ≤ bi ≤ bmax and i bi = B. Algorithm 1 Block-GTQ: greedy bit allocation per layer and KV head Require: Scores s1 , . . . , sL , feasible integer budget B, bounds bmin , bmax Ensure: Per-block bit widths b1 , . . . , bL 1: bi ← bmin for all i = 1, . . . , L 2: Bextra ← B − Lbmin 3: Initialize a max-priority queue with key ∆i = 34 si 4−bi for all i with bi < bmax 4: while Bextra > 0 and the queue is nonempty do 5: Pop i⋆ with largest ∆i 6: bi⋆ ← bi⋆ + 1, Bextra ← Bextra − 1 7: if bi⋆ < bmax then 8: Push i⋆ back with updated key ∆i⋆ = 34 si⋆ 4−bi⋆ 9: end if 10: end whilereturn b1 , . . . , bL To solve this objective, we use greedy bit allocation guided by the marginal reduction. Adding one bit to block i at current width bi reduces Jℓ,h by ∆i (bi ) = sℓ,h,i 4−bi − sℓ,h,i 4−(bi +1) = 43 sℓ,h,i 4−bi : high-score blocks ask for bits first, but each bit they receive divides their next marginal gain by four. Algorithm 1 initializes every block at bmin and repeatedly assigns the next bit to the block with the largest current ∆i until the budget is spent. In fact, greedy is optimal for this objective: Theorem 1 (Greedy optimality for the allocation objective). For positive si and feasible P scores −bi integer budget B ∈ [Lbmin , Lbmax ], Algorithm 1 minimizes J(b) = s 4 over all integer i i P allocations satisfying bmin ≤ bi ≤ bmax and i bi = B. The proof is in Appendix A.4. The greedy output is a bit schedule, not yet a physical cache layout. Block-GTQ realizes this schedule by grouping RoPE blocks with the same assigned bit width. For each nonempty group (ℓ,h) Gb = {i : bℓ,h,i = b}, we concatenate the corresponding post-RoPE key blocks and encode the resulting subvector with one TQ-MSE encoder at b bits/dim. This keeps the allocation decision at RoPE-block granularity while avoiding a separate tiny quantizer for every two-dimensional block. Uniform TQ-MSE is the special case in which all blocks belong to one same-rate group.

Serving Block-GTQ from a Packed Cache

At inference time, we serve Block-GTQ directly from a packed cache. The cache update writes packed K/V code streams, norms, and static layout metadata into HBM. The fused attention kernel loads only the current time tile, decodes it into kernel-local temporaries, and consumes them in QK and PV products; a full fp16 KV cache is never materialized in HBM. Figure 2 shows this path. The K stream follows the mixed-rate Block-GTQ schedule, with low-bit groups stored in nibble containers and higher-bit groups stored as bytes; the V stream is also packed, with uniform-allocation TQ-MSE. This layout turns the packed cache into a memory-bandwidth win. Single-token decoding is memorybandwidth bound—one query attends to all T cached keys, so each step is governed by streaming the KV cache from HBM. The fused kernel unpacks each tile (nibble extraction for ≤ 4-bit groups, byte loads for higher-bit K groups), dequantizes through a shared fp16 codebook small enough to stay resident in the L1 cache, rescales by the per-group K and per-token V norms, and forms QK ⊤ and P V as two fp16-input, fp32-accumulate tensor-core matmuls under a fully fp32 online softmax; 4

HBM Resident Packed Cache b=1

b=2

metadata schedule segments offsets

b=3

b=4

…

Inside Fused Attention Kernel No FP16 KV in HBM

b>4 …

Load K tile

Unpack segment

Lookup + norm

Transient K tile

Load V tile

Unpack segment

Lookup + norm

Transient V tile

K_NIBBLE mixed K stream b=3 …

V_NIBBLE uniform V3 stream

Tensor Core Attention

Load Q tile

Online SoftMax

𝑄𝐾 !

𝑃𝑉

Output

Figure 2: Packed-cache serving path. Persistent HBM stores packed K/V code streams plus norms and metadata. The fused attention kernel decodes only the current tile into kernel-local temporaries and consumes them directly in QK and PV products, avoiding a resident decoded fp16 KV cache. long contexts are split along the key axis and recombined with an exact log-sum-exp merge. The dequantized K/V stay in registers as tensor-core operands and are never written back to HBM, so the per-step HBM traffic is only the packed codes and norms—about 157 B per token and KV head at K3V3 versus 512 B for an fp16 pair (Table 18, ∼ 3.26×). The in-kernel unpack adds a fixed per-step cost that fp16 FlashAttention-2 does not pay, so the packed path is marginally slower at short context and overtakes fp16 only once the sequence is long enough for KV bandwidth to dominate—crossing over at T =128K, where it decodes 1.34× faster at 3.26× less KV memory (Table 19). Per-step launch overhead is removed by capturing the cache update as a CUDA graph, and the matching Q-side rotation from TQ-MSE is a small QR⊤ matmul after the q-projection that can be folded into the q-projection weights offline. Prefill. We populate the cache one transformer layer at a time, so each layer runs in a single full-length pass instead of the O(T ) steps of an autoregressive fill. The QKV/MLP projections and rotary embedding execute as full-T matrix multiplications, and two batched Triton kernels—one for K, one for V —each launch once per layer to quantize every head’s keys and values for the whole prompt, packing the codes into the cache’s nibble/mixed-byte layout and writing the code streams and norms straight into the persistent buffers in the same layout the decode kernel reads. Prefill attention is then a FlashAttention-2–style kernel over the packed cache: each program owns a tile of queries, streams the compressed K/V tiles, decodes them into kernel-local temporaries for a per-segment (per-rate-group) QK ⊤ accumulation and a full-width P V product, and accumulates under a causal mask with an online softmax—without materializing an fp16 attention matrix or KV cache. Constant launches per layer (vs O(T )) remove the dispatch overhead that otherwise dominates long-context prefill, making hundred-thousand-token prefill feasible on a single H800.

Related Work

Long-context inference and KV-cache memory. Autoregressive long-context decoding is often limited by repeatedly reading a KV cache that grows with sequence length [27]. Serving systems such as PagedAttention and CacheGen manage and reuse this state more carefully [17, 22], while contextextension methods such as YaRN and LongLoRA change how models reach longer windows [26, 4]. KV-cache quantization. Most KV-cache quantizers optimize reconstruction or outlier objectives at channel, token, group, or vector granularity. KIVI, KVQuant, ZipCache, Coupled Quantization, MiKV, MoQAE, and AQUA-KV [24, 14, 13, 43, 39, 34, 29] pair low-bit KV storage with outlier or mixed-precision adjustments; KVSink, Outlier Tokens Tracing, and SQuat target sink tokens, outliers, and query-subspace structure [32, 31, 35]. PolarQuant and TurboQuant [12, 41] provide local vectorquantization primitives, while GEAR adds low-rank and sparse error recovery [16]. Block-GTQ uses TurboQuant-MSE as its local primitive but greedily allocates bits per RoPE frequency block via a block-energy score, since the key-side attention logit decomposes into block terms. RoPE-aware KV-cache quantization. Several methods exploit RoPE structure when reducing KVcache cost. KVQuant and RotateKV both operate before RoPE—the former quantizes keys, the latter applies outlier-aware rotations [14, 33]; CommVQ learns codebooks that commute with RoPE [18]. 5

EliteKV combines head-specific RoPE-frequency selection with joint low-rank projection [45]; RAP prunes RoPE-aligned pairs [38]; TriAttention scores key importance via pre-RoPE Q/K geometry and trigonometric distance [25]. Block-GTQ instead greedily allocates precision per RoPE block using a block-energy score derived from the RoPE logit-error bound; no block is dropped. Non-uniform precision allocation. The closest line assigns precision non-uniformly: PM-KVQ at per-layer granularity (shared by K and V) [21]; MixKVQ (query-aware) and Kitty at key-channel granularity [42, 36]; and Ada-KV via head-wise eviction budgets [10]. Block-GTQ differs in both unit and score: its greedy allocator assigns bits to RoPE frequency blocks inside each head from a block-energy score derived from the RoPE logit-error bound. Token retention and low-rank compression. Another family reduces cache cost by keeping, merging, or sampling cached tensors. Attention Sinks, H2 O, Scissorhands, FastGen, SnapKV, PyramidKV, MagicPIG, and SubGen retain or sample tokens based on sink behavior, heavy hitters, persistence, profiled head patterns, attention scores, pyramidal layer budgets, or clustering [37, 44, 23, 11, 19, 2, 5, 40]. Low-rank and hybrid methods (Palu, MiniCache, GEAR for KV cache; UniQL for edge LLMs) reduce cache dimensionality, merge across depth, or recover quantization error [3, 20, 16, 6]. These directions are orthogonal to Block-GTQ’s per-RoPE-block bit allocation.

Experiments

6.1 Allocation and Attention Diagnostics We empirically verify Block-GTQ on a ten-model panel, inspecting (i) the bit allocation it produces, (ii) the resulting RoPE-logit error, and (iii) the resulting attention distributions. The panel covers GQA backbones from Qwen, Llama, DeepSeek, Mistral, and GLM plus an MLA-based DeepSeek model. All experiments in this subsection quantize K only; V remains in fp16 to isolate the effect of K-cache quantization. Details about the panel and experimental setup are in Appendix B.1. We first inspect the bit allocation Block-GTQ produces at the 3 b/dim budget. Figure 3 shows that every architecture has a non-uniform RoPE-block energy profile, which Block-GTQ translates into a non-uniform allocation. Aggregate distributions and per-layer heterogeneity across the panel are in Appendix B.2. We then measure per-layer RoPE-logit error. Table 1 shows that Block-GTQ reduces mean RoPE-logit MAE versus uniform TQ-MSE on all 10 models and wins 367/367 (100%) layer comparisons; the same 367/367 pattern holds at the tighter 2 b/dim budget (Table 11). Definition and protocol are in Appendix B.3. Finally, we test whether these per-layer reductions propagate to the attention distribution itself. Figure 4 shows that, without a recent-token buffer, Block-GTQ achieves both the lowest mean softmax KL versus fp16 and the highest top-10 attended-token overlap at every budget. Setup, metrics, and additional results are in Appendix B.4. Table 1: Per-layer RoPE-logit error at the 3 b/dim budget, K-only. Values are mean RoPE-logit MAE across model layers; lower is better. ∆ is the relative reduction versus TQ-MSE; “Wins” counts layers where Block-GTQ beats uniform TQ-MSE. Definition and protocol in Appendix B.3. Model Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B-A3B Llama-3.1-8B

TQ-MSE

Block-GTQ

∆

Wins

Model

6.43 4.14 5.81 6.76 3.80

3.23 2.61 2.96 3.00 2.55

+49.9% +37.1% +49.0% +55.6% +32.7%

36/36 48/48 36/36 48/48 32/32

DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

TQ-MSE

Block-GTQ

∆

Wins

3.44 11.44 3.46 7.30 6.01

2.33 2.40 2.28 4.51 3.87

+32.2% +79.1% +34.2% +38.2% +35.5%

32/32 28/28 40/40 40/40 27/27

Per-Layer Bit Distribution Fingerprint (3b avg budget) 1b Qwen2.5-3B (H=1.55)

100

Qwen3-8B (H=1.60)

100

8b Qwen3-30B-A3B (H=1.56)

100

40 20

Layer

DS-R1-Llama-8B (H=1.35)

100

40 20

Layer

DS-R1-Qwen-7B (H=1.58)

100

% of blocks

Layer

Mistral-Nemo-12B (H=1.28)

100

Layer

GLM-4-9B (H=1.33)

100

20 0

Layer

40 20

Layer

% of blocks

40 20

Layer

DS-V2-Lite (H=1.53)

100

Llama-3.1-8B (H=1.42)

100

% of blocks

100

Qwen2.5-14B (H=1.44)

40 20

Layer

Figure 3: Allocator fingerprint at the 3 b/dim budget. One subplot per model; each vertical slice is one layer, with stacked color bands giving its bit-width distribution (1b red → 8b green). (a) No-buffer Softmax-KL across models

(b) No-buffer Top-10 vs Softmax-KL

Softmax KL (log

lower is better)

Block-GTQ KIVI-ScaleOnly TQ-MSE

10 1

2-bit

3-bit

Top-10 attention overlap, % (higher is better)

Method

4-bit

Block-GTQ KIVI-ScaleOnly TQ-MSE

2-bit 3-bit 4-bit

80 70 60 50 40

10 2

10 1

Softmax KL (per-model, log

lower is better)

100

Figure 4: Mean softmax KL and Top-10 attended-token overlap across models. K-only quantization (V stays fp16). (a) Mean softmax KL versus fp16 per method, averaged over the ten-model panel at 2, 3, and 4 b/dim budgets. (b) Top-10 attended-token overlap versus softmax KL, one marker per model (color: method, shape: bit-rate); the upper-left corner is best. See Appendix B.4 for the KIVI no-buffer setting. 6.2

Calibration Robustness Ablation

We ablate calibration along two axes (Table 2): (a) length, sweeping Ncal ∈ {64, . . . , 2048} tokens from WikiText-2 test; and (b) corpus, drawing 2048 tokens each from WikiText-2, PG19, C4, and code. We further extend the length axis to a continuous metric (Table 3): for each cell we draw three independent calibration prefixes from WikiText-2 train (offsets 0/10k/20k) and report sliding-window PPL on the full WikiText-2 test set as mean ± std. Both show K3V3 is robustly less sensitive than K2V2: at K3V3 the six NIAH subtasks stay within 1.07 pp and PPL within ±1σ across seeds, while at K2V2 NIAH swings 1.57–4.09 pp and PPL by several σ. This reflects the 4−b rate law in P Block-GTQ’s allocator objective i si · 4−bi : a misplaced bit at b=3 (K3V3) costs roughly 4× less than at b=2 (K2V2), so the same calibration noise has a proportionally smaller downstream effect. Details and more results are in Appendix C.

Table 2: Calibration ablations on Llama-3.1-8B-Instruct along two axes: (a) calibration length and (b) calibration corpus. NIAH Overall (%) is the unweighted mean of six NIAH subtasks; higher is better. (a) Calibration length: ∆ vs Ncal = 2048 baseline. K2V2

(b) Calibration corpus: ∆ vs WikiText-2 baseline. Each row draws 2048 tokens from the named corpus.

K3V3

Ncal

Overall

∆

Overall

∆

64 128 256 512 1024 2048

95.68 94.28 94.08 95.79 93.27 97.36

−1.68 −3.08 −3.28 −1.57 −4.09 (base)

98.46 98.63 97.33 98.01 98.29 98.40

+0.06 +0.23 −1.07 −0.39 −0.11 (base)

K2V2

K3V3

Corpus

Overall

∆

Overall

∆

WikiText-2 PG19 C4 code

97.36 97.08 94.58 93.66

(base) −0.28 −2.78 −3.70

98.40 98.12 98.23 98.06

(base) −0.28 −0.17 −0.34

Table 3: Calibration length sensitivity under prefix noise. Cells are PPL mean ± std across three WT2-train calibration prefixes (offsets 0/10k/20k). ∆ is the change in mean PPL from Ncal =128 to Ncal =2048. Model

fp16

Ncal =128

Ncal =512

Ncal =2048

∆

Llama-3.1-8B-Instruct Llama-3.1-8B-Instruct DS-R1-Qwen-7B DS-R1-Qwen-7B

K3V3 K2V2 K3V3 K2V2

6.4095 6.4095 18.4800 18.4800

6.6414 (±0.0192) 7.9949 (±0.0235) 19.0611 (±0.0229) 23.2010 (±0.2150)

6.6462 (±0.0137) 7.8474 (±0.0668) 19.0303 (±0.0294) 23.1319 (±0.1782)

6.6256 (±0.0130) 7.8459 (±0.0784) 19.0576 (±0.0328) 23.0211 (±0.1132)

−0.0158 −0.1490 −0.0035 −0.1799

6.3

Downstream Evaluation

In Section 6.1 we showed that Block-GTQ reduces RoPE-logit error and preserves the softmax attention distribution. We now ask whether this attention-interface advantage carries to downstream task quality, focusing on two regimes where K-cache errors are most consequential: long-context retrieval and understanding, where old keys must remain useful across a long prompt; and reasoningstyle generation, where small attention perturbations can compound over many decode steps. Long-Context Tasks fp16

KIVI-ScaleOnly

Block-GTQ (2k calib)

TQ-MSE

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

1.00

0.00

0.67

0.33

0.67

1.00

0.33

0.00

0.33

1.00

0.00

0.33

0.00

0.33

1.00

0.67

0.33

0.67

0.00

0.67

0.33

1.00

0.00

0.33

0.00

0.67

0.33

1.00

0.00

0.33

0.00

0.67

1.00

0.67

0.33

0.67

0.00

0.67

1.00

0.33

0.00

0.67

0.00

0.33

0.00

1.00

0.67

0.33

0.00

1.00

0.33

1.00

0.33

0.00

0.33

1.00

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

1.00

0.00

0.33

0.00

0.67

1.00

0.67

1.00

0.00

0.33

0.00

1.00

0.67

0.00

0.33

0.00

0.33

1.00

0.33

0.00

0.67

0.33

1.00

0.33

0.00

0.67

0.33

0.00

1.00

0.67

1.00

0.33

0.67

0.00

0.33

1.00

0.67

1.00

0.33

0.00

0.33

0.67

0.00

0.33

1.00

0.67

1.00

0.33

0.00

0.33

0.00

0.33

0.67

1.00

0.67

1.00

0.33

0.00

0.67

0.33

0.00

1.00

0.67

1.00

0.33

0.67

0.00

0.33

1.00

0.67

1.00

0.67

1.00

0.67

1.00

0.00

1.00

16K

32K

64K 128K

16K

32K

64K 128K

16K

32K

64K 128K

16K

32K

64K 128K

1.00

0.75

pass rate

K3V3 needle depth

6.3.1

K2V2 needle depth

0.50

context length

0.25

0.00

Figure 5: NIAH single-needle retrieval on Llama-3.1-8B-Instruct. Pass rate is shown over context length (4K–128K) and needle depth (0%–100%), averaged over three trials per cell. The two rows use the same method layout: fp16, KIVI-ScaleOnly (Appendix B.4), TQ-MSE, and Block-GTQ. NIAH. NIAH [15] probes where retrieval breaks across context length and needle depth. On Llama-3.1-8B-Instruct, Figure 5 shows that Block-GTQ’s NIAH retrieval pattern matches fp16’s at both K3V3 and K2V2. Table 4a quantifies this across the six NIAH subtasks: TQ-MSE drops from 97.7 at K3V3 to 70.6 at K2V2 and KIVI-ScaleOnly never exceeds 35.4 Avg, while Block-GTQ stays close to fp16’s 99.6 ceiling, scoring 98.4/96.8/97.4 at K3V3/K3V2/K2V2. The gap is wider 8

Table 4: Multi-task NIAH pass-rate (%) on (a) Llama-3.1-8B-Instruct and (b) Qwen2.5-7B-Instruct. Each entry is averaged over context lengths 4K–128K, needle depths (0%–100%), three trials per cell. Block-GTQ uses a 2048-token WikiText-2 calibration; KIVI-ScaleOnly is defined in Appendix B.4. (a) Llama-3.1-8B-Instruct.

(b) Qwen2.5-7B-Instruct.

Method

single distr. multi m-key m-val m-qry Avg

Method

single distr. multi m-key m-val m-qry Avg

fp16

100.0 100.0 100.0

fp16

96.0 92.9 83.8

99.5 100.0

97.8 99.6

K3V3 KIVI-ScaleOnly 38.4 49.5 36.4 33.3 26.6 TQ-MSE 100.0 99.5 100.0 100.0 99.5 Block-GTQ 100.0 100.0 100.0 100.0 99.7

28.3 35.4 87.4 97.7 90.7 98.4

KIVI-ScaleOnly TQ-MSE Block-GTQ

59.1 44.4 37.9 0.0 0.0 0.0 96.0 91.4 78.8

27.6 34.4 82.0 95.7 81.1 96.8

KIVI-ScaleOnly TQ-MSE Block-GTQ

59.1 42.4 24.8 0.0 0.0 0.0 92.9 90.9 75.3

K3V2

32.0 67.1

20.7 32.9 0.0 0.0 30.6 62.0

16.3 35.2 0.0 0.0 31.8 65.1

23.2 24.1 0.0 0.0 37.4 61.3

19.5 32.2 0.0 0.0 30.8 64.8

12.1 6.6 0.0 0.0 40.1 54.7

7.9 12.0 0.0 0.0 29.1 60.1

K3V2

KIVI-ScaleOnly 39.4 50.0 34.3 31.0 24.1 TQ-MSE 100.0 99.0 100.0 96.1 97.0 Block-GTQ 100.0 100.0 100.0 100.0 99.7 K2V2 KIVI-ScaleOnly 30.8 32.8 23.2 TQ-MSE 93.9 70.7 80.8 Block-GTQ 100.0 99.0 100.0

31.5 66.3

K3V3

K2V2 20.9 16.2 47.6 71.9 98.5 100.0

21.6 24.2 58.8 70.6 86.7 97.4

KIVI-ScaleOnly TQ-MSE Block-GTQ

20.7 14.1 10.6 0.0 0.0 0.0 85.9 77.8 73.2

on Qwen2.5-7B-Instruct: TQ-MSE collapses to 0.0 at every budget and KIVI-ScaleOnly never exceeds 35.2 Avg, while Block-GTQ stays close to fp16’s 67.1 ceiling, scoring 65.1/64.8/60.1 at K3V3/K3V2/K2V2 (Qwen2.5-7B-Instruct heatmap: Appendix D.1).

Table 5: LongBench-EN per-subtask scores on Llama-3.1-8B-Instruct. Subtask abbreviations and metrics are listed in Appendix D.1; Avg is the unweighted mean. KIVI denotes KIVI-ScaleOnly (Appendix B.4). Higher is better; bold marks the best quantized method within each budget. Rate

Method

Qasp

MFQA

Gov

TREC

Pass

LCC

Avg

fp16

44.70

55.77

57.62

48.85

34.54

72.50

99.50

65.13

59.83

K3V3

KIVI TQ-MSE Block-GTQ

43.83 43.96 44.65

49.56 54.33 54.80

51.53 53.37 55.25

37.31 44.12 46.14

31.82 33.31 33.79

70.50 71.00 74.50

46.56 95.50 98.50

49.56 63.59 64.97

47.58 57.40 59.08

K3V2

KIVI TQ-MSE Block-GTQ

42.71 44.02 47.04

47.61 53.75 53.49

47.10 52.18 53.88

36.23 41.20 48.20

30.82 33.08 33.88

70.50 71.00 70.50

36.69 97.00 97.50

48.49 60.96 66.20

45.02 56.65 58.84

K2V2

KIVI TQ-MSE Block-GTQ

36.47 32.66 45.26

43.34 43.61 50.95

43.71 33.65 46.41

34.33 27.25 37.41

28.19 28.12 32.25

70.50 55.50 70.50

7.67 31.92 91.50

43.49 42.24 52.17

38.46 36.87 53.31

LongBench-EN. LongBench-EN [1] is the natural-task counterpart to NIAH: it tests whether the quantizer preserves attention well enough for generation, not just retrieval. Table 5 reports Llama-3.1-8B-Instruct on eight subtasks; full subtask definitions and the inference protocol are in Appendix D.1. Across all three budgets, Block-GTQ Overall stays closest to the 59.83 fp16 ceiling (59.08/58.84/53.31 at K3V3/K3V2/K2V2); at the tight K2V2 budget, TQ-MSE drops to 36.87 and KIVI-ScaleOnly to 38.46. 6.3.2

Reasoning Tasks

Reasoning tests the cache differently from retrieval: in long chain-of-thought decoding, small cache errors compound across many decode steps and surface as a wrong final answer. We evaluate BlockGTQ on AIME 2024 and AIME 2025 in thinking mode on two DeepSeek-R1 [9] distilled backbones at K3V2, reporting average pass@1 over 8 samples per problem. To separate the contribution of the quantizer from that of the recent-token fp16 buffer, we compare two regimes. No buffer removes all uncompressed-token windows so every attended key is served from the compressed cache. Protected keeps the first 4 tokens (sink) and the last 128 tokens (recent) as fp16. The 128-token recent window matches PM-KVQ’s protected configuration [21]; the 4-token 9

sink follows the attention-sink convention [37]. We apply this 4/128 allowance identically across methods; per-method details are in Appendix D.2. Table 6: AIME 2024/2025 pass@1 (%) at K3V2. Protected: 4 sink + 128 recent fp16; No buffer: both 0 (Section 6.3.2). Under no-buffer, KIVI is run as KIVI-ScaleOnly (Appendix B.4). fp16 Model

Regime

TQ-MSE

KIVI

PM-KVQ

Block-GTQ

AIME’24 AIME’25 AIME’24 AIME’25 AIME’24 AIME’25 AIME’24 AIME’25 AIME’24 AIME’25

DeepSeek-R1Protected Distill-Qwen-7B No buffer

54.2 54.2

37.9 37.9

7.5 0.0

8.8 0.0

52.9 28.8

35.8 19.0

45.4 40.8

27.9 27.5

54.2 51.7

37.5 37.5

DeepSeek-R1Protected Distill-Llama-8B No buffer

43.3 43.3

28.8 28.8

37.9 26.2

26.7 20.0

43.3 7.5

27.1 10.4

43.3 42.9

28.8 24.6

43.8 32.5

30.0 23.3

In the protected regime, Block-GTQ stays close to fp16 on both backbones (matching on DeepSeekR1-Distill-Qwen-7B, slightly exceeding on DeepSeek-R1-Distill-Llama-8B). TQ-MSE is notably lower on both, especially on DeepSeek-R1-Distill-Qwen-7B. PM-KVQ matches fp16 on DeepSeekR1-Distill-Llama-8B (still slightly below Block-GTQ) but is lower on DeepSeek-R1-Distill-Qwen-7B. In the no-buffer regime (AIME 2024/AIME 2025), Block-GTQ stays close to fp16 on DeepSeekR1-Distill-Qwen-7B (51.7/37.5 vs 54.2/37.9) but is lower on DeepSeek-R1-Distill-Llama-8B (32.5/23.3 vs 43.3/28.8). PM-KVQ [21] shows the opposite pattern: leading at 42.9/24.6 on DeepSeek-R1-Distill-Llama-8B but lower at 40.8/27.5 on DeepSeek-R1-Distill-Qwen-7B. TQ-MSE collapses on both backbones (worst at 0.0/0.0 on DeepSeek-R1-Distill-Qwen-7B), as does KIVI without the buffer. Block-GTQ’s no-buffer drop on DeepSeek-R1-Distill-Llama-8B reflects a bitallocation difference: PM-KVQ allocates K and V jointly per layer via loss-gradient sensitivity, whereas Block-GTQ allocates only K per RoPE block (via energy) and leaves V at uniform TQ-MSE. Without the recent-token buffer, this K-only allocation can surface as a quality gap on V-sensitive backbones. Adding a V-side allocator to Block-GTQ is a natural extension. 6.4

Block-GTQ Deployment

We run Qwen2.5-3B-Instruct on a single H800 GPU at the K3V3 operating point and report decodestep latency, peak GPU memory, and downstream perplexity. We compare Block-GTQ against an fp16 FlashAttention-2 (FA-2) baseline and uniform-TQ-MSE. Block-GTQ and uniform-TQ-MSE both run through our fused-attention packed-cache path (Section 4), which unpacks compressed K/V codes inline within the attention kernel; they differ only in K layout: uniform-TQ-MSE uses a single bit-width per head, while Block-GTQ varies it across RoPE blocks.

fp16 FA-2 TQ-MSE Block-GTQ fp16 OOM

150 125

1000 8.3×

800 600

100

145 μs

200 0

2.5× 59 μs

1.3×

44 μs

1.3×

17k

16.7 16.9

6.7

8.1

7.7

352

299k

11k 7.5

6.5

7.8

numbers near points = PPL

16K

64K

25 35 μs

V ne ne re K raph seli used utotu r-co AG +F ile a UD enso +T +C +T

a on b Trit

608

75 50

400

25k

13k

128

Context length T

fp16 FA-2 TQ-MSE Block-GTQ fp16 OOM

1.2k

256

512

Peak GPU memory (GB)

1200 μs

1200

(b) End-to-end decode latency

ms / step

Decode latency (μs / step)

(a) Optimization stages at T = 8K

H800 80GB ceiling

60 40 20 0

16K

64K

128

Context length T

256

512

Figure 6: Kernel-optimization gains, decode latency, and peak memory for Block-GTQ on Qwen2.53B-Instruct (single H800). Panel (a) decomposes the speedup contribution of each kernel-optimization stage—each stage targets a specific bottleneck of the packed-cache decode path. As context length grows (panels (b), (c)), Block-GTQ’s optimized decode kernel overtakes fp16 FA-2 at T = 128K and continues to run cleanly at T ≥ 256K where fp16 OOMs. PPL is annotated on in panel (b). At short context (T ≤ 64K), Block-GTQ’s decode kernel is slower than fp16 FA-2: the packed-cache path pays per-step overhead for in-kernel unpacking of compressed K/V codes that fp16 FA-2 does not incur. As context grows, KV bandwidth dominates per-step decode and Block-GTQ overtakes fp16; at T = 128K, Block-GTQ runs 1.34× faster than fp16 and cuts peak memory from 56.31 GB 10

to 19.85 GB. Beyond this, fp16 OOMs at T ≥ 256K because peak total memory exceeds the 80 GB GPU budget, while Block-GTQ continues to run. Uniform-TQ-MSE is modestly faster than Block-GTQ on decode (∼ 14% at T = 128K) and has a slightly smaller KV footprint (3.88× vs. 3.24× compression; the gap comes from per-segment metadata needed by Block-GTQ’s mixed-rate K storage). However, TQ-MSE’s quality collapses: its PPL is orders of magnitude worse than Block-GTQ’s at every tested context length, while Block-GTQ stays close to fp16’s PPL (annotated in Figure 6(b); full values in Table 21)—making Block-GTQ the deployable operating point. Full latency, memory, and prefill matrices are in Appendix E.

Conclusion

We reframe low-bit K-cache compression for RoPE models as a block-level rate-allocation problem. Because RoPE attention decomposes exactly over two-dimensional frequency blocks and block energy is non-uniform, Block-GTQ uses a label-free energy score to assign more bits to high-energy RoPE blocks. Both K and V are encoded with TQ-MSE, V at a uniform bit-width. On a diverse ten-model panel, at both 2 and 3 b/dim K-only, Block-GTQ cuts per-layer RoPE-logit MAE by 32–80% across models and wins all 367/367 layer comparisons at each budget against uniform TQ-MSE. Across NIAH, LongBench-EN, and AIME, Block-GTQ stays close to the fp16 ceiling at tight K budgets, where uniform TQ-MSE typically collapses. On a single H800 at the K3V3 budget, our packed-cache serving path enables long-context inference that fp16 FlashAttention2 cannot reach: with 3.24× KV-cache compression and quality comparable to fp16, it runs 1.34× faster at 128K context and remains feasible at 256K/512K where fp16 OOMs. Limitations and future work. Block-GTQ allocates bits only on K, leaving V uniform. A V-side allocator, joint K+V optimization, and denser packing could further reduce memory. The fused decode path is an initial single-GPU implementation; multi-GPU and batched serving are open directions.

References [1] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, 2024. [2] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. In Conference on Language Modeling (COLM), 2025. arXiv:2406.02069. [3] Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: Kv-cache compression with low-rank projection. In International Conference on Learning Representations (ICLR), 2025. [4] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In International Conference on Learning Representations (ICLR), 2024. [5] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Léon Bottou, Zhihao Jia, and Beidi Chen. MagicPIG: Lsh sampling for efficient llm generation. In International Conference on Learning Representations (ICLR), 2025. [6] Hung-Yueh Chiang, Chi-Chih Chang, Yu-Chen Lu, Chien-Yu Lin, Kai-Chiang Wu, Mohamed S. Abdelfattah, and Diana Marculescu. UniQL: Unified quantization and low-rank compression for adaptive edge llms. In International Conference on Learning Representations (ICLR), 2026. 11

[7] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. [8] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [9] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [10] Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2407.11550. [11] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations (ICLR), 2024. [12] Insu Han, Praneeth Kacham, Vahab Mirrokni, Amir Zandieh, and Amin Karbasi. Polarquant: Quantizing kv caches with polar transformation. In International Conference on Machine Learning (ICML), 2025. [13] Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [14] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [15] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. [16] Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint arXiv:2403.05527, 2024. [17] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023. [18] Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Colorado Reed, Chong Wang, and Chuang Gan. CommVQ: Commutative vector quantization for KV cache compression. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 36831–36845. PMLR, 2025. URL https: //proceedings.mlr.press/v267/li25du.html. [19] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [20] Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. MiniCache: KV cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366, 2024. [21] Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. PM-KVQ: Progressive mixed-precision kv cache quantization for long-cot llms. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.18610; code: https://github.com/thu-nics/PM-KVQ. 12

[22] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, 2024. doi: 10.1145/3651890.3672274. [23] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. arXiv preprint arXiv:2305.17118, 2023. [24] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Forty-first International Conference on Machine Learning (ICML), 2024. [25] Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, and Yukang Chen. TriAttention: Efficient long reasoning with trigonometric KV compression. arXiv preprint arXiv:2604.04921, 2026. [26] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In International Conference on Learning Representations (ICLR), 2024. [27] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. [28] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. arXiv preprint arXiv:2303.06865, 2023. [29] Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Nikita Surkov, Ivan Ermakov, and Dan Alistarh. Cache me if you must: Adaptive key-value quantization for large language models. In Forty-second International Conference on Machine Learning (ICML), 2025. [30] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [31] Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. Accurate kv cache quantization with outlier tokens tracing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12895–12915, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.631. URL https: //aclanthology.org/2025.acl-long.631/. [32] Zunhai Su and Kehong Yuan. KVSink: Understanding and enhancing the preservation of attention sinks in kv cache quantization for llms. In Conference on Language Modeling (COLM), 2025. [33] Zunhai Su, Hanyu Wei, Zhe Chen, Wang Shen, Linge Li, Huangqi Yu, and Kehong Yuan. RotateKV: Accurate and robust 2-bit KV cache quantization for LLMs via outlier-aware adaptive rotations. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), pages 6200–6208, 2025. doi: 10.24963/ijcai.2025/690. URL https://www.ijcai.org/proceedings/2025/690. [34] Wei Tao, Haocheng Lu, Xiaoyang Qu, Bin Zhang, Kai Lu, Jiguang Wan, and Jianzong Wang. MoQAE: Mixed-precision quantization for long-context llm inference via mixture of quantization-aware experts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10810–10820, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.531. URL https://aclanthology.org/2025.acl-long.531/. 13

[35] Hao Wang, Ligong Han, Kai Xu, and Akash Srivastava. SQuat: Subspace-orthogonal kv cache quantization. arXiv preprint arXiv:2503.24358, 2025. [36] Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, Junxiong Wang, Jue Wang, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou, Ben Athiwaratkun, Zhen Zheng, and Shuaiwen Leon Song. Kitty: Accurate and efficient 2-bit kv cache quantization with dynamic channel-wise precision boost. arXiv preprint arXiv:2511.18643, 2025. Code: https://github.com/Summer-Summer/Kitty. [37] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations (ICLR), 2024. [38] Jihao Xin, Tian Lyu, David Keyes, Hatem Ltaief, and Marco Canini. RAP: KV-cache compression via RoPE-aligned pruning. arXiv preprint arXiv:2602.02599, 2026. [39] June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096, 2024. [40] Amir Zandieh, Insu Han, Vahab Mirrokni, and Amin Karbasi. SubGen: Token generation in sublinear time and memory. arXiv preprint arXiv:2402.06082, 2024. [41] Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. Turboquant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations (ICLR), 2026. [42] Tao Zhang, Ziqian Zeng, Hao Peng, Huiping Zhuang, and Cen Chen. MixKVQ: Queryaware mixed-precision kv cache quantization for long-context reasoning. arXiv preprint arXiv:2512.19206, 2025. [43] Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [44] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2 O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [45] Yuhao Zhou, Sirui Song, Boyang Liu, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Zhihao Zhang, Wei Li, and Xuanjing Huang. EliteKV: Scalable KV cache compression via RoPE frequency selection and joint low-rank projection. arXiv preprint arXiv:2503.01586, 2025.

Appendix Roadmap Appendix A collects proofs for Block-GTQ (error bound, block weight, greedy optimality). Appendix B details the ten-model panel, the bit-allocation analysis, and the attention-fidelity diagnostics. Appendix C ablates calibration along length, score, and corpus, and reports cross-model PPL and allocation stability. Appendix D provides long-context (NIAH, LongBench) and reasoning (AIME) protocols. Appendix E provides the deployment data tables: footprint, latency/memory, and longcontext perplexity.

Supplementary Theory Details

This appendix collects the theory details that are useful for auditability but are not needed in the main narrative. The main text uses three facts: the deployed K-cache error is a RoPE-logit error (Section 2.3), that error admits a per-block bound (Lemma 2), and the resulting allocation objective is optimized exactly by greedy allocation (Theorem 1). The details below explain the coordinate change, the proof of the per-block error bound, the absolute-error chain behind the block weight, and the greedy allocation proof. 14

A.1

Post-RoPE Cache and Pre-RoPE Coordinates

Although the cache stores post-RoPE keys, the analysis can be written in pre-RoPE coordinates. If k̂R m is the decoded post-RoPE key and Rt denotes the absolute RoPE rotation at position t, define ⊤ R k̂m := Rm k̂m . Then, for a query at position n, ⊤ R ⊤ R ⊤ ⊤ ⊤ (qR n ) k̂m = (Rn qn ) k̂m = qn Rn Rm k̂m = qn Rm−n k̂m .

(2)

RoPE is orthogonal block by block, so this coordinate change does not change block norms. It only lets us express the deployed post-RoPE cache error as a relative-position logit error. A.2

Proof of the Per-Block Accounting Bound

Lemma 2 (Per-block accounting of attention-logit error). For a query at position n, a cached key at (i) (i) (i) ⊤ R position m, and the equivalent pre-RoPE decoded key k̂m = Rm k̂m , let ek,m = km − k̂m and En,m := Km−n (qn , km ) − Km−n (qn , k̂m ) . Then En,m ≤

(i)

∥q(i) n ∥2 ∥ek,m ∥2 .

Proof. With ∆ = m − n, the block decomposition in Section 2.1 gives X (i) K∆ (qn , km ) − K∆ (qn , k̂m ) = q(i)⊤ R(∆θi )ek,m . n i

The triangle inequality and Cauchy–Schwarz yield X X (i) (i) En,m ≤ q(i)⊤ R(∆θi )ek,m ≤ ∥q(i) n n ∥2 ∥R(∆θi )ek,m ∥2 . i

i (i)

(i)

Each R(∆θi ) is a rotation, so it preserves the block norm: ∥R(∆θi )ek,m ∥2 = ∥ek,m ∥2 . A.3

From the Block Bound to the RoPE-Block Weight

Lemma 2 gives, for each query-key pair, En,m ≤

(i)

∥q(i) n ∥2 ∥ek,m ∥2 .

Suppose a local quantizer at bit width bi contributes a relative error factor αi (bi ) in block i, so that the typical block error is bounded by αi (bi )∥k(i) ∥2 . Taking expectations over future query-key pairs gives X E[E] ≲ αi (bi ) E[∥q(i) ∥2 ∥k(i) ∥2 ] . (3) {z } | i

s⋆ i

This derivation identifies the logit-error block weight s⋆i ; it is not the final method loss. Block-GTQ then uses the energy surrogate si from Section 3.1 together with the TQ-MSE bit-error decay. The resulting allocation objective X J(b) = si 4−bi i

should be read as a rate-allocation proxy rather than a tight consequence of the absolute-error bound above: the score comes from the RoPE-logit sensitivity, while the factor 4−bi comes from the local MSE-oriented quantizer. The 4−bi rate is not arbitrary: it matches the rate at which the squared logit error decays. Squaring the P (i) (i) 2 per-block bound and applying Cauchy–Schwarz gives En,m ≤ L i ∥qn ∥22 ∥ek,m ∥22 ; together with the TQ-MSE squared-error E∥e(i) ∥22 ≲ 4−bi ∥k(i) ∥22 , this yields a mean-squared logit-error P −bi bound (i) 2 bound of the form i 4 E[∥q ∥2 ∥k(i) ∥22 ]. The 4−bi rate in J is thus consistent with bounding the squared logit error, with si serving as a simpler second-moment proxy for the product weight. 15

A.4

Proof of Greedy Allocation Optimality

This section gives the full exchange proof for Theorem 1. Fix positive scores si , bounds bmin ≤ bi ≤ bmax , and a feasible integer budget B ∈ [Lbmin , Lbmax ]. Start all blocks at bmin and let K = B − Lbmin be the number of extra bit units to assign. For block i, define the gain of its r-th extra bit as gi,r := si 4−(bmin +r−1) − si 4−(bmin +r) = 34 si 4−(bmin +r−1) , for r = 1, . . . , bmax − bmin . These gains decrease geometrically in r. Choosing a final bit width bi = bmin + ki is equivalent to choosing the first ki gains gi,1 , . . . , gi,ki from block i. Hence every feasible allocation chooses exactly K gains subject to a prefix constraint: it may choose gi,r only if it also chooses gi,1 , . . . , gi,r−1 . The value of an allocation is the total chosen gain, because subtracting these gains from the all-bmin objective gives J(b). Algorithm 1 repeatedly chooses the largest available gain, where available means that the required prefix for that block has already been chosen. We prove optimality by induction on the greedy prefix. Assume there is an optimal feasible set O containing the first t greedy gains, and let Pt denote that prefix. Let g be the next greedy gain, from block a. If g ∈ O, the invariant already holds for Pt ∪ {g}. Otherwise, add g to O; this is prefix-feasible because g was available after Pt , and Pt ⊆ O. The enlarged set has one too many gains, so we remove a terminal gain from another block without reducing value. Since O contains K gains but omits g, some block j has gains in O \ Pt . Let hj be the first such gain after the prefix of block j already present in Pt . This gain was available to greedy at step t + 1, so g ≥ hj . Remove instead the last selected gain from block j in O; monotonicity gives this terminal gain value at most hj , hence at most g, and removing a terminal gain preserves the prefix constraint. The exchange therefore produces an optimal feasible set containing Pt ∪ {g}. Repeating for t = 0, . . . , K − 1 proves the greedy allocation is optimal.

Attention-Interface Diagnostic Details

The main text reports the bit-allocation fingerprint, the cross-model RoPE-logit error summary, and the panel-wide softmax-KL bars and top-10 overlap scatter. This appendix supplies the model panel, activation-extraction rules, the panel-level bit-allocation analysis (aggregate distributions and per-layer heterogeneity), metric definitions, the per-layer RoPE-logit error protocol, and per-model softmax-KL and top-10 overlap tables. B.1

Model Panel and Activation Extraction

Table 7 lists the ten-model panel used for the cross-architecture attention diagnostics. The panel is chosen for architectural coverage rather than leaderboard coverage. Nine models use GQA (small to larger Qwen2.5, Qwen3 with QK-RMSNorm including the MoE Qwen3-30B-A3B, Llama-3.1, two reasoning-distilled DeepSeek-R1 backbones, Mistral-Nemo, and the fused-QKV GLM-4-9B), and one uses MLA (DS-V2-Lite, which is also MoE). For brevity in tables, we abbreviate DeepSeekR1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, and DeepSeek-V2-Lite as DS-R1-Llama-8B, DS-R1-Qwen-7B, and DS-V2-Lite, respectively. Two models in the panel deviate from the standard GQA Q/K layout, so they need an extra step. GLM-4-9B fuses Q, K, and V into a single query_key_value projection matrix instead of the three separate matrices (q_proj, k_proj, v_proj) used by the other GQA models. This is an implementation-level fusion that leaves the attention math unchanged. We apply the fused projection, slice its output along the last dimension into Q, K, V, and feed Q and K through the same GQA averaging used elsewhere. DeepSeek-V2-Lite uses MLA, in which the K vector consumed by attention has two components: a content part recovered from a low-rank latent representation (not RoPE-rotated, identical across query heads) and a small decoupled RoPE-key that carries position through RoPE rotation (also shared across all query heads). Block-GTQ targets only RoPE-rotated keys, so the latent is outside its scope and the diagnostic uses only the decoupled RoPE-key path. In the panel table this path appears as one shared head with drope =64, treated as a single KV head common to all query heads.

Table 7: Ten-model panel used for the attention diagnostics and aggregate bit-allocation tables. “Geometry” reports number of layers, query/KV head counts, and per-head dimension; the last column notes each model’s role in the panel and any non-standard calibration handling. Model

Geometry

Role and calibration handling

Qwen2.5-3B Qwen2.5-14B-Instruct Qwen3-8B

36L, 16/2, dh =128 48L, 40/8, dh =128 36L, 32/8, dh =128

Qwen3-30B-A3B

48L, 32/4, dh =128

Llama-3.1-8B-Instruct DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B

32L, 32/8, dh =128 32L, 32/8, dh =128 28L, 28/4, dh =128 40L, 32/8, dh =128 40L, 32/2, dh =128

DS-V2-Lite

27L, 16/1, drope =64

Small-KV-head GQA stress case. Larger Qwen2.5 GQA. Dense Qwen3 GQA; calibrate after the model’s post-projection QK-RMSNorm. Sparse MoE Qwen3 GQA; calibrate after QK-RMSNorm. Llama-family GQA reference. Reasoning-distilled Llama GQA. Reasoning-distilled Qwen GQA. Non-Qwen/Llama dense GQA reference. GQA with fused QKV; slice fused projection into Q and K. MLA + MoE; uses the shared decoupled RoPE-key subspace.

B.2

Bit Allocation across Models

Aggregate distributions. Block-GTQ’s energy scores are calibrated on 2048 WikiText-2 train tokens (full protocol in Appendix B.4). At both the 3 b/dim and 2 b/dim budgets, every model produces a non-uniform allocation: the budget bit width is the mode, with nontrivial mass at lower and higher widths; the mode shifts from 3 to 2 bits between the two budgets for ten models. Tables 8 and 9 give the per-model percentages at each budget, and Figure 7 plots the per-layer fingerprint at the 2 b/dim budget. Table 8: Aggregate Block-GTQ bit-allocation distribution at the 3 b/dim budget (numeric counterpart of Figure 3). Each cell is the percentage of all (layer, head, frequency-block) triples in that model assigned the given bit width. Model

Arch

Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B-A3B Llama-3.1-8B DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

GQA GQA GQA MoE GQA GQA GQA GQA GQA MLA

Freqs 1b% 2b% 3b% 4b% 5b% 6b% 7b% 8b% 64 64 64 64 64 64 64 64 64 32

0.5 0.1 1.2 1.7 0.3 0.1 1.2 — 0.4 —

23.5 20.4 22.0 20.0 19.0 18.1 23.5 16.7 18.0 24.2

57.2 62.4 57.4 60.7 65.0 67.2 56.9 70.3 67.7 57.5

14.6 14.3 15.4 13.5 12.2 11.3 13.9 10.3 10.4 12.7

3.1 2.0 3.4 3.0 3.1 3.0 2.5 1.9 2.3 5.2

0.9 0.7 0.6 0.8 0.4 0.3 1.4 0.9 1.1 0.3

0.2 — 0.1 0.3 — — 0.3 — 0.1 —

— — — 0.1 — — 0.3 — — —

Table 9: Aggregate Block-GTQ bit-allocation distribution at the 2 b/dim budget (per-layer fingerprint in Figure 7). Each cell is the percentage of all (layer, head, frequency-block) triples in that model assigned the given bit width. Model

Arch

Freqs 1b% 2b% 3b% 4b% 5b% 6b% 7b% 8b%

Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B-A3B Llama-3.1-8B DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

GQA GQA GQA MoE GQA GQA GQA GQA GQA MLA

Qwen2.5-3B (H=1.52)

Qwen2.5-14B (H=1.43)

64 64 64 64 64 64 64 64 64 32

24.2 20.6 24.1 22.8 19.4 18.2 25.1 16.7 18.4 24.2

57.1 62.4 56.8 60.0 65.1 67.2 56.7 70.3 67.9 57.5

14.6 14.2 15.2 13.2 12.1 11.3 13.8 10.3 10.2 12.7

3.1 2.0 3.3 2.8 3.1 3.0 2.4 1.9 2.2 5.2

0.8 0.7 0.6 0.8 0.4 0.3 1.4 0.9 1.1 0.3

0.2 — 0.1 0.3 — — 0.3 — 0.1 —

— — — 0.1 — — 0.2 — — —

— — — — — — — — — —

Per-Layer Bit Distribution Fingerprint (2b avg budget) 1b 100

Qwen3-8B (H=1.56)

100

8b Qwen3-30B-A3B (H=1.51)

100

40 20

Layer

DS-R1-Llama-8B (H=1.35)

100

40 20

Layer

DS-R1-Qwen-7B (H=1.53)

100

% of blocks

40 20

Layer

Mistral-Nemo-12B (H=1.28)

100

40 20

Layer

GLM-4-9B (H=1.31)

100

20 0

Layer

40 20

Layer

% of blocks

40 20

Layer

DS-V2-Lite (H=1.53)

100

Llama-3.1-8B (H=1.40)

100

% of blocks

100

40 20

Layer

Figure 7: Allocator fingerprint at the 2 b/dim budget. Per-layer bit-width distribution for each model. The numeric counterpart is Table 9. Per-layer heterogeneity. The aggregate tables above show distributions at a single budget but hide how the allocation varies across layers within each model. For each layer ℓ we collapse the (ℓ) allocation over all (head, frequency-block) pairs into a histogram {nb }8b=1 and report two layer(ℓ) level statistics: the distinct-bit-width count grps(ℓ) = |{b : nb > 0}| and the Shannon entropy P (ℓ) (ℓ) H (ℓ) = − b pb log2 pb in bits (H=0 marks a single-bit-width layer; H ≈ 3 marks near-uniform coverage of all 8 widths). Table 10 reports the per-model mean and spread of both statistics at 3 b/dim, and Figure 8 plots H (ℓ) against normalized layer depth for each model. Every model uses multiple bit widths per layer (grps ∈ [4.0, 5.6]), the entropy curves typically oscillate around H ∈ [1.3, 1.6], and the most heterogeneous layer varies by model. Table 10: Per-layer bit-distribution summary across all ten models at 3 b/dim. grps is the mean number of distinct bit levels per layer; H is the mean per-layer Shannon entropy (bits); σH is its standard deviation across layers; Hmax (L) and Hmin (L) are the most/least heterogeneous layer indices, with the corresponding entropy value. Model

Arch

Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B-A3B Llama-3.1-8B DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

GQA GQA GQA MoE GQA GQA GQA GQA GQA MLA

Layers grps 36 48 36 48 32 32 28 40 40 27

4.9 5.0 5.4 5.6 5.0 4.9 5.6 5.0 4.8 4.0

σH

Hmax (L) Hmin (L)

1.55 1.44 1.60 1.56 1.42 1.35 1.58 1.28 1.33 1.53

0.24 0.16 0.17 0.19 0.13 0.14 0.32 0.15 0.23 0.16

2.32 (0) 1.98 (0) 1.94 (29) 2.18 (45) 1.64 (25) 1.63 (14) 2.43 (0) 1.57 (2) 2.07 (37) 1.80 (14)

0.98 (3) 1.12 (14) 1.32 (2) 1.16 (11) 1.04 (0) 1.03 (0) 1.16 (8) 0.95 (33) 0.87 (33) 1.14 (24)

Per-Layer Bit-Distribution Entropy Across Models (3b avg budget)

Shannon entropy of per-layer bit histogram (bits)

2.5

Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B-A3B Llama-3.1-8B

2.0

DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

1.5

1.0

0.5

0.0

0.2

0.4 0.6 Normalized layer depth (0 = first, 1 = last)

0.8

1.0

Figure 8: Per-layer Shannon entropy H (ℓ) of the bit-width histogram at 3 b/dim across normalized layer depth (0 = first layer, 1 = last layer). Each curve is one model; per-model means and extrema are listed in Table 10.

B.3

Per-Layer RoPE-Logit MAE

The per-layer RoPE-logit MAE between the original key k and its quantized reconstruction k̂, averaged over all KV heads at the layer, is ⊤ MAEℓ = Eh∈HKV Eg∈Gℓ (h) E(qℓ,g ,kℓ,h )∼Tℓ E∆∈D q⊤ ℓ,g R∆ kℓ,h − qℓ,g R∆ k̂ℓ,h , ℓ

where qℓ,g and kℓ,h are pre-RoPE query and key activations at layer ℓ, query head g, and KV head h (the analytic block-diagonal rotation R∆ is applied identically to clean and quantized keys); Gℓ (h) is the set of query heads served by KV head h (for the DS-V2-Lite MLA, HKV = 1 and kℓ,h is the single shared decoupled RoPE-key; for the partial-rotary GLM-4, R∆ rotates only the first 64 of the 128 key dimensions and is the identity on the remaining 64, which therefore contribute a static, offset-independent term); and D is a grid of 50 evenly spaced relative offsets in [−1024, 1024]. Architectural specifics for non-standard projections (Qwen3’s QK-RMSNorm, GLM-4’s fused QKV) are described in Appendix B.1. We compute MAEℓ independently for every (model, layer) pair under a K-only setting (V is unchanged). Block-GTQ’s frequency-block energy scores are fit on the first 2048 tokens of the WikiText-2 train split; MAEℓ is then evaluated on the first 2048 tokens of the WikiText-2 test split (TQ-MSE is data-free and needs no fit). Table 1 reports MAEℓ at the 3 b/dim budget; Table 11 repeats it at 2 b/dim, where Block-GTQ again wins all 367/367 layer comparisons with comparable relative reductions (absolute MAE rises at the tighter budget for both methods). Table 11: Per-layer RoPE-logit error at the 2 b/dim budget, K-only (appendix counterpart of Table 1, which is at 3 b/dim). Values are mean RoPE-logit MAE across model layers; lower is better. ∆ is the relative reduction versus TQ-MSE; “Wins” counts layers where Block-GTQ beats uniform TQ-MSE. Model Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B-A3B Llama-3.1-8B

B.4

TQ-MSE

Block-GTQ

∆

Wins

Model

13.25 8.20 11.46 12.97 7.49

6.30 5.15 5.88 5.89 5.02

+52.5% +37.2% +48.7% +54.6% +33.0%

36/36 48/48 36/36 48/48 32/32

DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

TQ-MSE

Block-GTQ

∆

Wins

6.77 25.20 6.75 16.31 11.31

4.54 5.01 4.42 9.90 7.25

+33.0% +80.1% +34.5% +39.3% +35.9%

32/32 28/28 40/40 40/40 27/27

Attention Diagnostics across Models

Test protocol. Test contexts are drawn from the held-out WikiText-2 test split. We forward the first 2048 tokens through the model as a single long-context sequence, and collect pre-RoPE Q/K at every transformer layer. Attention metrics are then computed via this process: each query position t ∈ {1025, . . . , 2048} attends to its full causal prefix {1, . . . , t − 1}, with RoPE attention logits 19

st,i = q⊤ t Rt−i ki formed analytically (the same rotation Rt−i is applied to clean and quantized keys). Each (model, method, bit rate) cell is averaged over all 1024 query positions. We report the no-buffer setting, where every cached key is read from its quantized representation. An fp16 recent-key buffer leaves the most-recent keys exact for every method; since attention places an outsized share of its mass on recent positions, a buffered comparison reflects that shared fp16 region more than the quantizer under test. We therefore isolate the K-quantizer with no buffer and represent KIVI by its buffer-free ScaleOnly variant (Appendix B.4). Calibration. The calibration sample (qcal , kcal ) is drawn from a 2048-token WikiText-2 train prompt. Each quantizer uses this sample differently: KIVI fits its initial per-channel scale on kcal in the no-buffer setting (see Appendix B.4); TQ-MSE is data-free; Block-GTQ computes the per-block energy score from (qcal , kcal ), then derives the per-block bit allocation and the same-rate group codebooks. Metrics. For a query q ∈ Rd and original/quantized context-key matrices K, K̂ ∈ RC×d√ , let s, ŝ ∈ RC be the original and quantized RoPE attention logit rows and p = softmax(s/ d), √ p̂ = softmax(ŝ/ d). We report two diagnostics: | top10 (s) ∩ top10 (ŝ)| Softmax KL = E KL(p ∥ p̂), Top-10 overlap = E . 10 Softmax KL is the divergence between the quantized and fp16 softmax distributions; because KL weights each token by its fp16 attention mass, errors at high-attention tokens dominate. Top-10 attended-token overlap reports the fraction of fp16’s ten most-attended tokens that the quantized version also ranks in its top-10. Table 12: Per-model softmax KL (↓, lower is better). Per-model values behind the panel-mean bars in Figure 4(a), with columns grouped by the 2, 3, and 4 b/dim budgets. KIVI refers to the no-buffer KIVI-ScaleOnly variant (Appendix B.4). Best per (model, budget) in bold; the last row is the panel mean. 2 b/dim Model

Block-GTQ

3 b/dim

KIVI TQ-MSE

Block-GTQ

4 b/dim

KIVI TQ-MSE

Block-GTQ

KIVI TQ-MSE

Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B Llama-3.1-8B DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

0.1636 0.0773 0.1210 0.1234 0.0569 0.0378 0.1229 0.0515 0.4546 0.5709

0.3114 0.1596 0.5126 0.2020 0.1298 0.1256 0.7481 1.9675 0.3492 0.2023

0.6359 0.2118 0.6349 0.8097 0.1444 0.1024 0.8782 0.1288 0.9397 1.3615

0.0444 0.0214 0.0327 0.0335 0.0151 0.0099 0.0271 0.0140 0.1617 0.1799

0.0989 0.0558 0.3594 0.0696 0.0614 0.0663 0.5569 1.8197 0.1188 0.0684

0.2544 0.0567 0.1960 0.2288 0.0367 0.0245 0.4840 0.0332 0.3450 0.4381

0.0121 0.0056 0.0091 0.0088 0.0041 0.0026 0.0104 0.0035 0.0546 0.0548

0.0680 0.0405 0.3338 0.0490 0.0510 0.0584 0.5326 1.7932 0.0829 0.0435

0.0859 0.0153 0.0594 0.0571 0.0098 0.0063 0.2915 0.0088 0.1072 0.1274

Mean

0.1780 0.4708

0.5847

0.0540 0.3275

0.2097

0.0166 0.3053

0.0769

Cross-method results. Per-model numbers behind Figure 4 are in Tables 12 and 13. Block-GTQ has the lowest mean softmax KL at every budget and jointly wins both axes—lowest softmax KL and highest top-10 overlap—on 7/10 models at 2 b/dim, 8/10 at 3 b/dim, and 9/10 at 4 b/dim. Relative to TQ-MSE, panel-mean softmax KL drops by 3.28 × / 3.88 × / 4.63× and panel-mean top-10 overlap rises by 12.8 / 8.9 / 5.6 percentage points at the three budgets. The advantage widens with the bit budget: as more bits become available, the non-uniform allocator routes incremental bandwidth to high-energy RoPE blocks that uniform-rate baselines cannot exploit; at the tight 2-bit budget all three methods absorb relatively similar quantization noise, so the gap is the smallest. Where KIVI is competitive. Block-GTQ beats TQ-MSE on every (model, bit-budget) cell—on both softmax KL and top-10—and beats KIVI-ScaleOnly on eight of the ten panel models. The remaining two, DS-V2-Lite and GLM-4-9B, are the architectures whose RoPE substructure is halfwidth: both leave the allocator only 32 RoPE-carrying frequency blocks, half the 64 of the standard GQA models. DS-V2-Lite uses MLA, whose single shared decoupled RoPE-key is 64-dimensional (32 blocks total); GLM-4-9B is partial-rotary, rotating only the first 64 of its 128 key dimensions, 20

Table 13: Per-model top-10 attended-token overlap (↑, higher is better). Per-model values behind the scatter in Figure 4(b), with columns grouped by the 2, 3, and 4 b/dim budgets. Best per (model, budget) in bold; the last row is the panel mean. 2 b/dim Model

Block-GTQ

3 b/dim

KIVI TQ-MSE

Block-GTQ

4 b/dim

KIVI TQ-MSE

Block-GTQ

KIVI TQ-MSE

Qwen2.5-3B Qwen2.5-14B Qwen3-8B Qwen3-30B Llama-3.1-8B DS-R1-Llama-8B DS-R1-Qwen-7B Mistral-Nemo-12B GLM-4-9B DS-V2-Lite

75.2% 71.6% 75.7% 72.3% 76.3% 69.4% 74.4% 72.6% 77.1% 77.2% 77.1% 76.3% 81.4% 72.9% 80.5% 69.5% 60.2% 61.7% 49.9% 68.0%

62.1% 64.8% 61.6% 55.2% 68.3% 68.2% 66.5% 72.5% 47.0% 33.9%

86.1% 82.1% 86.4% 82.4% 86.7% 78.5% 85.5% 82.5% 87.2% 85.0% 87.2% 84.6% 89.7% 82.3% 89.1% 76.9% 76.1% 75.8% 68.9% 81.2%

76.8% 79.3% 77.2% 72.5% 81.4% 81.5% 79.4% 84.2% 66.0% 55.4%

92.5% 85.7% 92.6% 85.6% 92.6% 81.1% 92.0% 85.5% 92.9% 87.3% 93.0% 87.0% 94.2% 85.1% 94.2% 78.8% 86.0% 80.8% 81.6% 85.9%

86.4% 88.3% 86.8% 84.1% 89.5% 89.6% 87.2% 91.2% 79.6% 73.3%

Mean

72.8%

60.0%

84.3%

75.4%

91.2%

85.6%

71.2%

81.1%

84.3%

so only 32 of its 64 blocks carry RoPE-frequency structure. With half the structure to differentiate, Block-GTQ’s RoPE-aware advantage shrinks and per-channel KIVI-ScaleOnly becomes competitive. The effect is decisive on DS-V2-Lite—KIVI wins both axes at every budget, by a wide margin at 2 b/dim (top-10 68.0% vs 49.9%)—but only partial on GLM-4-9B, where KIVI wins both axes at 2 b/dim and softmax KL at 3 b/dim before Block-GTQ recovers both by 4 b/dim. We attribute the persistence on DS-V2-Lite to its MLA geometry: the single decoupled RoPE-key is consumed by all 16 query heads, so its quantization error is shared layer-wide and the allocator has only those 32 blocks to work with; GLM-4-9B instead keeps a full 128-dimensional key, so once the budget loosens the allocator can spend the extra bandwidth on its non-rotary half and recover. Fair-comparison note on KIVI. KIVI as originally proposed ships with a 32-token fp16 residual buffer as an integral part of the method—every cached key passes through this buffer before being quantized. Our diagnostic uses a custom KIVI-ScaleOnly variant that retains KIVI’s per-channel rolling-scale quantizer but removes the residual buffer. KIVI-ScaleOnly is therefore not a deployment configuration; it exists only to make the K-quantizer comparable across methods. The above describes only the K side of KIVI-ScaleOnly. In the attention diagnostics (Section 6.1), V stays fp16, so this K-only variant is used directly. In the downstream tasks (Section 6.3: NIAH, LongBench, AIME) at K3V3/K3V2/K2V2 budgets, Block-GTQ, TQ-MSE, and KIVI-ScaleOnly share the same V quantizer (TQ-MSE); the K quantizer is where they differ.

Calibration Robustness

Block-GTQ’s bit allocation is computed from a per-RoPE-block energy score over a short calibration prefix; Algorithm 2 states the calibration procedure and Equation 4 below gives the GQA-aware score formula. This appendix quantifies the sensitivity of the resulting allocation to three calibration choices: prefix length, score function, and calibration corpus. GQA energy formula. Under grouped-query attention (GQA), one KV head is shared by multiple query heads. With Gℓ (h) denoting the layer-specific query-head group from Subsection 3.1 and N the number of calibration tokens, the Block-GTQ score sℓ,h,i for layer ℓ, KV head h, and RoPE block i is   N N X X X 2 2 1 1 1 (i) (i) . sℓ,h,i =  qℓ,g,t + k (4) 2 N |Gℓ (h)| t=1 N t=1 ℓ,h,t g∈Gℓ (h)

The Q-side term averages squared norms over the query heads served by the KV head. Averaging the Q vectors over heads before squaring would yield a strictly smaller value by Jensen’s inequality (with equality only when the heads are collinear), and would therefore systematically under-count the Q-side energy.

Algorithm 2 RoPE-block score calibration Require: Model M, calibration tokens X Ensure: Score vectors {sℓ,h,i } for every layer ℓ, KV head h, and RoPE block i 1: Run M on X and capture Q/K vectors used by RoPE attention 2: for each layer ℓ do 3: for each KV head h do 4: Identify the query-head group G(h) served by KV head h 5: Split each captured query and key head into RoPE blocks i = 1, . . . , L 6: for each RoPE block i do (i) 7: Average ∥qℓ,g,t ∥22 over tokens t and query heads g ∈ G(h) (i)

Average ∥kℓ,h,t ∥22 over tokens t 9: Set sℓ,h,i by Equation 4 10: end for 11: end for 12: end for 8:

C.1

Calibration length ablation

K2V2 is sensitive to the calibration length (Table 14): the curve is non-monotone—N = 64 (95.68) beats N = 128, 256, and 1024, and only N = 2048 wins cleanly (97.36); the per-task breakdown (same table) shows multi-query alone swinging 12 pp peak-to-trough across the smaller budgets (74.24 at N = 1024 to 86.70 at N = 2048), with the binary subtasks staying ≥ 91.92. K3V3 is much less sensitive—every N lies within 1.07 pp of N = 2048, and even m-query stays within 3.20 pp of the N = 2048 baseline. The K2V2 non-monotonicity comes from finite-sample noise in the per-block energy estimates: small Ncal flips roughly five of 64 marginal-gain comparisons per head, damped out only at N = 2048.

Table 14: Calibration length ablation, per-task NIAH pass-rate (%) on Llama-3.1-8B-Instruct. ∆2048 is the change in Overall vs N = 2048. At K2V2 the budget-noise effect concentrates on the fractional-scored subtasks, with m-query swinging 12 pp peak-to-trough (74.24 at N = 1024 vs. 86.70 at N = 2048); binary subtasks stay ≥ 91.92. At K3V3 every subtask is within ∼ 3.54 pp of the N = 2048 baseline. Budget Ncal

single

dist.

multi

m-key m-value m-query Overall ∆2048

K2V2 K2V2 K2V2 K2V2 K2V2 K2V2

64 128 256 512 1024 2048

100.00 100.00 100.00 100.00 100.00 100.00

96.46 98.48 96.46 97.98 98.48 98.99

97.98 97.47 91.92 96.46 94.95 100.00

95.29 95.79 95.96 97.47 92.59 98.48

98.48 96.97 99.16 99.49 99.33 100.00

85.86 76.94 80.98 83.33 74.24 86.70

95.68 94.28 94.08 95.79 93.27 97.36

−1.68 −3.08 −3.28 −1.57 −4.09 (base)

K3V3 K3V3 K3V3 K3V3 K3V3 K3V3

64 128 256 512 1024 2048

100.00 100.00 100.00 100.00 100.00 100.00

100.00 97.98 100.00 100.00 100.00 96.46 99.49 96.97 100.00 99.49 100.00 100.00

100.00 100.00 100.00 100.00 100.00 99.66

92.76 91.75 87.54 91.58 90.24 90.74

98.46 98.63 97.33 98.01 98.29 98.40

+0.06 +0.23 −1.07 −0.39 −0.11 (base)

C.2

Energy score ablation

We compare five energy score functions on Block-GTQ—the default qk_avg (Eq. 4) and four alternatives spanning symmetric aggregations and single-sided variants: qk_avg = 12 E∥q∥2 + E∥k∥2 , qk_max = max E∥q∥2 , E∥k∥2 , p qk_product = E∥q∥2 · E∥k∥2 , k_only = E∥k∥2 , q_only = E∥q∥2 . qk_max is another symmetric aggregator (the larger of the two squared norms); qk_product is their geometric mean; the two single-sided variants k_only and q_only drop one side of attention entirely and pin down which side carries the signal. All five variants share the same calibration—the first 2048 tokens of the WikiText-2 test split—and the same Block-GTQ allocator; we run NIAH on Llama3.1-8B-Instruct at the rate-sensitive K2V2 budget, where the score choice is most consequential (Table 15). The symmetric default qk_avg wins by Overall (97.36) and on most per-task columns. Table 15: Energy score ablation. Block-GTQ per-task NIAH pass-rate (%) on Llama-3.1-8B-Instruct at K2V2; per-column best in bold.

C.3

Variant

single

distract.

multi

m-key

m-value

m-query

Overall

qk_avg (default) qk_max k_only qk_product q_only

100.00 98.99 99.49 100.00 100.00

98.99 98.48 97.47 97.98 99.49

100.00 98.99 99.49 99.49 98.99

98.48 98.15 95.79 93.10 94.95

100.00 98.32 99.49 98.48 97.31

86.70 87.71 79.97 81.65 73.57

97.36 96.77 95.29 95.12 94.05

Calibration corpus ablation

To test whether the per-block energy ranking is sensitive to the calibration corpus, we compare four 2048-token calibration sources and re-evaluate Block-GTQ NIAH on Llama-3.1-8B-Instruct (Table 16): • WikiText-2 test (baseline): curated Wikipedia prose; first 2048 tokens of the WikiText-2 test split. • PG19: Project Gutenberg literary text in a comparatively older English register; first 2048 tokens of the HuggingFace pg19 train split. • C4: heterogeneous web text from Common Crawl—prose interleaved with boilerplate, URLs, and navigation fragments; first 2048 tokens of the HuggingFace c4 en validation split. • Code: Python source code from the CPython 3.11.0 standard library—first 2048 tokens of a concatenation of argparse.py, json/encoder.py, and json/decoder.py fetched from the cpython GitHub repository. NIAH evaluates retrieval over long passages of natural English prose (“haystacks”) with short factual “needles” inserted at varying depths. The four corpora span an ordered range of distance from this deployment distribution: WikiText-2 and PG19 are both natural prose (closest); C4 is mostly prose interleaved with web artifacts; code is structurally different in both surface form and distributional statistics (furthest). K2V2 is sensitive to that distance (Table 16): PG19 stays within 0.28 pp of WikiText-2, C4 drops 2.78 pp, and code drops 3.70 pp Overall—monotonic in how far the calibration diverges from prose. K3V3 is much less sensitive; all four corpora are within 0.34 pp of one another, 11–16× tighter than at K2V2. P This separation reflects the 4−b rate law in Block-GTQ’s allocator objective i si · 4−bi . When an off-domain calibration shifts the per-block energy ranking, the allocator misplaces some bits; the cost of each misplacement (e.g., assigning b where b+1 would have been better) is si (4−b − 4−(b+1) ), 23

which at K2V2 (average b=2) is si (4−2 − 4−3 ) and at K3V3 (average b=3) only si (4−3 − 4−4 )—a factor of ∼ 4 smaller per misplaced bit. The same calibration-induced ranking shift therefore translates into a ∼ 4× smaller objective penalty at K3V3, and the observed 11–16× NIAH swing reduction is the downstream manifestation of this rate-law amortization. Table 16: Calibration corpus ablation, per-task NIAH pass-rate (%) on Llama-3.1-8B-Instruct. ∆ is the change in Overall vs the WT2 baseline.. Budget Corpus

C.4

∆

single

dist.

multi

m-key m-value m-query Overall

K2V2 K2V2 K2V2 K2V2

WT2 (baseline) 100.00 PG19 99.49 C4 100.00 code 98.99

98.99 97.47 98.99 97.47

100.00 100.00 100.00 96.46

98.48 98.65 90.40 94.28

100.00 99.83 98.82 95.96

86.70 87.04 79.29 78.79

97.36 97.08 94.58 93.66

— −0.28 −2.78 −3.70

K3V3 K3V3 K3V3 K3V3

WT2 (baseline) PG19 C4 code

100.00 100.00 100.00 100.00 100.00 98.99 100.00 98.32 100.00 100.00 100.00 98.99 100.00 100.00 100.00 99.49

99.66 100.00 99.66 100.00

90.74 91.41 90.74 88.89

98.40 98.12 98.23 98.06

— −0.28 −0.17 −0.34

Cross-model PPL and allocation-distance diagnostics

We run Block-GTQ at Ncal ∈ {128, 512, 2048} on Llama-3.1-8B-Instruct and DeepSeek-R1-DistillQwen-7B at both K3V3 and K2V2. For each cell we draw three calibration prefixes from WikiText-2 train at offsets 0, 10,000, and 20,000 tokens (different articles, three near-independent draws). We evaluate using four metrics; (ii)–(iv) compare each perturbed allocation b against the Ncal = 2048, seed-0 reference bref over all (layer, KV head, RoPE-block) triples I: • (i) Output PPL: sliding-window perplexity on the full WikiText-2 test set (C = 4096, S = 512, ∼ 99K tokens), reported in main-text Table 3. PPL captures robustness at the output level but cannot tell whether the allocation itself is stable or whether it moves with small objective cost—hence (ii)–(iv) below. • (ii) Hamming distance, the fraction of slots whose bit value changed: X 1 Hamm(b, bref ) = 1 bℓ,h,i ̸= bref ℓ,h,i . |I|

(5)

(ℓ,h,i)∈I

• (iii) High-bit Jaccard at threshold 4, measuring the overlap of the slots the allocator protected with ≥ 4 bits: HB@4(b, bref ) =

{(ℓ, h, i) : bℓ,h,i ≥ 4} ∩ {(ℓ, h, i) : bref ℓ,h,i ≥ 4} {(ℓ, h, i) : bℓ,h,i ≥ 4} ∪ {(ℓ, h, i) : bref ℓ,h,i ≥ 4}

(6)

• (iv) Energy-weighted regret, the cost in the allocator’s own objective with each change −b weighted by importance sref i and bit magnitude 4 : P ref ref −bℓ,h,i − 4−bℓ,h,i ) (ℓ,h,i)∈I sℓ,h,i (4 Regret(b) = . (7) P −bref ref ℓ,h,i (ℓ,h,i)∈I sℓ,h,i 4 The two non-reference N = 2048 seed cells differ from the reference only in the random WikiText-2 slice, so their disagreement with the reference defines a within-source noise floor for each metric. At K3V3, the within-source noise floor (two non-reference N = 2048 seeds in Table 17) is Hamming 0.07–0.08, HB@4 0.86–0.93, regret +0.4–0.9% on both models. N = 128 pushes Hamming to 1.4–2.7× the floor and drops HB@4 by 10–13 pp (about 27% of the high-bit tail reshuffled on Llama), yet regret stays at 1.2–3.1%: the allocation visibly moves above what calibration randomness alone explains. This is the 4−b rate law in action—each misplaced bit at b = 3 costs roughly 4× less than at b = 2, so the same allocator movement amortizes into ≤ 3.1% regret at K3V3.

Table 17: Allocation distance against the Ncal = 2048, seed-0 reference allocation, K3V3. Hamming counts changed bit slots (Eq. 5); HB@4 is the high-bit Jaccard at threshold 4 (Eq. 6); Regret is Eq. 7. The 2048-token non-reference seeds define the within-source noise floor. Ncal

Model

seed

Hamming

HB@4

Regret

0.730 0.776 0.736 0.845 0.844 0.831 1.000 0.855 0.870

+2.99% +2.15% +2.78% +0.80% +0.97% +1.09% +0.00% +0.89% +0.73%

DeepSeek-R1-Distill-Qwen-7B (K3V3) 128 0 0.141 0.827 128 1 0.112 0.877 128 2 0.178 0.802 512 0 0.065 0.926 512 1 0.084 0.915 512 2 0.084 0.912 2048 0 0.000 1.000 2048 1 0.073 0.918 2048 2 0.066 0.931

+1.89% +1.22% +3.11% +0.45% +0.67% +0.70% +0.00% +0.49% +0.44%

Llama-3.1-8B-Instruct (K3V3) 128 0 0.148 128 1 0.124 128 2 0.141 512 0 0.077 512 1 0.083 512 2 0.089 2048 0 0.000 2048 1 0.081 2048 2 0.074

Downstream Evaluation Details

D.1

Long-Context Tasks

D.1.1

NIAH Protocol

fp16

KIVI-ScaleOnly

Block-GTQ (2k calib)

TQ-MSE

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

1.00

0.67

1.00

0.67

0.00

1.00

0.67

1.00

0.33

0.00

1.00

0.67

0.33

0.00

1.00

0.67

0.00

1.00

0.33

0.00

1.00

0.33

1.00

0.33

0.67

0.33

0.00

1.00

0.33

1.00

0.67

0.33

0.67

0.33

0.00

1.00

0.33

1.00

0.00

1.00

0.33

0.67

0.00

1.00

0.00

1.00

0.67

1.00

0.33

0.00

1.00

0.33

0.67

0.33

0.00

1.00

0.33

1.00

0.33

0.00

1.00

0.00

1.00

0.00

1.00

0.00

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

1.00

0.67

1.00

0.67

0.00

1.00

0.67

0.00

0.33

0.00

1.00

0.33

1.00

0.67

0.00

1.00

0.67

1.00

0.67

0.33

1.00

0.33

0.00

1.00

0.67

0.33

1.00

0.33

0.00

1.00

0.33

1.00

0.33

1.00

0.67

0.00

1.00

0.00

1.00

0.00

0.33

0.67

0.00

0.33

0.00

1.00

0.00

1.00

0.00

0.33

0.00

1.00

0.00

1.00

0.00

1.00

0.67

0.00

1.00

0.33

0.00

0.33

0.00

1.00

0.00

1.00

0.00

1.00

0.75

pass rate

K3V3 needle depth

The main text shows the Llama-3.1-8B-Instruct single-needle heatmap (Figure 5) and the combined Llama / Qwen multi-task scores aggregated across six NIAH variants (Table 4). This appendix adds the matching Qwen2.5-7B-Instruct heatmap (Figure 9) and details the protocol and subtask definitions.

K2V2 needle depth

0.50

1.00

0.00

0.33

0.00

16K

32K

64K 128K

context length

0.33

0.00

16K

32K

64K 128K

context length

0.00

1.00

16K

32K

64K 128K

context length

1.00

0.67

16K

32K

64K 128K

context length

0.25

0.00

Figure 9: NIAH single-needle retrieval on Qwen2.5-7B-Instruct. Pass rate is shown over context length (4K–128K) and needle depth (0%–100%), averaged over three trials per cell. The two rows use the same method layout as Figure 5: fp16, KIVI-ScaleOnly (Appendix B.4), TQ-MSE, and Block-GTQ. Top row: K3V3. Bottom row: K2V2. The fp16 panel is identical across budgets for a fixed model and serves as the retrieval ceiling.

Calibration. Block-GTQ is calibrated on the first 2048 tokens of the WikiText-2 test split, concatenated as a single contiguous raw-text stream with article headings stripped. TQ-MSE is dataindependent; KIVI-ScaleOnly is defined in Appendix B.4. Subtasks. The six NIAH variants in Table 4 stress different facets of long-context retrieval. Across all six, the haystack is filler text into which one or more synthetic key–value needles are inserted; the model receives the haystack plus a query and must return the matching value(s). Variants differ in needle count, distractor structure, and query structure. • single: one key–value needle is inserted at a given depth and the model is queried for its value. Tests basic retrieval—the model must locate the needle by key and return the corresponding value. • distract.: one target needle is inserted alongside several distractor key–value pairs with similar formatting (but unrelated to the query). Tests discrimination against same-format distractors: the model must not be misled by lookalike but incorrect needles. • multi: several distinct needles are inserted in the haystack, and the model is queried for one specific value. Tests selective retrieval when multiple plausible candidates exist. • m-key: three distinct key–value needles are placed close together in the haystack, and the model is queried for each key in turn. Tests fine-grained key discrimination among nearby needles—the model must not conflate adjacent key–value pairs. • m-value: several values are bound to a single entity, and the query requires returning all of them. Tests recall completeness: partial answers are penalized. • m-query: several distinct queries are run against a haystack holding multiple needles. Tests robustness across multi-fact recall—the score is averaged over all queries. The first three tasks are scored 0/1 (the model either returns the correct value or not); the last three (m-key, m-value, m-query) are scored as the fraction of correct answers among multiple expected responses. Sampling. Each (task, context length, depth) cell averages three haystack samples (random filler text, fixed needles) over six context lengths (4K–128K) and eleven needle depths {0%, 10%, . . . , 100%}. All methods and bit budgets share the same needle set, so cross-method and cross-budget comparisons are paired on identical needle facts. D.1.2

LongBench-EN Protocol

The LongBench-EN table in Section 6.3.1 uses Llama-3.1-8B-Instruct on eight subtasks spanning single-document QA, multi-document QA, summarization, few-shot classification, synthetic retrieval, and code completion. Calibration. Block-GTQ uses the same calibration as for NIAH: the first 2048 tokens of the WikiText-2 test split. TQ-MSE is data-independent; KIVI-ScaleOnly is defined in Appendix B.4. Subtasks. Each subtask is listed below with its column abbreviation, scoring metric, and outputtoken cap. Metrics follow LongBench [1]: QA-F1 is token-level F1 between predicted and reference answers; ROUGE-L is LCS-based ROUGE for summarization; classification score and retrieval score are answer accuracy; edit similarity is edit-distance-based similarity for code. The output-token cap is the maximum number of tokens the model may generate per example, set by LongBench per task. • qasper (Qasp; QA-F1, 128-token cap): single-document QA over NLP research papers (arXiv NLP papers as input). Questions require extracting specific factual details from a paper-length input, often spanning multiple sections (methodology, results, related work). • multifieldqa_en (MFQA; QA-F1, 64-token cap): single-document QA over English-language documents from diverse domains (legal, government, encyclopedic, etc.). Questions require locating specific information in a long structured document. 26

• hotpotqa (HP; QA-F1, 32-token cap): multi-document QA from HotpotQA, requiring 2-hop reasoning across multiple Wikipedia paragraphs. The model must locate facts in different paragraphs and chain them to produce the answer. • 2wikimqa (2W; QA-F1, 32-token cap): multi-document QA from 2WikiMultiHopQA, with multi-hop bridges between Wikipedia articles—similar to hotpotqa but with explicit entity-bridge reasoning chains. • gov_report (Gov; ROUGE-L, 512-token cap): abstractive summarization of long U.S. government reports (often several thousand to tens of thousands of tokens). The model must produce a faithful compressed summary covering key findings. • trec (TREC; classification score, 64-token cap): few-shot question-type classification using the TREC label set. The model sees many in-context exemplars and must classify a test question into one of 50 fine-grained categories (e.g., ABBR:abbreviation, NUM:date). • passage_retrieval_en (Pass; retrieval score, 32-token cap): synthetic retrieval—given a paraphrased question and a set of candidate Wikipedia passages, the model must identify which passage contains the answer by outputting its index. • lcc (LCC; edit similarity, 64-token cap): line-level code completion over long source files (often > 10K tokens). The model sees a file with the last line removed and must reproduce that line, requiring understanding of surrounding code structure. Inference. Inputs are middle-truncated to at most 31,500 tokens and decoded greedily, with pertask output caps as listed above. The Avg column in Table 5 is the unweighted mean over the eight subtasks. D.2

Reasoning Tasks (AIME)

This appendix is organised in three parts: the AIME protocol, the buffer configurations of the two regimes, and the per-method calibration recipes that produced the numbers in Section 6.3.2. Protocol. All AIME runs use the K3V2 cache budget (K at 3 bits/dim, V at 2 bits/dim). Generation is stochastic at temperature 0.6 and top-p 0.95, with a 32,768-token output cap. For each problem we draw eight samples and report pass@1 (avg@8). PM-KVQ is run through its official code1 ; KIVI is run with its official quantization scheme2 in the protected regime, and the KIVI-ScaleOnly variant (Appendix B.4) in the no-buffer regime. Buffer configurations. During decoding the KV cache is conceptually laid out as [ sink (fp16) | compressed | recent (fp16) ]; the two regimes differ only in sink and recent-window sizes. • In the protected-buffer regime we keep the first 4 tokens as fp16 sink and the most recent 128 tokens as fp16 recent. The 128-token recent span matches PM-KVQ’s protected configuration [21]; the 4-token sink follows the attention-sink convention of Xiao et al. [37], overriding PM-KVQ’s native default of sink = 1 so that the same protected allowance is applied uniformly across methods. TQ-MSE and Block-GTQ, which are buffer-free by design, run under this 4/128 allowance. KIVI uses its default path: per-(T =32, channel) asymmetric quantization for K and per-(token, D=32) asymmetric quantization for V. KIVI’s native 128-token fp16 residual coincides with the shared recent window, and its 32-token grouping is the K quantization group size along the token axis, not an additional fp16 buffer. • In the no-buffer stress regime we set both sink and recent windows to 0, so every attended token is served from the compressed cache. KIVI is replaced by the KIVI-ScaleOnly variant: K uses per-channel quantization with a 32-token rolling buffer of fp32 statistics for scale refresh (the statistics never enter the attention path), and V uses TQ-MSE. PM-KVQ is 1 https://github.com/thu-nics/PM-KVQ 2 https://github.com/jy-yuan/KIVI. The upstream packed kernel requires bit-width ∈ {2, 4, 8}; at the 3-bit budgets

used here we substitute a bit-exact round-trip that reproduces KIVI’s quant–dequant numerics.

run with neither sink nor sliding window, so no token is kept at high precision. Each K/V is quantized on arrival with per-group (128-channel) asymmetric quantization; following PM-KVQ’s progressive scheme, the cache enters at 16-bit and a layer’s entire cache is halved in bit-width (16→8→4→2) whenever it exceeds its calibrated per-layer memory budget, so the effective precision decreases as the sequence grows. TQ-MSE and Block-GTQ already operate without any buffer. Calibration. • Block-GTQ: calibrated on the first 2048 tokens of the WikiText-2 test split; see Appendix C for the energy score and bit-allocation procedure. • PM-KVQ [21]: a progressive mixed-precision quantizer whose calibration produces two offline artifacts from a single PI calibration set: (i) kv_budgets—a per-layer memory budget (shared by K and V), obtained by integer programming over loss-gradient sensitivity with bit choices {4, 2} and a 2.5 b/d average target (matching our K3V2 average); (ii) rep_scales—a SmoothQuant-style per-channel pre-scaling folded into k_proj/q_proj, obtained via a three-stage offline search and disabled in all our PM-KVQ runs. The PI calibration set is 512 sequences of 2048 tokens (about 1M tokens, more than two orders of magnitude beyond the 2048 tokens Block-GTQ uses) randomly sampled from the WikiText2 train split, with position-ids stretched by stride 4 to an effective length of 8192. • KIVI [24]: tuning-free—per-channel K scales and per-token V scales are computed online from running statistics, with no calibration data. We use KIVI as-is in the protected regime. In the no-buffer regime we substitute KIVI-ScaleOnly. Since the very first tokens are quantized before any rolling statistics exist, we seed the estimator’s per-channel K scale from a single forward pass over the first 64 tokens of the same wikitext prefix used by Block-GTQ; this seed only governs the first 32-token group of each sequence, after which the scale is fully refreshed online from the 32-token rolling fp32-statistics buffer. • TQ-MSE [41]: data-independent.

Deployment Protocol and Extended Results

This appendix gives the measurement protocol and numerical details behind Section 6.4. The deployment benchmarks run Qwen2.5-3B-Instruct on a single H800 80GB GPU and compare BlockGTQ and uniform TQ-MSE against an fp16 FlashAttention-2 (FA-2) baseline. Block-GTQ metadata uses the first 64 tokens of the WikiText-2 train as its calibration prefix; decode latency is the median per-step time over 20 timed autoregressive steps on T consecutive WikiText-2 input tokens, and peak memory includes model weights, the resident cache, and transient activations/buffers. All three methods run the same Qwen2.5-3B-Instruct in fp16, with identical fp16 weights and fp16 attention compute, so the only quantity that varies across methods is the KV-cache representation: a dense f16 cache for the baseline versus the packed low-bit codes of uniform TQ-MSE and Block-GTQ, both quantized from the same fp16 keys and values emitted by the model’s projections. The comparison is thus dtype-matched—the reported latency, memory, and perplexity differences are attributable to the cache representation alone, not to any change in weight or attention precision. The public Qwen2.5-3B-Instruct checkpoint is released in bf16; we cast it to fp16 uniformly for every method, including the baseline, so the dtype choice advantages none of them. E.1

Allocated Footprint Accounting

The deployed cache footprint should be read as an allocated tensor footprint, not as the ideal number of information bits. For D = 128, a pure 3-bit code-only K+V cache would use 48 + 48 = 96 bytes per token and KV head, giving an ideal 512/96 = 5.33× compression relative to fp16. The deployed K3V3 path allocates ≈ 157 bytes per token and KV head (Table 18); the ≈ 61-byte overhead is dominated by two sources: (i) per-coordinate bit widths are rounded up to nibble (4-bit) or byte (8-bit) storage so that GPU decoding stays bit-shift–free, and (ii) each same-rate group carries an fp16 normalization scalar. The overhead is a physical layout cost of serving from a packed cache, not a change to the Block-GTQ allocation objective.

Table 18: Allocated footprint per token and KV head. Qwen2.5-3B-Instruct K3V3 deployment, in bytes.

E.2

Cache representation

Allocated bytes

Compression

FP16 KV cache Ideal 3-bit code-only Deployed K3V3

512 96 ≈ 157

1.00× 5.33× ≈ 3.26×

Decode Latency and Memory: Three-Way Comparison

Tables 19 and 20 give the numerical data behind Fig. 6. The two compressed paths share the same packed-cache interface: uniform TQ-MSE applies a uniform 3-bit K budget, while Block-GTQ assigns K bits by RoPE block. Table 19 reports per-step decode-latency statistics, prefill time, and the median-latency decode speedup over fp16 FA-2 across five context lengths T ; Table 20 reports KV-cache footprint, KV compression ratios, and total/non-weight peak GPU memory at the same T . Table 19: Decode latency and prefill time. Qwen2.5-3B-Instruct K3V3 deployment on a single H800, comparing fp16 FA-2, uniform TQ-MSE, and Block-GTQ. Med., Mean, p5, p95 are the median, mean, and 5th/95th percentiles of per-step decode latency over 20 timed autoregressive steps. Prefill is the wall-clock time to construct the cache for the full prompt: fp16 prefill is FA-2-backed; the compressed paths build the packed cache. Med. speedup vs. fp16 FA-2 is the ratio of fp16 FA-2’s median decode latency to the method’s median decode latency. T Method

Med. Mean ms/step ms/step

p5 ms

p95 Prefill Med. speedup ms s vs. fp16 FA-2

16K FP16 FA-2 TQ-MSE Block-GTQ

17.84 45.47 45.86

17.86 45.46 45.83

17.76 45.32 45.66

18.01 45.61 46.01

0.37 0.87 1.33

1.00× 0.39× 0.39×

64K FP16 FA-2 TQ-MSE Block-GTQ

36.15 45.58 45.57

36.15 45.59 45.56

36.12 45.41 45.44

36.17 45.68 45.63

2.84 9.88 16.62

1.00× 0.79× 0.79×

128K FP16 FA-2 TQ-MSE Block-GTQ

70.96 45.53 52.95

70.95 45.63 60.52

70.85 71.03 45.39 45.97 52.69 121.72

9.09 36.41 63.53

1.00× 1.56× 1.34×

256K FP16 FA-2 OOM TQ-MSE 60.95 Block-GTQ 82.24

60.97 82.27

60.76 81.97

61.21 141.50 82.61 250.17

— —

512K FP16 FA-2 OOM TQ-MSE 101.18 101.22 101.02 101.45 558.75 Block-GTQ 140.27 140.40 140.04 140.86 997.19

— —

At short context (T ≤ 64K), in-kernel decoding adds a per-step overhead that outweighs the KV-bandwidth savings, so fp16 FA-2 is fastest. As T grows, KV-bandwidth dominates and the compressed paths overtake fp16 at T = 128K in median per-step decode latency: Block-GTQ runs 1.34× faster than fp16 FA-2, uniform TQ-MSE 1.56×. Beyond this, fp16 OOMs because peak total memory exceeds 80 GB. Uniform TQ-MSE is consistently faster than Block-GTQ in our current implementation because its K layout is simpler, but this speed advantage comes at a steep quality cost: as Appendix E.3 shows, TQ-MSE’s PPL collapses across all tested context lengths while Block-GTQ stays close to fp16, identifying Block-GTQ as the preferred operating point. Block-GTQ uses slightly more resident KV memory than uniform TQ-MSE—mixed-rate K allocation carries additional per-segment metadata—but both compressed paths reduce the KV footprint by roughly 3.4× in K3V3 budget relative to fp16. Figure 6(c) shows the peak-memory curves for the two compressed paths nearly overlap. From Table 20, their entire peak-memory difference comes from the KV-cache gap (at most 0.98 GB at T = 512K)—all other peak components (weights and transient activations) are identical between the two.

Table 20: KV footprint and peak GPU memory. Same Qwen2.5-3B-Instruct K3V3 deployment as Table 19. KV comp.: ratio of fp16’s KV cache size to this method’s. Peak total: total GPU memory (model weights + KV cache + transient activations/buffers). Peak minus weights: peak total minus the 6.17GB Qwen2.5-3B-Instruct weights, exposing non-weight memory. T Method

E.3

KV cache KV comp. Peak total Peak minus weights MB GB GB

16K FP16 FA-2 TQ-MSE Block-GTQ

604.0 155.7 186.4

1.00× 3.88× 3.24×

12.53 7.94 7.97

6.36 1.77 1.80

64K FP16 FA-2 TQ-MSE Block-GTQ

2415.9 622.9 745.5

1.00× 3.88× 3.24×

31.29 12.94 13.06

25.12 6.77 6.89

128K FP16 FA-2 TQ-MSE Block-GTQ

4831.8 1245.7 1491.1

1.00× 3.88× 3.24×

56.31 19.60 19.85

50.13 13.43 13.67

256K FP16 FA-2 OOM TQ-MSE 2491.4 Block-GTQ 2982.2

3.88× 3.24×

32.93 33.42

26.76 27.25

512K FP16 FA-2 OOM TQ-MSE 4982.8 Block-GTQ 5964.3

3.88× 3.24×

59.58 60.56

53.41 54.39

Perplexity at Long Context

Under the same deployment setting (Qwen2.5-3B-Instruct, K3V3), for each context length T , we feed T consecutive WikiText-2 tokens into the model as input and score its perplexity on the next 1000 tokens—the same 1000 positions for all three methods. A larger nppl reduces token-averaging noise. Calibration (tokens [0, 64)) and PPL evaluation (tokens [T, T + 1000), with T ≥ 4096) read from the same WikiText-2 train stream but use non-overlapping windows, so the setup is leakage-free. Table 21: Long-context perplexity. Same Qwen2.5-3B-Instruct K3V3 deployment as Table 19. PPL on the 1000 tokens following T consecutive WikiText-2 train input tokens; all three methods score the same 1000 positions. ∆: Block-GTQ’s PPL increase relative to fp16. TQ-MSE/Block-GTQ: ratio of the two PPLs. T

FP16 FA-2

Block-GTQ

TQ-MSE

∆ BGT vs. fp16

TQ-MSE/Block-GTQ

4K 16K 64K 128K 256K 512K

6.48 7.84 7.47 16.67 — (OOM) — (OOM)

6.66 8.12 7.72 16.94 608.11 1,216.11

352.25 299,419.12 11,477.91 13,450.49 24,517.60 16,682.48

+2.8% +3.6% +3.3% +1.6% — —

53× 36,879× 1,486× 794× 40× 14×

Qwen2.5-3B-Instruct supports context up to 128K with YaRN extension. For T ≤ 128K, BlockGTQ’s PPL stays within 1.6%–3.6% of fp16 (∆ column). At T = 256K and 512K, fp16 runs out of memory, and both packed paths’ PPL values rise sharply—this reflects the model itself failing to extrapolate beyond its supported context, not cache compression. By contrast, TQ-MSE’s PPL is 14× to 36,879× higher than Block-GTQ’s at every T (TQ-MSE/Block-GTQ column), including T = 4K well within the supported range—a failure of uniform K allocation, not of context length. For instance, at T = 16K, TQ-MSE’s PPL collapses to 299,419, while Block-GTQ’s PPL is 8.12, close to fp16’s 7.84.

Record · ID 303238 · SHA-256 7dde7ff45103c79b
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.