DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation Tan T. Nguyen∗ , and Quan V. Dang∗ ,† ∗ Full Stack Data Science † Department of Computer Science, University College London
Emails: [email protected], [email protected]
arXiv:2607.24331v1 [cs.LG] 27 Jul 2026
Abstract—As the inference phase of Large Language Models (LLMs) requires handling long context windows, the KeyValue (KV) cache initially appears to address this challenge but eventually becomes a significant bottleneck as the context window continues to grow. Low-rank compression has recently been studied as an effective approach to reduce KV cache memory while maintaining model performance. However, only a few existing methods treat the Key and Value caches differently, despite their distinct roles. Moreover, these methods typically employ fixed attention-head grouping, which may not fully exploit the structural similarity among attention heads. In this paper, we propose an improved low-rank KV cache compression framework. For the Key cache, we dynamically group attention heads based on Centered Kernel Alignment (CKA) similarity and allocate the rank budget adaptively under a parameter budget. For the Value cache, we adopt the same approach as ReCalKV, refining the low-rank decomposition through offline calibration to improve reconstruction quality. Experimental results on three instructiontuned LLMs show that our method reduces the number of Key cache parameters while maintaining competitive accuracy. We further observe that the proposed strategy is particularly effective for Multi-Head Attention (MHA) models, whereas it should be applied more conservatively to Grouped-Query Attention (GQA) models, especially in long-context settings. Index Terms—Large Language Models, Key-Value Cache Compression, Low-Rank Decomposition.
I. INTRODUCTION Large Language Models (LLMs) have recently demonstrated a significant role across almost all natural language tasks. A considerable factor in the success of these LLMs is the context window, which encompasses all the information, including prompts, chat histories, and uploaded documents, that the LLMs can read, remember, and process to generate a response. As real-world language tasks become more complex, the context window must also become larger, requiring the capability to handle long-context windows. As a result, memory consumption during inference has become a critical bottleneck. One of the most famous approaches to addressing this is the Key-Value (KV) cache, which stores the attention representations of previously generated tokens. By doing so, it avoids repeated computation; however, the memory still grows linearly with the sequence length, becoming a new obstacle. To handle this issue, numerous KV cache compression methods have been proposed. Previous approaches can generally be categorized into quantization-based methods, token eviction methods, and low-rank compression methods. Among these, low-rank approaches are particularly attractive because
they preserve the entire context while substantially reducing memory usage. Nevertheless, only a few recent works focus on the differences between the behaviors of Key and Value caches, which require distinct compression strategies; moreover, these existing methods still rely on simplifying assumptions. For instance, ReCalKV [1] groups attention heads using a fixed grouping strategy and allocates the rank budget uniformly across all groups. However, attention heads may exhibit varying degrees of similarity, making fixed grouping suboptimal and motivating a dynamic grouping strategy with adaptive rank allocation. In this paper, we propose an improved low-rank KV cache compression framework that answers these questions. For the Key cache, we dynamically group attention heads based on their similarity measured by Centered Kernel Alignment (CKA), and subsequently allocate the rank budget adaptively according to the importance of each group, while guaranteeing that the number of parameters does not exceed that of previous methods. For the Value cache, we decompose the entire original projection matrix rather than processing it in groups, and then refine the decomposed factors through another procedure that minimizes the reconstruction error on calibration data. Various experiments on language models indicate that the effectiveness of dynamic head grouping depends on the underlying attention architecture. While the proposed method is highly suitable for standard MHA models, it should be applied more conservatively to GQA models in long-context settings. The main contributions of this work are summarized as follows: • We propose DynaCalKV, a dynamic Key cache compression framework based on CKA similarity and a subsequent adaptive rank allocation algorithm, which preserves the parameter budget comparing to the previous methods. • Experimental results on multiple LLMs and datasets show that DynaCalKV improves memory efficiency while maintaining competitive performance, and reveal that the proposed strategy is particularly effective for MHA models but should be applied more cautiously to GQA models. II. RELATED WORK KV Cache Compression. To reduce the memory required for the KV cache during LLM inference, especially in long-
context scenarios, numerous methods have been proposed. Quantization-based methods, such as KIVI [2], KVQuant [3], aim to reduce the size of KV cache representations by using lower-bit numerical formats. Token eviction methods, such as H2 O [4], SnapKV [5], CAKE [6], discard tokens that are deemed unimportant or redundant. Notably, these two methods can be applied orthogonally to another group of methods, low-rank compression methods. These methods reduce the dimensionality of KV representations through lowrank projections or matrix decomposition techniques. Low-Rank KV Cache Compression. Among those categories, quantization-based and token eviction approaches have received more attention, whereas low-rank compression methods represent a promising research direction with considerable potential for future development. MatryoshkaKV [7] and Eigen-Attention [8] learn additional projection matrices to map the KV cache into a lower-dimensional latent space. In contrast, LoRC [9] and Palu [10] directly utilize singular value decomposition (SVD) to represent the KV cache using smaller matrices instead of the original one. These studies reveal the effectiveness of low-rank compression for KV cache reduction. Different strategies for Key and Value in Low-Rank KV Cache Compression. As research in this area continues to evolve, a promising approach that has not yet been broadly explored is to treat the Key cache and Value cache differently, given their distinct roles within the transformer architecture. Most studies following this line of research share the observation that the Key cache can be compressed more aggressively than the Value cache, based on the assumption that the Key primarily serves as an index for attention, whereas the Value contains the complete information. Both AsymKV [11] and ReCalKV [1] propose compressing the Key cache using static groups while handling the Value cache without grouping. In a different approach, Thin Keys, Full Values [12] focuses only on the Key cache while leaving the Value cache completely untouched. Unlike prior works, our method introduces dynamic grouping for the Key cache while also compressing the Value cache instead of leaving it unmodified. III. METHODOLOGY A. Preliminary Singular Value Decomposition. Singular Value Decomposition (SVD) is a popular matrix factorization method, which is widely adopted in constructing low-rank approximations of matrices. For any matrix W ∈ Rm×n , SVD decomposes it into three matrices, W = U ΣV ⊤ , where U ∈ Rm×m and V ∈ Rn×n are orthogonal matrices containing the left and right singular vectors, respectively, while Σ ∈ Rm×n is a diagonal matrix whose entries are the singular values. Given r ≪ min(m, n), to obtain a rank-r approximation, only the top-r singular values and vectors are retained, yields W ≈ LR, 1/2 1/2 where L = Ur Σr , R = Σr Vr⊤ , with Ur ∈ Rm×r , Vr ∈ Rn×r , and Σr ∈ Rr×r correspond to the leading r singular components. For an input x ∈ R1×m , we cache the compressed representation z = xL ∈ R1×r instead of
xW ∈ R1×n , and reconstruct the projection as xW ≈ zR, reducing KV cache memory with a compression ratio of r/n. Centered Kernel Alignment Similarity. Centered Kernel Alignment (CKA) is a similarity metric for comparing the representations learned by different neural network layers or models. Given two centered representation matrices X ∈ Rn×dx and Y ∈ Rn×dy , the linear CKA similarity is defined as ∥Y ⊤ X∥2F , CKA(X, Y ) = ⊤ ∥X X∥F ∥Y ⊤ Y ∥F where ∥ · ∥F denotes the Frobenius norm. The CKA score ranges from 0 to 1, where a larger value indicates greater similarity between the two representations. B. Compression ratio allocation Driven by Palu [10], we evaluate the relative importance of each layer by computing layer-wise Fisher Information scores on the calibration data. These scores are subsequently utilized to allocate the rank budget across layers. For a given layer and its allocated rank budget, we apply distinct compression strategies depending on whether it is a Key or Value layer, as detailed in the following subsections. C. Key compression Given a Key projection matrix W (k) ∈ Rm×n , where n = h · dh corresponds to h attention heads each with a hidden dimension of dh , and a total rank budget r for this matrix, we aim to further allocate the rank to groups of heads. Suppose W (k) ∈ Rm×n is partitioned column-wise into K submatrices, where each submatrix represents a distinct group of heads. Formally, those submatrices are denoted as Wi ∈ Rm×hi dh for i ∈ {1, . . . , K}, where hi represents PKthe number of heads assigned to the i-th group such that i=1 hi = h. The head ratio for group i is defined as αi = hi /h. For each group i, a low-rank approximation Wi ≈ Li Ri is performed, where rank Li ∈ Rm×ri and Ri ∈ Rri ×hi dh . Here, ri denotes Pthe K allocated to the i-th group, satisfying the constraint i=1 ri = r. The total number of parameters used for each W (k) after decomposition is then ! K K X X [(m · ri ) + (ri · hi dh )] = mr + n ri αi . i=1
i=1
The next step is to determine how to partition W (k) . Following ReCalKV [1], we first measure the pairwise similarity between attention heads to identify those with closely related structures. Specifically, each entry Si,j of the similarity matrix S ∈ Rh×h , where h denotes the number of attention heads, represents the similarity between i-th head and jth head. In our implementation, we utilize Centered Kernel Alignment (CKA), which measures representational similarity by comparing the feature representations produced by different heads, making it suitable for identifying heads that encode similar information. After obtaining the similarity matrix S, ReCalKV [1] reorders the attention heads and partitions them into groups of
Algorithm 1 Key Compression for a layer 1: Input: Projection matrix W
(k)
, similarity matrix S, rank budget r, candidates K 2: Output: Optimal factors {Li , Ri } 3: procedure COMPRESS (W (k) , S, r, K) 4: for all K ∈ K do 5: {Wi } ← CLUSTERHEADS(S, K) 6: {ri } ← INITRANKSBYENERGY({Wi }, r) 7: {ri } ← GREEDYADJUST({ri }, r) 8: {Li , Ri } ← GROUPDECOMPOSE({Wi }, {ri }) 9: L(K) ← COMPUTEERROR({Wi }, {Li , Ri }, {ri }) 10: end for 11: K ∗ ← arg minK∈K L(K) 12: return {Li , Ri } corresponding to K ∗ 13: end procedure
Nevertheless, the initialized ri does not necessarily guarantee this constraint. This motivates an idea of adjusting the ranks ri using a greedy heuristic. To limit energy loss, we prioritize reducing the rank of groups where each rank reduction results in the least energy loss. We therefore adopt a greedy strategy that iteratively decreases PKthe rank of the least important group until the constraint i=1 ri αi ≤ 4r/h is satisfied. Specifically, at each iteration, we evaluate the energy loss caused by reducing riP by one, denoted as ∆Ei . K Since reducing ri by one decreases i=1 ri αi by αi , we select the group with the smallest normalized cost ∆Ei /αi at each iteration. Note that after this greedy adjustment, the sum of ri is no longer equal to r, implying that the final parameter count for each W (k) must be written as ! K K K X X X [(m · ri ) + (ri · hi dh )] = m ri + n ri αi . i=1
a fixed size, specifically four heads per group. Moreover, all groups are compressed using the same compression ratio. In other words, in this case, αi = 1/K, ri = r/K for all i, and K = h/4. Under this uniform allocation configuration, the total number of parameters required to represent W (k) after low-rank decomposition is 4 mr + rn . h Instead of enforcing a predetermined number of heads per group as ReCalKV, in DynaCalKV, groups are now formed adaptively according to S. This dynamic grouping strategy better captures the similarity patterns across attention heads and preserves highly coherent groups for joint compression. This approach also eliminates the need to manually specify the group size, making the method more flexible across different models. To achieve this, we formulate the process as a clustering problem, resulting in groups with varying numbers of heads. Since the group proportions αi are determined automatically by the clustering process, the remaining task is to allocate the ranks ri . A straightforward strategy is to estimate the importance of each group based on its low-rank properties. Motivated by the fact that singular values quantify the information captured by a low-rank representation, we measure group importance using the sum of squared singular values, commonly referred to as the energy. Accordingly, the rank ri of the i-th group is initialized to be proportional to the energy and rounded to the nearest integer as energy of Wi ri = round r . energy of W (k) To ensure our approach offers a promising trade-off between structural compression and model accuracy, we constrain the total number of parameters used to represent each W (k) to be no greater than that of ReCalKV. Based on the analysis above, this requirement leads to K X i=1
ri αi ≤
4r . h
i=1
i=1
A remaining question is how to determine the number of clusters used in the clustering algorithm. Rather than relying on conventional clustering criteria, such as the Silhouette score or the Elbow method, we propose a heuristic objective considering reconstruction error and rank utilization. In particular, for each candidate clustering configuration, after obtaining all ri , the reconstruction error is simply measured as sum of the squared Frobenius norms of the difference matrix between Wi and Li Ri . As noted above, since all ri are modified to satisfy the parameter budget, the number of ranks used may be smaller than the available rank PKbudget r. Thus, we introduce a rank utilization penalty r− i=1 ri . Consequently, the optimal clustering configuration is selected by minimizing the objective function ! K K X X ri , ∥Wi − Li Ri ∥2F + λ r − i=1
i=1
where λ is a balancing coefficient to prevent either term from dominating the other. In our experiments, both terms are observed to be comparable, leading us to set λ = 1 in the final configuration. Consequently, Algorithm 1 summarizes the complete key compression pipeline for a Key projection matrix W (k) ∈ Rm×n . Moreover, since all the computations are performed only once during the offline compression stage, they introduce no additional inference overhead compared with ReCalKV. D. Value compression Guided by ReCalKV [1], we compress the Value cache by applying SVD to the original projection matrix W (v) ∈ Rm×n rather than processing in groups. The decomposed approximation is then represented as W (v) ≈ Lv Rv , where Lv ∈ Rm×r , Rv ∈ Rr×n and r is the rank budget. To quantify the reconstruction quality, the approximation error on a calibration set X is evaluated as ε = ∥Lv Rv X − W (v) X∥2F . The previous allocation step implies that the Value projection matrix contains substantially higher Fisher Information
than the Key projection matrix. Therefore, minimizing its reconstruction error is crucial for preserving model performance. Although SVD provides an optimal low-rank approximation in terms of the Frobenius norm, it is not necessarily optimal with respect to the activations induced by the calibration data. This indicates the need to further refine the decomposed matrices using the calibration set X. Specifically, the left factor Lv is adjusted first. By setting the gradient of ε with respect to Lv to zero, we obtain −1 Lv = W (v) XX ⊤ Rv⊤ Rv XX ⊤ Rv⊤ . After that, we update the right factor Rv by minimizing the same objective. Setting the gradient of ε with respect to Rv to zero yields −1 ⊤ (v) Rv = L⊤ Lv W . v Lv These two closed-form solutions for Lv and Rv lead to a lower reconstruction error, result in a more faithful low-rank approximation of the original Value projection matrix W (v) . IV. EXPERIMENTS A. Experimental Settings Implementation Details. We adopt the official Palu [10] repository as our implementation backbone. Given that ReCalKV [1] is also built on top of Palu, we first re-implement ReCalKV from this codebase to establish a consistent baseline, before extending it to incorporate our proposed method. For a fair comparison, we use a group size of 4 when reproducing ReCalKV, following its original experimental setup. All experiments are conducted on a single NVIDIA T4 GPU. Models. We evaluate our method on three instructiontuned language models with different attention architectures to assess its generality across settings. Specifically, we consider Llama-3.2-1B-Instruct, a Grouped-Query Attention (GQA) model with 8 attention heads per group, Qwen1.5-1.8B-Chat and SmolLM2-1.7B-Instruct, which use Multi-Head Attention (MHA) with 16 and 32 attention heads, respectively. This allows us to evaluate whether our proposed strategy is effective under both modern efficient attention GQA and standard fullattention MHA architectures. Datasets. We utilize WikiText-2 as our calibration dataset in both the compression ratio allocation and value compression steps. In the evaluation phase, we assess zero-shot accuracy across six QA benchmarks (OpenBookQA, HellaSwag, PIQA, ARC-e, ARC-r, and Winogrande) to evaluate the general knowledge and reasoning capabilities after KV cache compression. Furthermore, we adopt eight datasets from the LongBench benchmark (TriviaQA, Qasper, TREC, SAMSum, LCC, RepoBench-P, QMSum, and MultiNews) to examine the effectiveness of our proposed strategies in handling longcontext tasks. B. Results Scope and Limitations. We acknowledge that our experimental settings are limited by the available GPU resources, which is the primary reason we evaluate only models with
a relatively small number of parameters. Another limitation is that we do not report the long-context evaluation results for SmolLM2-1.7B-Instruct because all three methods - Palu, which serves as the foundation for ReCalKV and our proposed approach, ReCalKV, and our method - achieve nearly zero performance on the LongBench benchmark, making the comparison under this setting uninformative. Table I COMPARISON OF K STRATEGY AND K PARAMETERS BETWEEN RECALKV AND DYNACAL KV. K strategy
K params
Llama-3.2-1B-Instruct ReCalKV 3,612,672 DynaCalKV 2,953,920 Difference -658,752 (18.23%) Qwen1.5-1.8B-Chat ReCalKV 29,982,720 DynaCalKV 25,184,384 Difference -4,798,336 (16.00%) SmolLM2-1.7B-Instruct ReCalKV 33,988,608 DynaCalKV 11,818,176 Difference -22,170,432 (65.23%)
Parameter reduction. Since we remain the same strategy for the Value cache, we focus our analysis on the Key cache. Our theoretical analysis above shows that the proposed method always reduces the number of parameters required for the Key cache. In this section, we quantify the exact amount of this reduction. Table I shows that the parameter reduction is relatively modest for Qwen1.5-1.8B-Chat (16.00%), whereas larger reductions are achieved for Llama-3.2-1B-Instruct (18.23%) and SmolLM2-1.7B-Instruct (65.23%). These results suggest that the magnitude of the reduction depends on the model architecture and can be attributed to the difficulty of grouping attention heads with a larger head dimension. Qwen1.5-1.8B-Chat uses a head dimension of dh = 128, while both Llama-3.2-1BInstruct and SmolLM2-1.7B-Instruct use a head dimension of dh = 64. The higher-dimensional head representations tend to be less similar, making the Agglomerative Clustering algorithm less likely to merge multiple heads into the same group. As a result, more singleton groups are retained in Qwen1.5-1.8B-Chat, which is also illustrated in Figure 1. Since singleton groups preserve more distinctive information, they are less affected by the ri adjustment step, resulting in only a small parameter reduction. In contrast, larger groups contain more redundant heads and are allowed to sacrifice more rank components during the adjustment step, leading to a larger reduction. Regarding the lower-dimensional head representations, although the reduction on Llama-3.2-1B-Instruct is much smaller than that on SmolLM2-1.7B-Instruct, this is mainly because the former employs only 16 Key projection layers with a total dimension of 512, while the latter utilizes 24 Key
Figure 1. Visualization of attention head grouping structures across different models. Each block represents a group, where the block width corresponds to the group dimension and the color indicates the number of heads within the group. While Qwen1.5-1.8B-Chat shows a high number of 59 singleton groups, Llama-3.2-1B-Instruct and SmolLM2-1.7B-Instruct record only 17 and 38, respectively.
projection layers with a total dimension of 2048. Since each Key projection layer contributes to the parameter reduction, having fewer such layers naturally leads to a smaller total reduction. Furthermore, this difference is also magnified by their head configurations, 8 heads with a 512 total dimension for Llama versus 32 heads with a 2048 total dimension for SmolLM2. Following the same mechanism detailed above, the fewer heads in Llama are more likely to form smaller clusters, whereas the 32 heads in SmolLM2 tend to form larger groups, allowing them to sacrifice more rank components during the adjustment step. Zero-shot Accuracy Evaluation. Table II reports the zero-shot performance across six standard benchmarks. The empirical results suggest that while our strategy drastically downscales the Key cache parameter budget, it maintains highly competitive accuracy compared to ReCalKV. Specifically, the metrics highlight two distinct behaviors based on the characteristics of the tasks. On common knowledge tasks, such as PIQA and ARC-e, DynaCalKV consistently yields performance improvements over ReCalKV across all three models, achieving gains of +0.87% on PIQA and +2.44% on ARC-e. On more complex reasoning tasks, such as OpenBookQA, HellaSwag, ARC-r, and Winogrande, DynaCalKV experiences only minor degradation, with performance drops generally remaining within 0.28% to 3.00%. This suggests that grouping redundant Key cache does not significantly harm the retrieval of fundamental factual knowledge, but may adversely affect the reasoning procedure. Another phenomenon is observed in SmolLM2-1.7B-Instruct, where DynaCalKV outperforms ReCalKV on HellaSwag by a notable +4.72% increase, contributing to a positive average gain of +0.4433% across all six benchmarks.
LongBench Evaluation. For Qwen1.5-1.8B-Chat, DynaCalKV demonstrates remarkable robustness, incurring only a negligible average performance degradation of −1.02% compared to ReCalKV, while even achieving slight gains on summarization tasks such as +0.71 on SAMSum and +0.22 on QMSum. In contrast, Llama-3.2-1B-Instruct experiences a substantial drop in performance, with an average decline of −8.03 score. This degradation is evident across both retrievalheavy tasks (−18.96 on TriviaQA and −12.50 on TREC) and long-document summarization (−10.50 on SAMSum and −7.40 on MultiNews). These long-context scenarios heavily expose the architectural difference between MHA and GQA. Standard MHA models retain many independent Key heads, allowing DynaCalKV to cluster redundant heads without losing functionality. Conversely, Llama-3.2-1B-Instruct already employs GQA with only 8 Key heads. Applying further head clustering on such GQA models over-compresses these few heads, causing critical loss of the fine-grained positional and contextual features required for long-context retrieval and summarization. Practical Implications. Our findings suggest that DynaCalKV should be viewed as an architecture-aware compression strategy: • Ideal Case: DynaCalKV excels on standard Multi-Head Attention models with high head counts, achieving massive Key cache reductions while preserving or even enhancing performance across both short- and long-context tasks. • Cautionary Case: On models already using GroupedQuery Attention with few Key heads, while DynaCalKV remains effective for short-context tasks, it should be applied conservatively in long-context scenarios.
Table II ZERO-SHOT ACCURACY (%) PERFORMANCE ACROSS SIX STANDARD BENCHMARKS COMPARING RECALKV AND DYNACALKV. K strategy
OpenBookQA
HellaSwag
PIQA
ARC-e
ARC-r
Winogrande
Average
26.71 26.19 -0.52
52.88 52.49 -0.39
-0.1533
32.76 30.97 -1.79
58.64 57.30 -1.34
-0.8100
26.79 25.77 -1.02
53.28 51.93 -1.35
+0.4433
Llama-3.2-1B-Instruct ReCalKV DynaCalKV Difference
19.60 19.20 -0.40
36.22 35.63 -0.59
64.47 65.07 +0.60
53.79 54.17 +0.38
Qwen1.5-1.8B-Chat ReCalKV DynaCalKV Difference
25.40 22.80 -2.60
44.18 43.90 -0.28
ReCalKV DynaCalKV Difference
24.20 21.20 -3.00
29.90 34.62 +4.72
71.16 71.76 +0.60
63.09 63.64 +0.55
SmolLM2-1.7B-Instruct 58.05 58.92 +0.87
35.94 38.38 +2.44
Table III PERFORMANCE EVALUATION ON LONGBENCH DATASETS COMPARING RECALKV AND DYNACALKV. K strategy
TriviaQA
Qasper
TREC
SAMSum
LCC
RepoBench-P
QMSum
MultiNews
Average
27.86 24.85 -3.01
17.33 15.15 -2.18
19.19 11.79 -7.40
-8.03
30.44 29.62 -0.82
17.04 17.26 +0.22
23.68 23.53 -0.15
-1.02
Llama-3.2-1B-Instruct ReCalKV DynaCalKV Difference
56.40 37.44 -18.96
15.10 06.00 -9.10
33.00 20.50 -12.50
24.84 14.34 -10.50
22.24 21.67 -0.57
Qwen1.5-1.8B-Chat ReCalKV DynaCalKV Difference
68.37 67.52 -0.85
17.20 15.35 -1.85
42.00 38.00 -4.00
33.01 33.72 +0.71
V. CONCLUSION In this paper, we presented an improved low-rank KV cache compression framework for efficient inference in Large Language Models. Unlike previous approaches that rely on fixed attention-head grouping, our method DynaCalKV dynamically groups Key attention heads based on CKA similarity and allocates the rank budget adaptively while reducing the overall parameter budget. For the Value cache, we adopt the offline calibration strategy of ReCalKV to further improve reconstruction quality after low-rank decomposition. Experimental results on three instruction-tuned LLMs reveal that the effectiveness of DynaCalKV depends heavily on the underlying architecture: it is particularly well-suited for standard MHA architectures, whereas it should be applied more conservatively on models already utilizing GQA during long-context tasks. REFERENCES [1] X. Yan, Z. Li, T. Zhang, H. Qin, L. Kong, Y. Zhang, and X. Yang, “Recalkv: Low-rank kv cache compression via head reordering and offline calibration,” arXiv preprint arXiv:2505.24357, 2025. [2] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,” arXiv preprint arXiv:2402.02750, 2024. [3] C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “Kvquant: Towards 10 million context length llm inference with kv cache quantization,” Advances in Neural Information Processing Systems, vol. 37, pp. 1270–1303, 2024.
28.59 27.15 -1.44
[4] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 34 661–34 710, 2023. [5] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen, “Snapkv: Llm knows what you are looking for before generation,” Advances in Neural Information Processing Systems, vol. 37, pp. 22 947–22 970, 2024. [6] Z. Qin, Y. Cao, M. Lin, W. Hu, S. Fan, K. Cheng, W. Lin, and J. Li, “Cake: Cascading and adaptive kv cache eviction with layer preferences,” arXiv preprint arXiv:2503.12491, 2025. [7] B. Lin, Z. Zeng, Z. Xiao, S. Kou, T. Hou, X. Gao, H. Zhang, and Z. Deng, “Matryoshkakv: Adaptive kv compression via trainable orthogonal projection,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 86 669–86 690. [8] U. Saxena, G. Saha, S. Choudhary, and K. Roy, “Eigen attention: Attention in low-rank space for kv cache compression,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 15 332–15 344. [9] R. Zhang, K. Wang, L. Liu, S. Wang, H. Cheng, C. Zhang, and Y. Shen, “Lorc: Low-rank compression for llms kv cache with a progressive compression strategy,” arXiv preprint arXiv:2410.03111, 2024. [10] C.-C. Chang, W.-C. Lin, C.-Y. Lin, C.-Y. Chen, Y.-F. Hu, P.-S. Wang, N.-C. Huang, L. Ceze, M. S. Abdelfattah, and K.-C. Wu, “Palu: Compressing kv-cache with low-rank projection,” arXiv preprint arXiv:2407.21118, 2024. [11] W. Cui and M. Xu, “Homogeneous keys, heterogeneous values: Exploiting local kv cache asymmetry for long-context llms,” Advances in Neural Information Processing Systems, vol. 38, pp. 81 628–81 650, 2026. [12] H. Yao, X. Chen, A. Murtadha, and G. Wang, “Thin keys, full values: Reducing kv cache via low-dimensional attention selection,” arXiv preprint arXiv:2603.04427, 2026.