Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving

Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving Junhong Wu1,2* , Zedong Liu1,2* , Xinyang Ma1,2 , Yuhan Chen1,2 , Guangming Tan1 , and Dingwen Tao2

arXiv:2611.04382v1 [cs.DC] 3 Nov 2026

1

Institute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences taodingwen@ict.ac.cn

Abstract. KV cache compression reduces the communication volume of disaggregated large language model serving, but its practical benefit depends not only on compression ratio and model quality, but also on how efficiently the compression pipeline executes. This requirement becomes increasingly important as modern serving systems adopt high-bandwidth interconnects, where inefficient compression can offset a substantial portion of the saved communication time. This paper presents FastKVComp, a high-throughput KV cache compression data path built on top of KVServe. FastKVComp improves compression execution in two complementary ways. First, it fuses the distribution transformation and low-bit quantization stages into a KVaware TileLang kernel, eliminating intermediate tensor materialization and redundant memory accesses. Second, it replaces the original lossless compression backend with an open, extensible, and searchable backend based on LC, enabling high-performance compression operator pipelines. The fused kernel achieves up to 5.9× speedup, while the new lossless backend increases compression throughput from below 90 GB/s to over 500 GB/s. Across the evaluated workloads, FastKVComp reduces job completion time by up to 2.04×. End-to-end evaluation shows that these optimizations allow KV compression to retain meaningful communication benefits under modern high-bandwidth network. Keywords: KV cache compression · disaggregated LLM serving · operator fusion · lossless compression · GPU systems

1

Introduction

Disaggregated large language model (LLM) serving increasingly separates computation and storage responsibilities across devices and nodes. Representative scenarios include prefill–decode disaggregation, remote KV cache offloading [15, 16, 11]. In these systems, the key-value (KV) cache is no longer only local inference state; it also becomes a major communication object whose volume grows with model size, batch size, and context length. *

These authors contributed equally.

2

J. Wu and Z. Liu et al.

Recent KV cache compression methods reduce this communication volume through low-bit quantization, mixed-precision representations, distribution transformations, and lossless encoding [14, 7, 11, 13]. Most existing studies focus primarily on the tradeoff among compression ratio, model quality, and workloadspecific compression policies. These studies show that KV caches are highly compressible, but they do not fully address the efficiency of executing the resulting compression pipeline. Compression reduces end-to-end communication latency only when the saved transfer time is larger than the encoding and decoding overhead. This condition becomes more demanding as serving systems adopt 100–400 Gbps networks and high-performance remote direct memory access (RDMA) data paths [15, 16, 13]. Reducing KV volume is therefore not sufficient: compression must also execute fast enough to keep pace with modern serving networks. We identify two major optimization opportunities in the execution path of practical KV cache compression. First, distribution transformation and quantization are commonly implemented as separate GPU kernels. Their intermediate tensor is written to high-bandwidth memory (HBM) and immediately read by the following quantization kernel, introducing redundant memory traffic, temporary storage, and kernel-launch overhead. Second, the original lossless compression backend used by KVServe [13] provides limited openness and extensibility for integrating and combining high-performance compression primitives. Its effective throughput is therefore insufficient to fully exploit high-bandwidth serving data paths. To address these issues, we present FastKVComp, a high-throughput KV cache compression data path integrated into KVServe. FastKVComp fuses transformation and quantization into a single KV-aware TileLang kernel [18], avoiding the materialization of intermediate transformed tensors. It also introduces an open and searchable lossless compression backend based on LC [3, 1], which composes high-performance primitives into efficient compression pipelines for quantized KV data. These optimizations preserve the compression policies and model-quality characteristics of KVServe while substantially improving their execution efficiency. Our preliminary implementation achieves up to 5.9× acceleration for fused transformation and quantization. The new lossless compression backend improves effective throughput from below 90 GB/s to over 500 GB/s. In summary, this paper makes the following contributions: – We characterize the execution bottlenecks of practical KV cache compression, revealing substantial overhead from fragmented GPU operators and the inflexible lossless compression path. – We design a KV-aware TileLang frontend that fuses distribution transformation and low-bit quantization, while optimizing the inverse reconstruction path through tiled execution and workspace reuse. – We develop an open and searchable lossless compression backend based on LC, which composes GPU primitives into pipelines optimized for end-to-end KV transfer.

High-Throughput KV Cache Compression

3

– We integrate the optimized data path into KVServe and evaluate it on both host-staged and cross-node InfiniBand deployments, achieving up to 2.04× JCT speedup and 2.01× throughput improvement over the original KVServe implementation.

2

Background and Gap Analysis

2.1

KV Transfer and Compression Pipeline

KV Cache Compression. KV caches grow linearly with context length and can occupy substantial memory for long-context requests. Existing compression methods reduce this footprint through low-bit quantization, mixed-precision representations, distribution transformations, and lossless coding [11]. While originally motivated largely by memory capacity, these techniques are increasingly important when KV states must move across devices or nodes. Compression in KV Transfer. In disaggregated serving, KV caches are transferred between prefill and decode workers, offloaded to remote memory or storage, and reused across requests or serving sessions [16, 11]. These scenarios turn the KV cache into a large communication payload, often reaching gigabytes for large models and long contexts. Systems such as KVServe [13] therefore apply compression directly to the KV transfer path, reducing communication volume while preserving serving quality. A typical compression path first transforms the KV distribution, quantizes the transformed values into a compact low-bit representation, and optionally applies lossless compression before transfer. The receiver performs the inverse operations to reconstruct the KV states required by attention. For an original payload of size Draw , compressed size Dcomp , and effective network bandwidth Bnet , compression is beneficial when the saved transfer time exceeds its encoding and decoding overhead: Tencode + Tdecode <

Draw − Dcomp . Bnet

(1)

As network bandwidth increases, this condition leaves progressively less room for compression overhead. Practical KV compression must therefore optimize not only compressed size, but also the efficiency of the compression data path itself. 2.2

Optimization Opportunities

Fragmented Transformation and Quantization. In the original execution path, transformation and quantization are implemented as separate GPU kernels. The transformation kernel reads the original KV tensor and writes a full intermediate tensor to HBM. The quantization kernel then reads this intermediate tensor, computes or applies the quantization parameters, and writes the low-bit output. This design introduces two avoidable costs. First, the intermediate tensor incurs an additional HBM write and read, even though it is consumed only by

4

J. Wu and Z. Liu et al. (a) Original KVServe Original KV Tensor

Transform

nvCOMP Encoding

Quantization

Compressed KV Transfer

nvCOMP Decoding

Reconstruction

Recovered KV Tensor

Repeated full-size global-memory traffic

(b) Fused + LC Fused Original KV Tensor

Transform

Fused Quantization

LC Encoding

Compressed KV Transfer

LC Decoding

Reconstruction

Recovered KV Tensor

Reduced intermediate global-memory traffic

Fig. 1. Original KVServe and the proposed FastKVComp execution paths.

the immediately following quantization stage. Second, separate kernels introduce additional launch and synchronization overheads. These costs are especially important because both stages are closely coupled and process the same KV tensor layout. Limited Lossless Compression Extensibility. The original KVServe implementation uses nvCOMP through its available integration path. Although nvCOMP provides several GPU compression algorithms, the original integration offers limited flexibility for inspecting, extending, and composing the underlying compression operations. In particular, it is difficult to incorporate new highperformance primitives or construct specialized operator pipelines for quantized KV data. As a result, the effective end-to-end throughput of the original lossless compression stage remains below 100 GB/s in our initial measurements. This leaves substantial room for improving the execution efficiency of KV compression, particularly for systems equipped with modern high-bandwidth interconnects. Based on these observations, we establish two design goals: – G1: Eliminate redundant memory traffic. Closely coupled transformation and quantization operations should execute within a single GPU kernel whenever possible. – G2: Build a high-throughput and extensible backend. The lossless compression layer should expose composable high-performance primitives and support efficient pipeline construction for quantized KV data.

3

Design Overview

FastKVComp is a high-throughput implementation of the KV compression data path below the compression-policy layer of KVServe. It preserves the original strategy-selection and compression semantics while optimizing how a selected strategy is executed.

High-Throughput KV Cache Compression

5

FastKVComp consists of two components: a tiled TileLang frontend that streamlines transformation, quantization, and reconstruction using reusable GPU workspaces, and an LC-based lossless backend that provides composable compression primitives and offline pipeline search. At the sender, quantized KV data is written directly into a preallocated output buffer before lossless encoding. At the receiver, lossless decoding is followed by the optimized reconstruction path. The design prioritizes high throughput, low execution overhead, compatibility with existing KVServe policies, and extensibility to new compression components.

4

Ultra-fast KV Compression

4.1

Fused Transform–Quantization (Addressing G1)

To eliminate the redundant memory traffic identified as G1, we fuse transformation and quantization into a single tiled TileLang kernel (D1). The fused frontend combines the randomized Hadamard transformation, quantization-statistics computation, and low-bit conversion within a single GPU kernel. Each KV tile is loaded once and processed using on-chip storage. Intermediate transformed values are directly consumed by the quantization stage instead of being materialized as a full-size tensor in global memory. The operator then writes the quantized representation directly into a preallocated all-layer output buffer, reducing temporary allocation, head splitting, per-layer stacking, layout conversion, and framework-level kernel-launch overhead. The fused operator preserves the original KVServe compression policy, including the deterministic randomized Hadamard transformation, head-importanceaware mixed precision, channel-wise quantization for keys, token-wise quantization for values, and the original min–max quantization rule. Transformation signs, precision assignments, compiled TileLang operators, and auxiliary workspaces are cached and reused across layers and requests. Fused Reconstruction. On the receiver, the TileLang reconstruction operator combines dequantization, inverse Hadamard transformation, normalization, and sign restoration in a tiled execution pipeline. It avoids materializing a full-size dequantized intermediate tensor and writes only the reconstructed KV representation to global memory. This optimization is independent of the selected lossless backend and therefore applies to both nvCOMP and LC. 4.2

Searchable High-Throughput Lossless Backend (Addressing G2)

To meet G2, we replace the fixed nvCOMP path with a searchable high-throughput lossless backend based on LC (D2). nvCOMP provides a readily available GPUresident codec and achieves a high compression ratio. Such a design is effective when network transfer dominates the KV delivery path, because the reduction

6

J. Wu and Z. Liu et al.

in transferred data can compensate for the additional encoding and decoding cost. However, this tradeoff changes under high-bandwidth communication and after frontend optimization. Once transformation and quantization overhead is reduced, lossless encoding and decoding account for a larger fraction of the remaining critical path. Moreover, nvCOMP is distributed as a binary library with a fixed public interface, which limits the ability to inspect, recombine, or specialize its internal operations for low-bit KV data. Unlike a fixed general-purpose codec, LC [3, 1] constructs compression algorithms from composable GPU primitives, including tuple reordering, bit shuffling, and redundancy encoding. This allows the backend to be specialized for the statistical properties of quantized KV buffers. It also enables pipeline selection to consider encoding latency, decoding latency, and compressed size jointly rather than optimizing compression ratio alone. Offline Pipeline Search. We evaluate candidate two- and three-stage LC pipelines offline using quantized KV buffers generated by the optimized frontend. For each candidate, we measure GPU encoding time, GPU decoding time, and compressed size. We then retain the Pareto-efficient candidates and rank them for the evaluated high-bandwidth communication setting. This procedure selects TUPL8_1→BIT_8→ RZE_2. Tuple reordering and bit shuffling expose regularities in the low-bit KV representation, while zero-oriented encoding removes the resulting redundancy. Although the selected LC pipeline provides a lower compression ratio than nvCOMP ANS, its substantially lower encoding and decoding latency makes it competitive on the end-to-end critical path. Once selected, the pipeline remains fixed during online serving, so no search is performed at runtime. Native GPU Integration. We generate an in-process GPU runtime specialized for the selected LC pipeline. The runtime directly consumes the CUDA buffer produced by the fused TileLang frontend and returns a GPU-resident compressed bitstream. The online serving path therefore avoids LC’s file-based command-line interface, host-side payload serialization, and CPU–GPU movement of the compressed payload. Only a small compressed-size scalar is returned to the host during encoding. The runtime exposes device-pointer encoding and decoding APIs directly to the KV transfer path. Persistent runtime scratch space and a reusable encode staging buffer reduce repeated allocation on the critical path. The componentlevel effects of this integration are evaluated in Sec. 5.2. Extensibility. Because LC is component based, additional transformations, encoders, and architecture-specific implementations can be incorporated into the offline search without modifying the upper-level KVServe control logic. The same selection method can therefore adapt the lossless backend to future KV formats, models, communication environments, and GPU architectures.

High-Throughput KV Cache Compression Raw transfer

8

JCT (s)

6

Original + nvCOMP

Original + LC

Fused + nvCOMP

7

FastKVComp

Llama-3.1-8B (HotpotQA)

Llama-3.1-8B (Qasper)

Llama-3.1-8B (2WikiMQA)

Qwen2.5-7B (HotpotQA)

Qwen2.5-7B (Qasper)

1.83×

1.94×

1.93×

1.91×

1.99×

Qwen2.5-7B (2WikiMQA) 2.04×

1.83×

1.88×

1.93×

1.91×

1.99×

2.04×

4

2

0

Output tok/s

80

60

40

20

0

Fig. 2. End-to-end results over the host-staged NCCL SHM path.

D1 removes the dominant transform/quantization overhead (G1), while D2 prevents lossless coding from becoming the next bottleneck (G2); together they keep the whole path fast enough for high-bandwidth transfer.

5

Evaluation

We evaluate FastKVComp in terms of end-to-end serving performance, componentlevel efficiency, and the contribution of individual optimizations. Experimental setup. Our local experiments use two NVIDIA RTX 5090 GPUs (32 GB each) for prefill and decode, respectively, without tensor parallelism. KV payloads are transferred through the host-staged NCCL SHM path over PCIe Gen5 x16, with measured bandwidth up to 306.5 Gbps. The software stack includes CUDA 12.8, PyTorch 2.10.0, and vLLM 0.18.0 [9]. We additionally evaluate a two-node deployment connected by 200-Gb/s InfiniBand, using an NVIDIA RTX PRO 6000 for prefill and an NVIDIA H100 for decode. KV caches are compressed on the prefill node, transferred over InfiniBand, and reconstructed on the decode node. Workloads. We evaluate Llama-3.1-8B-Instruct [5] and Qwen2.5-7B-Instruct [19] on HotpotQA, Qasper, and 2WikiMQA from LongBench [2]. Inputs contain up to 20K tokens under a 32K context window. Each configuration processes 50 requests, generating 8 tokens locally and up to 64 tokens in the remote deployment. Compilation, initialization, and warm-up costs are excluded. Baselines and metrics. We compare Raw Transfer, Original KVServe, Fused Only, LC Only, and FastKVComp. All compressed configurations use identical compression policies, isolating the effects of operator fusion and the lossless backend. We report JCT, output throughput, KV transfer cost, compression ratio, and component-level compression and reconstruction overhead. 5.1

End-to-End Results

Figure 2 reports the end-to-end performance over the host-staged NCCL SHM path. Compared with Raw Transfer, FastKVComp improves output throughput

8

J. Wu and Z. Liu et al. Raw transfer

16

JCT (s)

12

Original + nvCOMP

Original + LC

Fused + nvCOMP

FastKVComp

Llama-3.1-8B (HotpotQA)

Llama-3.1-8B (Qasper)

Llama-3.1-8B (2WikiMQA)

Qwen2.5-7B (HotpotQA)

Qwen2.5-7B (Qasper)

1.62×

1.63×

1.57×

1.48×

1.45×

Qwen2.5-7B (2WikiMQA) 1.43×

1.61×

1.63×

1.59×

1.48×

1.46×

1.44×

8

4

0

Output tok/s

320

240

160

80

0

Fig. 3. End-to-end JCT and output throughput over 200-Gb/s InfiniBand. Red annotations show the improvement of FastKVComp over Raw Transfer.

by 1.83–2.04× and reduces JCT by 45.3–51.1% across both models and all workloads. Over Original KVServe, throughput improves by 1.87–2.01×. Although Original KVServe reduces KV volume by 9.4–10.1×, its fragmented compression path offsets most of the communication savings. In contrast, the fused configurations nearly double throughput, showing that efficient compression execution is critical at high network bandwidths. Figure 3 further evaluates FastKVComp in the two-node 200-Gb/s InfiniBand deployment. Across the remote workloads, FastKVComp reduces JCT by 30.2–38.6% and improves output throughput by 44.2–62.6% over Raw Transfer. These results confirm that the optimized compression path remains effective in heterogeneous, high-bandwidth cross-node serving. 5.2

Component-Level Results

We isolate the costs of the fused frontend and the lossless backend using two representative KV requests. Fused Transform–Quantization. The original transformation–quantization path takes 27.309–34.394 ms, whereas the fused TileLang implementation reduces this to 5.876–6.926 ms, yielding a 3.94–5.85× speedup and 74.6–82.9% lower preprocessing latency. The gain comes primarily from eliminating full-size intermediate KV materialization and reusing tiled workspaces, while preserving the original transformation and quantization semantics. Lossless Compression Backend. Using the LC pipeline selected in Sec. 4.2, lossless encoding latency drops from 3.179–3.274 ms with nvCOMP ANS to 0.443–0.531 ms, corresponding to a 6.0–7.4× speedup. Decoding similarly decreases from 2.543–2.666 ms to 0.488–0.575 ms, or about 4.9× on average. As shown in Fig. 4, after frontend fusion, lossless encoding accounts for 31.7– 35.1% of the encoding path with nvCOMP but only 6.1–8.2% with LC. This lower codec overhead comes at a moderate compression-ratio tradeoff: 8.850–9.168× for nvCOMP versus 6.344–6.360× for LC.

High-Throughput KV Cache Compression Original + nvCOMP

Fused + nvCOMP

(a) Latency - Request 0 Original + nvCOMP

10

70.5

86.4

71.9

7.4 ms 6.1% codec

20

30

40

0

10

Encoding latency (ms)

20

(d) Throughput - Request 1

85.7

27.8 ms 1.6% codec

6.4 ms 8.2% codec

0

Dark segment: lossless codec

(c) Throughput - Request 0

10.1 ms 31.7% codec

34.9 ms 1.5% codec

Original + LC

Fused + LC

Fused + LC

30.6 ms 10.7% codec

9.1 ms 35.1% codec

Fused + nvCOMP

Original + LC

(b) Latency - Request 1 37.6 ms 8.5% codec

30

40

Encoding latency (ms)

0

9

200

517.4 (6.0x)

520.7 (7.4x)

524.3 (6.1x)

512.6 (7.1x)

400

Codec throughput (GB/s)

600

0

200

400

600

Codec throughput (GB/s)

Fig. 4. Encoding-time breakdown for the original and fused frontends using nvCOMP and the searched LC pipeline. Table 1. Component-level compression and decompression performance in the 200Gb/s InfiniBand deployment. The RTX PRO 6000 and H100 serve as the prefill and decode GPUs, respectively. GPU

Path

KV comp. ratio Comp. throughput Comp. latency Decomp. throughput Decomp. latency (×) (GB/s) (ms) (GB/s) (ms)

RTX PRO 6000 Original + nvCOMP RTX PRO 6000 Original + LC RTX PRO 6000 Fused + LC

8.867 6.474 6.276

34.773 41.591 139.218

30.879 25.817 7.713

32.284 37.336 155.841

33.260 28.759 6.890

H100 H100 H100

8.804 6.478 6.280

27.659 30.195 91.606

38.820 35.560 11.721

26.698 29.038 103.104

40.219 36.977 10.414

Original + nvCOMP Original + LC Fused + LC

Complete Compression Path. Table 1 summarizes the complete compression and reconstruction paths on the GPUs used in the remote deployment. The critical transfer direction compresses KV caches on the RTX PRO 6000 prefill node and reconstructs them on the H100 decode node; results on both GPUs are reported for completeness. On the RTX PRO 6000 prefill GPU, FastKVComp reaches 139.218 GB/s compression throughput with 7.713 ms latency, providing a 4.00× throughput improvement and 75.0% latency reduction over Original KVServe. On the H100 decode GPU, it reaches 103.104 GB/s decompression throughput with 10.414 ms latency, corresponding to a 3.86× throughput improvement and 74.1% lower latency. These results demonstrate substantial overhead reduction on both ends of the remote KV-transfer path. For the two representative requests, the complete encoding path drops from 30.583–37.599 ms with Original KVServe to 6.400–7.376 ms with FastKVComp, yielding a 4.15–5.88× speedup and 75.9–83.0% latency reduction. Together, the component-level results explain the end-to-end gains: operator fusion removes the dominant frontend overhead, while LC prevents lossless coding from becoming the next bottleneck. 5.3

Ablation Study

Across both models and all three datasets, Fused Only improves output throughput by 1.87–2.01× over Original KVServe, confirming operator fusion as the primary source of end-to-end improvement under the evaluated SHM bandwidth. LC Only remains within 2.1% of Original KVServe: its lower codec latency is largely offset by the increased transfer volume from a lower compression ratio.

10

J. Wu and Z. Liu et al.

Fused Only and FastKVComp differ by at most 1.3% in output throughput. The former benefits from the higher compression ratio of nvCOMP, whereas FastKVComp favors the substantially lower codec overhead of LC. This tradeoff shows that the optimal lossless backend depends on the target communication regime rather than compression ratio alone.

6

Related Work

KV Cache Compression Existing KV cache compression methods reduce memory and communication costs through low-bit quantization [14, 7, 8], mixed precision [20], distribution transformations [12], sparsification [22, 10], and lossless coding [11, 13]. They improve the compression–quality tradeoff by reshaping KV distributions, allocating precision, or exploiting redundancy in quantized representations [21]. Recent systems combine KV compression with runtime management or GPU-friendly lossless coding [6], but complete pipeline efficiency remains less explored. FastKVComp is complementary: it preserves the quantization policy selected by KVServe while optimizing its execution. Operator Fusion Operator fusion reduces kernel launches, intermediate tensors, and memory traffic. Frameworks such as Triton [17], TileLang, and TVM [4] support specialized fused kernels. FastKVComp applies these principles to the tightly coupled transformation–quantization pair, consuming transformed values within one tiled GPU kernel without global-memory materialization.

7

Conclusion

Compression throughput is a first-class metric for KV cache compression in disaggregated LLM serving. FastKVComp combines a KV-aware fused transformation– quantization kernel with an open, searchable lossless backend, addressing overhead in both the frontend and codec. The fused kernel achieves up to 5.9× speedup, while the new backend exceeds 500 GB/s.These optimizations preserve the end-to-end benefits of KV compression over high-bandwidth networks. Acknowledgments. This work was supported by the National Natural Science Foundation of China through the Excellent Young Scientists Fund Program (Overseas) (Project No. E401470), the National Key Research and Development Program of China (Grant Nos. 2020YFA0907000 and 2025YFB30037002), the National Natural Science Foundation of China (Grant Nos. 62032023 and T2125013), and Ant Group. The AIdriven experiments, simulations, and model training were performed on the robotic AI-Scientist platform of the Chinese Academy of Sciences. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

High-Throughput KV Cache Compression

11

References 1. Azami, N., Fallin, A., Burtscher, M.: Efficient lossless compression of scientific floating-point data on CPUs and GPUs. In: Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. pp. 395–409 (2025). https://doi.org/10.1145/3669940.3707280 2. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., Li, J.: LongBench: A bilingual, multitask benchmark for long context understanding. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. pp. 3119–3137 (2024). https://doi.org/10.18653/v1/2024.acl-long.172 3. Burtscher, M., et al.: LC: Automatic generation of high-speed reliable lossy and lossless data compressors. https://userweb.cs.txstate.edu/~burtscher/ LC/ (2025), accessed 2026-07-29 4. Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., Guestrin, C., Krishnamurthy, A.: TVM: An automated end-to-end optimizing compiler for deep learning. In: 13th USENIX Symposium on Operating Systems Design and Implementation. pp. 578–594. USENIX Association (2018), https://www.usenix.org/conference/osdi18/presentation/chen 5. Dubey, A., et al.: The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024), https://arxiv.org/abs/2407.21783 6. Guo, Y., Joshi, S.: Splitzip: Ultra fast lossless kv compression for disaggregated llm serving. arXiv preprint arXiv:2605.01708 (2026) 7. Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M.W., Shao, Y.S., Keutzer, K., Gholami, A.: KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In: Advances in Neural Information Processing Systems. vol. 37 (2024). https://doi.org/10.52202/079017-0040 8. Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., Zhao, T.: GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint arXiv:2403.05527 (2024), https://arxiv.org/abs/2403. 05527 9. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J.E., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with PagedAttention. In: Proceedings of the 29th Symposium on Operating Systems Principles. pp. 611–626 (2023). https://doi.org/10.1145/3600006.3613165 10. Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., Chen, D.: SnapKV: LLM knows what you are looking for before generation. In: Advances in Neural Information Processing Systems. vol. 37 (2024). https://doi.org/10.52202/079017-0722 11. Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., Maire, M., Hoffmann, H., Holtzman, A., Jiang, J.: CacheGen: KV cache compression and streaming for fast large language model serving. In: Proceedings of the ACM SIGCOMM 2024 Conference. pp. 38–56 (2024). https://doi.org/10.1145/3651890.3672274 12. Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., Blankevoort, T.: SpinQuant: LLM quantization with learned rotations. In: International Conference on Learning Representations (2025), https://arxiv.org/abs/2405.16406 13. Liu, Z., Ma, X., Luo, D., Zhao, H., Lu, B., Huang, W., Gu, Y., Liu, X., Wei, Z., Liu, J., Tao, D., Tan, G.: KVServe: Service-aware KV cache com-

12

J. Wu and Z. Liu et al.

pression for communication-efficient disaggregated LLM serving. arXiv preprint arXiv:2605.13734 (2026), https://arxiv.org/abs/2605.13734 14. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., Hu, X.: KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In: Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 235, pp. 32332–32344 (2024), https://proceedings.mlr. press/v235/liu24bz.html 15. Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., Bianchini, R.: Splitwise: Efficient generative LLM inference using phase splitting. In: 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (2024). https://doi.org/10.1109/ISCA59077.2024.00019 16. Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., Xu, X.: Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In: 23rd USENIX Conference on File and Storage Technologies. pp. 155–170. USENIX Association (2025), https://www. usenix.org/conference/fast25/presentation/qin 17. Tillet, P., Kung, H.T., Cox, D.: Triton: An intermediate language and compiler for tiled neural network computations. In: Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. pp. 10–19 (2019). https://doi.org/10.1145/3315508.3329973 18. Wang, L., Cheng, Y., Shi, Y., Tang, Z., Mo, Z., Xie, W., Ma, L., Xia, Y., Xue, J., Yang, F., Yang, Z.: TileLang: A composable tiled programming model for AI systems. arXiv preprint arXiv:2504.17577 (2025), https://arxiv.org/abs/2504. 17577 19. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024), https://arxiv.org/ abs/2412.15115 20. Yang, J.Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S.J., Lee, D.: No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096 (2024), https://arxiv. org/abs/2402.18096 21. Yuan, J., Liu, H., Zhong, S., Chuang, Y.N., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V., Xu, Z., Liu, Z., Hu, X.: KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 4623–4648. Association for Computational Linguistics (2024). https://doi.org/10.18653/v1/2024.findings-emnlp.266, https:// aclanthology.org/2024.findings-emnlp.266/ 22. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., Chen, B.: H2O: Heavy-hitter oracle for efficient generative inference of large language models. In: Advances in Neural Information Processing Systems. vol. 36 (2023). https://doi.org/10.52202/075280-1506

Related documents

• Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving#971420
• KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving#180611
• WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse#1129857
• WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse#1108699
• Consistent LLM Call Handling Across Java Frameworks: A Reference Implementation and Benchmark#1400308