arXiv:2605.01708v1 [cs.DC] 3 May 2026
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
Yipin Guo, Siddharth Joshi University of Notre Dame IN, USA [email protected], [email protected]
Abstract Contemporary systems serving large language models (LLMs) have adopted prefilldecode disaggregation to better load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers before token generation can begin. With these workers residing on different physical systems, this transfer becomes a significant bottleneck to serving LLMs at scale. This bottleneck gets exacerbated for long-input and agentic workloads, which typically require long inputs. Existing lossless codecs are not well suited to this setting as they primarily target offline weight compression, rely on CPU-side, or use variable-length coding that decompresses fast but compresses too slowly for online use. SplitZip is a GPU-friendly lossless compressor for KV-cache transfer. It exploits redundancy in floating-point exponents of KV activations, encoding the most frequent exponent values with fixed-length codes, and encoding (position, value) pairs and value of rare exponents in an escape stream. An offline calibrated top-16 exponent codebook enables online encoding, while the regular dense path and sparse escape correction make both encoding and decoding efficient on GPUs. On real BF16 activation tensors, SplitZip achieves 613.3 GB/s compression throughput and 2181.8 GB/s decompression throughput, substantially outperforming prior lossless compressors on the latency-critical codec path. End-to-end transfer experiments show up to 1.32× speedup for BF16 KV-cache transfer, 1.30× speedup for TTFT and 1.23× increase on Request Throughput.
1
Introduction
Prefill-decode (PD) disaggregation has become a widely adopted systems design pattern [31, 11] for large-scale LLM serving because it enables specialization during two fundamentally different phases of inference: prefill is primarily compute-intensive, whereas decode is primarily memory-bandwidthintensive. This system direction has influenced the larger serving ecosystem through collaboration with open frameworks such as vLLM [23], SGLang [6], and Dynamo [18]. However, although PD disaggregation improves resource utilization and scheduling flexibility, its efficiency, critically, depends on KVCache transfer between the prefill and decode stages. When prefill and decode are deployed on different nodes or different clusters, the KVCache generated during prefill must be transmitted quickly enough to keep decode continuously fed. In tightly coupled deployments, this transfer can often be sustained by a high-bandwidth RDMA fabric. In more realistic production settings, however, prefill and decode are frequently separated across deployment boundaries for reasons such as resource isolation, cluster management, elasticity, distinct accelerator architectures, or independent scaling. In such settings, KVCache transfer must traverse slower and more constrained inter-node or inter-cluster links, making communication overhead a growing bottleneck.[21] Preprint.
Table 1: BF16 KV value exponent statistics across different model families and architecture. Top-16 coverage above 99% high frequency exponent. This motivates 4-bit exponent coding with escapes. Model
Family
Top-8
Top-16
Entropy
Realized SplitZip CR
Qwen3-30B-A3B Qwen3-32B Qwen3-Next-80B-A3B Llama-3.1-70B Llama-3-8B Phi-2
Qwen-MoE Qwen Qwen-Hyper Llama Llama Phi
83.5% 90.1% 96.0% 89.7% 90.9% 92.3%
99.3% 99.8% 99.9% 99.5% 99.6% 99.6%
3.59 b 3.29 b 2.89 b 3.41 b 3.30 b 3.11 b
1.31× 1.32× 1.33× 1.32× 1.32× 1.32×
This bottleneck is especially pronounced for long-input workloads, such as document-level question answering, codebase understanding, and multi-document summarization, where prefill produces a large KV cache that must be transferred to the decode workers. As context lengths continue to grow, the cost of KV transfer increase. Therefore, accelerating KVCache transfer is a key requirement for making PD disaggregation efficient in standard large-scale serving deployments. To address this practical bottleneck, existing approaches have explored lossy KV-cache compression, such as pruning[30, 13], quantizing[10, 28], or selectively retaining KV states[3, 25], as well as system-level solutions that depend on specialized hardware assumptions[7] or changes to the serving paradigm[22]. We propose a lossless compression approach for KVCache transfer in conventional PD-disaggregated deployments called SplitZip. Our goal is to reduce communication volume while preserving exact model semantics, thereby improving transfer efficiency without sacrificing model ability or changing any tensor values. Prior work has shown that, in BF16 model weights, redundancy is concentrated in the exponent field, while the mantissa is nearly incompressible[29]. We find that the same pattern also holds for activations, including KV tensors in PD-disaggregated serving. Across common LLMs, the exponent entropy is typically only 3 to 4 bits, indicating 4 bit room for lossless compression. Most prior work exploits this redundancy with Huffman coding[27, 29, 2]. Although it’s theoretically optimal, Huffman-based schemes often suffer from limited runtime efficiency because their variablelength bitstreams are difficult to decode efficiently in parallel. Some methods improve throughput by blockwise processing of bitstreams[27] or variable-length coding[1]. Our approach instead uses fixed 4-bit codes for the most frequent exponent values, and stores the rare uncovered cases as escapes with their positions and original values. In practice, 4-bit coding covers about 99% of cases, as shown in Table 1, so escape overhead is very small. At the same time, the fixed-length design maps naturally to GPU parallelism, enabling much higher encoding and decoding throughput while remaining fully lossless. Our contributions can be summarized as follows: • We propose a new lossless compression method for LLM KV cache. • We develop a GPU-friendly implementation that enables extremely high encoding and decoding throughput. • Our method is simple and lightweight. It provides a plug-and-play acceleration solution that can be easily integrated into different serving frameworks.
2
Related Work
Disaggregated PD serving. DistServe [31] first separates prefill and decode to improve goodput under latency constraints, while Splitwise [20] pipelines phase-split inference across heterogeneous GPUs. To make transmission more efficient, Mooncake [22] introduces a KV-cache-centric disaggregated architecture with a Transfer Engine for RDMA and TCP transport. FlowKV [12] focuses on fragmented KV memory transfer and segmented kernel launches. These systems reduce scheduling or transfer overhead, but they do not make KV transfer itself smaller through a lossless, GPU-native codec. Lossless LLMs compression. General GPU compression libraries such as nvCOMP[19] provide LZ4, Cascaded, and entropy-coding primitives. They are valuable for generic data, but floating-point KV tensors need a format-aware decomposition: the BF16 mantissa stream is hard to compress, while the exponent stream is highly structured. DFloat11 [29] compresses BF16 model weights by 2
1
Key Observation
2
3
Lossless Compression
BF16 =
Top-16 Exponent Codebook
Sign
Exponent
Mantissa
1 bit
8 bits
7 bits
e0
Extract exponent
e1
e2
…
e15
…
1111
Pack 2 codes per byte
GPU-Friendly Parallel Decode
4-bit packed exponent stream …
Escape buffer (pos, raw exp) …
… …
Exponent: highly redundant
0000 0001 0010
Fixed 4-bit codes (most frequent)
4-bit packed exponent stream
1
Parallel 4-bit decode Decode exponents in parallel
…
Entropy ~ 3–4 bits
...
...
KV Cache (BF16)
Exponent value
Block-wise Escape Sensing
Escapes (rare exponents) position raw exponent p1 p2
ex1 ex2 ⋯
⋮
...
This redundancy pattern consistently holds for activations / KV tensors.
…
Keep exact
Only exponent is compressed. Sign and mantissa are preserved.
⚡
Sign+mantissa stream (no compression)
Fixed-length 4-bit codes enable GPU parallelism (unlike Huffman coding).
Merge with sign+mantissa Exponents and sign+mantissa stream combine
…
3
…
Extract sign + mantissa
2
Store (position, raw exponent)
Escape buffer
Mantissa: little redundancy ... High entropy
Coverage ＞ 99%, escape overhead is tiny.
Escape fix-up Overwrite escaped exponents
… Reconstructed KV Cache (bit-exact)
✓ Truly lossless
Figure 1: Overview of SplitZip for lossless KV-cache compression. SplitZip exploits the redundancy of the BF16 exponent field while keeping sign and mantissa exact. It encodes the most frequent exponent values with fixed 4-bit codes and stores rare values in a small escape buffer with their positions and raw exponents. This design enables GPU-friendly parallel decoding and reconstructs the original KV cache bit-exactly.
coding exponent bits while preserving sign and mantissa information to improve decode throughput. ZipNN [9] studies lossless compression for weights, checkpoints, and low-precision tensors. ZipLLM [24] is also related, as it combines tensor-level deduplication with lossless compression for efficient LLM storage. They all employ compression methods based on Huffman coding. Furthermore, Falcon[14] and ZipServ[7] utilize multi-level fixed-length encoding and execution to the GPU, thereby further enhancing throughput. SplitZip is inspired by fixed-length coding in this line of work, but targets a different setting: the latency-critical KV transfer path in disaggregated serving, where both compression and decompression throughput must be optimized to enable efficient network transfer.
3
SplitZip
3.1
KV Exponent Entropy
Table 1 characterizes the exponent distribution of KV-cache across representative model families. Despite differences in architecture and numerical range, exponent entropy remains low, ranging from 2.89 to 3.59 bits, substantially below the 8-bit BF16 exponent width. More importantly, the distribution is highly concentrated: the most frequent 16 exponent values cover at least 99.3% of all exponent entries across all evaluated models. This phenomenon appears across multiple model families and various model architectures, thereby demonstrating its generality. In contrast, Top-8 coverage varies more significantly, from 83.5% on Qwen3-30B-A3B to 96.0% on Qwen3-Next-80BA3B, indicating that a 3-bit codebook is less robust across activation distributions. This motivates the Top-16 design used by SplitZip: although it sacrifices some compression ratio compared with more aggressive Top-8 coding, its near-complete coverage greatly reduces escape handling, leading to more regular GPU execution and higher throughput. Most existing lossless neuralnetwork compression systems target weights, where compression can be performed offline and the main concern is decompression overhead [29, 9, 7]. In prefill–decode disaggregated serving, however, KV caches are generated, transferred, and consumed online. Both compression and decompression therefore lie on the latency-critical path. SplitZip is designed for this setting: it reduces transferred bytes while keeping both codec directions simple and GPU-friendly. 3
3.2
Method
SplitZip combines fixed-length exponent coding with explicit escape capture, as illustrated in Figure 1. For a BF16 value represented as a 16-bit integer xi , SplitZip extracts e_i = (x_i >> 7) & 0xff; a_i = ((x_i >> 8) & 0x80) | (x_i & 0x7f); where e_i is the 8-bit exponent and ai is the exact sign–mantissa byte. Given a decoded exponent êi , the original BF16 bit pattern is reconstructed as x̂i = ((ai & 0x80) ≪ 8) | (êi ≪ 7) | (ai & 0x7f ). The sign bit and mantissa bits are stored losslessly as an 8-bit stream. For the exponent stream, SplitZip uses a calibrated codebook containing the 16 most frequent exponent values. Each common exponent is encoded using a 4-bit code, and two 4-bit codes are packed into one byte. Exponent values outside the top-16 set are treated as escape values. For each escape value, SplitZip records its block-relative element position as a 16-bit unsigned integer and its raw exponent value as an 8-bit integer. Let C = {c0 , . . . , c15 } be the calibrated top-16 exponent set and E = {i | ei ∈ / C} be the escape set. For a tensor with N BF16 elements and M = |E| escaped exponents, SplitZip stores N 3 BSZ = |{z} N + + 3ϵ Bytes. + 3M = N |{z} 2 2 |{z} sign–mantissa escape positions and values 4-bit exponent codes
Here the ϵ = M/N is the escape rate. Since the uncompressed BF16 tensor uses Braw = 2N bytes, the compression ratio is Braw 2 ρ= = 3 . BSZ 2 + 3ϵ When the escape rate is negligible, ρ approaches 4/3. The first two terms form the dense, regular compression path, while the final term is a sparse correction stream. When the top-16 exponent coverage is high (covers greater than 99%.), M is very small, so the escape overhead is negligible compared with the reduction from replacing each 8-bit exponent with a 4-bit code. The encoding path contains two GPU-friendly stages. The first stage performs the dense transformation: it extracts the exponent, maps it through the encoding lookup table, packs two 4-bit codes into one byte, and stores the sign–mantissa stream exactly. The second stage scans the original exponent stream and compacts all exponents not covered by the top-16 codebook into chunked escape-position and escape-value arrays. Using a separate escape-collection stage keeps the common path simple and regular. Decoding follows the reverse order. SplitZip first unpacks each byte into two 4-bit codes, maps each code through a 16-entry decoding lookup table, and reconstructs a BF16 bit pattern by combining the decoded exponent with the stored sign–mantissa bits. Since escaped elements were assigned an arbitrary dummy 4-bit code during dense encoding, they may initially decode to an incorrect common exponent. SplitZip then applies the sparse correction stream: for each recorded escape position, it overwrites the reconstructed exponent with the exact raw exponent value. This final overwrite makes the overall codec strictly lossless. The design is intentionally aligned with GPU execution. The main encode and decode paths use fixed-width loads, stores, bit shifts, and table lookups. The decoding table has only 16 entries, so lookup overhead is small and cache-friendly. 3.3
Calibration
Computing an exact exponent histogram during every compression call would be expensive, SplitZip performs a one-time calibration step on a small representative dataset. During calibration, SplitZip extracts all exponent values from the calibration tensors, counts their frequencies, selects the top-16 4
exponents, and constructs three lookup tables: an encoding table from raw exponent values to 4-bit codes, a decoding table from 4-bit codes back to raw exponent values, and a membership table for detecting escapes. This calibration-based design removes histogram construction from the online compression path. Although the codebook is selected using a calibration dataset, our experiments show that the selected high-frequency exponent set generalizes well across inputs. 3.4
Top-16 Instead of Top-15
A natural alternative to identification of escape value is to reserve one coded value as an explicit escape token. This top-15-plus-escape-token design allows the decoder to infer escape positions directly from the packed exponent-code stream, thereby reducing the escape positions overhead. However, this design also reduces the number of directly represented common exponents from 16 to 15 and introduces special-case handling into the dense decode path. SplitZip instead uses all 16 possible 4-bit codes for common exponent values. Escaped exponents are assigned a dummy code in the packed stream and are later corrected using the explicit escape-position array. It increases codebook coverage by retaining the 16th most frequent exponent and preserves a uniform dense decoding path: every element is decoded by the same 4-bit lookup operation. The objective is not only to maximize the compression ratio, but also to maximize end-to-end pipeline throughput. A codec with marginally smaller metadata can still be slower if it introduces divergence or serialized handling into the common path. By using top-16 coding and sparse post-correction, SplitZip keeps the dominant computation simple and parallel.
4
Experiments
4.1
Setup
Platform and timing protocol. Experiments are conducted on server with NVIDIA H200 GPUs and dual-socket Intel Xeon Platinum 8468 CPU. The server includes Mellanox ConnectX-7 MT2910 InfiniBand/RDMA adapters for high-speed KV transfer, as well as Mellanox ConnectX-4 Lx and Intel X550 Ethernet adapters. We report compression and decompression throughput in GB/s, where the byte count is the uncompressed BF16 tensor size. Each codec benchmark first verifies bitwise roundtrip correctness and then times repeated executions after warm-up. Codec throughput is measured over 10 runs, and tables report the average with measured fluctuation when available. Mooncake[22] transfer experiments report measured KV-transfer time; the Qwen3-32B[26] breakdown uses additive accounting for encode, compressed transfer, and decode under the RoCE 4×200G configuration with 700 Gb/s effective payload bandwidth. Datasets and workloads. All main experiments use authentic BF16 KV-cache activations. For codec and ablation measurements, we use Qwen3-32B KV tensors assembled into a 256M BF16 workload on wikitext2-test and run the chunked escape value with chunk size 1024. For Mooncake transfer, we evaluate Llama-3-8B[16] and Qwen3-30B-A3B under four sweeps: fixed batch size 1 with sequence length from 512 to 131K, fixed batch size 16 with sequence length from 128 to 64K, fixed sequence length 1024 with batch size from 1 to 256, and fixed sequence length 32768 with batch size from 1 to 128. For the Qwen3-32B transmission breakdown, we report sequence lengths 2048, 16384, and 65536. We also instrument SGLang on Qwen3-32B with the same sweep structure to record TTFT and serving throughput. Codebook statistics and stability experiments use BF16 KV caches from Qwen3-32B and Qwen3-Next-80B-A3B, and the model-family exponent table additionally includes Qwen3-30B-A3B, Llama-3.1-70B-Instruct, Llama-3-8B, and Phi-2[17]. Unless stated otherwise, the BF16 Top-16 codebook is calibrated on the WikiText-2[15] training subset. Baselines and ablations. For BF16 codec comparison, we evaluate, nvCOMP LZ4, nvCOMP Cascaded, nvCOMP Bitcomp[19], DFloat11[29], ZipNN[9], ZipServ[7], Falcon[14], and SplitZip. For transfer-time studies, the native baseline moves raw BF16 KV-cache bytes, while SplitZip sends the compressed payload and accounts for encode and decode where applicable. The ablation suite covers BF16 Top-8 3-bit coding versus Top-16 4-bit coding, explicit escape-position metadata versus sentinel-only escape discovery, pre-calibrated versus dynamic Top-16 codebooks, per-tensor/per-token/per-channel calibration granularity, chunk-size sensitivity, codec-stage break5
Speedup (x)
Table 2: Comparison of encoding/decoding throughput and compression ratio of various compression methods. While maintaining a respectable compression ratio, SplitZip demonstrates a significant advantage in codec throughput. Method
Ratio
nvCOMP LZ4[19] nvCOMP Cascaded[19] nvCOMP Bitcomp[19] ZipNN[9] DFloat11[29] Falcon[14] ZipServ - CPP [7] ZipServ - Kernel [7] SplitZip
1.019 13.4 ± 0.4 137.1 ± 8.8 1.000 111.8 ± 0.8 155.2 ± 5.6 1.056 341.5 ± 7.1 147.7 ± 4.8 1.515 1.2 1.7 1.423 4.0e−03 ± 5.0e−05 468.2 ± 2.5 1.428 8.9 ± 1.8 14.4 ± 3.3 1.236 0.1 ± 2.1e−04 499.5 ± 0.0 1.236 N/A 1260.9 ± 1.4 1.324 613.3 ± 2.6 2181.8 ± 38.5
TTFT
(a) BS=1 TTFT
(b) BS=1 Requests/s
1.0
1.0
0.5
0.5
512 1K 2K 4K 8K16K32K641K28K
512 1K 2K 4K 8K16K32K641K28K
Seq Len
Encode(GB/s)
Requests/s
Decode(GB/s)
Native
(c) BS=16 TTFT
1.0 0.5 0.0 8 6 2 K K K K 12 25 51 1 2 4 8
Seq Len
Seq Len
(d) BS=16 Requests/s
1.0 0.5 0.0 8 6 2 K K K K 12 25 51 1 2 4 8
Seq Len
Figure 2: End-to-end speedup on Qwen3-32B across sequence-length sweeps with SplitZip enabled in SGLang [6]. down, and codebook stability across layers and between K and V caches. Unless otherwise specified, ablations use Qwen3-32B BF16 KV values. 4.2
Results
Codec-level performance. Table 2 compares SplitZip with existing lossless compressors. The table includes GPU-oriented baselines evaluated during the revision, including nvCOMP LZ4, Cascaded, and Bitcomp, as well as Falcon and ZipServ. For ZipNN, we report the throughput numbers from its original paper. For ZipServ, the CPP row corresponds to its CPU computation kernel, while Kernel corresponds to its GPU computation kernel. SplitZip reaches 613.3 ± 2.6 GB/s encoding and 2181.8 ± 38.5 GB/s decoding. It is faster than all measured baselines on both encode and decode. Specifically, SplitZip improves encode throughput by 1.8× over nvCOMP Bitcomp, the fastest nvCOMP encoder, and improves decode throughput by 14.1× over nvCOMP Cascaded, the fastest nvCOMP decoder. Compared with ZipServ, SplitZip removes the CPU-side compression bottleneck and achieves 1.7× higher decode throughput than the GPU kernel path. Falcon is currently the GPU-encoding-optimized method among prior baselines, yet SplitZip achieves 68.9× higher encoding throughput. Meanwhile, SplitZip still maintains a strong compression ratio: it is 7.1% higher than ZipServ when metadata overhead is considered, and only 7.0% lower than DFloat11, a Huffman-based GPU-efficient compression method. This throughput advantage comes from SplitZip’s GPU-friendly codec design. SplitZip uses fixedlength exponent coding, regular bit manipulation, small lookup tables, and sparse escape correction, all of which map well to massively parallel GPU execution. As a result, compression and decompression can exploit otherwise underutilized GPU compute resources during KV-cache transfer. In contrast, DFloat11 and ZipNN rely on CPU-side compression, making their compression throughput insufficient for online KV-cache transfer. Moreover, their Huffman-based variable-length coding introduces sequential dependencies during decoding, which limits parallelism and prevents decompression from reaching very high throughput. SplitZip instead targets KV-cache communication directly: it preserves a simple dense GPU path for the common case and isolates rare values into a lightweight escape stream. KV Transfer time. Figure 3 reports the measured Mooncake transfer time for Llama-3-8B and Qwen3-30B-A3B across different batch sizes and sequence lengths. Theoretical Opt here represents the theoretically optimal speedup under the assumption of zero encoding/decoding overhead and 6
Mooncake Transfer (ms)
1.27x
BS=16, Seq Sweep
200
2000
100
1000
Mooncake Transfer (ms)
Qwen3-30B-A3B
Llama-3-8B
300
Native
BS=1, Seq Sweep
SplitZip
1.29x
Theoretical Opt.
Seq=1024, BS Sweep
1.32x
400
Seq=32768, BS Sweep
1.32x
7500 5000
200
2500
0 512 1K 2K 4K 8K16K32K641K28K
0 128256512 1K 2K 4K 8K16K32K64K
01 2 4 8 6 2 4 8 6 1 3 6 12 25
01 2 4 8 6 2 4 8 1 3 6 12
BS=1, Seq Sweep
BS=16, Seq Sweep
Seq=1024, BS Sweep
Seq=32768, BS Sweep
Seq Len
Seq Len
1.24x
1.32x
100
1000
50
500
0 512 1K 2K 4K 8K16K32K641K28K
0 128256512 1K 2K 4K 8K16K32K64K
300
1.32x
200
Batch Size
1.22x
4000 2000
100
Seq Len
Seq Len
Batch Size
01 2 4 8 6 2 4 8 6 1 3 6 12 25
Batch Size
01 2 4 8 6 2 4 8 1 3 6 12
Batch Size
Figure 3: KV-cache transfer time across sequence-length and batch-size sweeps using Mooncake[22]. Native transfer Encode Transfer Decode Seq Len = 2K Seq Len = 16K Seq Len = 64K 56.5 441.4 1749.3 Native Native Native
Ours 0
20
40
Time (ms)
53.1
Ours
60
0
200
Time (ms)
353.8
Ours
400
0
1397.0
500 1000 1500 2000
Time (ms)
Figure 4: Transmission-time breakdown on Qwen3-32B. Native transfer sends the raw BF16 KV cache, while SplitZip consists of encoding, compressed transfer, and decoding under RoCE 4 × 200G. the complete absence of any escape values. Across all eight panels, SplitZip consistently reduces Mooncake transfer time once the KV payload becomes large enough for bandwidth to dominate the end-to-end transfer cost. For short sequences, the benefit is smaller because fixed overheads and kernel launch costs account for a larger fraction of the total time. As sequence length increases, however, the transfer time becomes increasingly bandwidth-bound, and the reduction in transferred bytes directly translates into lower latency. At the largest point in each sweep, SplitZip achieves 1.27×–1.32× speedup on Llama-3-8B and 1.22×–1.32× speedup on Qwen3-30B-A3B. These results show that SplitZip provides consistent transfer-time reduction across model architectures and workload scales. Mooncake is the default KV-transfer backend for PD disaggregated serving in SGLang[6], a widely used LLM serving framework, and is therefore representative of practical deployments. We also integrate SplitZip into SGLang and evaluate end-to-end performance across sequence-length sweeps. As shown in Figure 2, SplitZip provides consistent improvements once the workload becomes transferdominated. For batchSize=1 with sequence length ranging from 512 to 128K, SplitZip achieves TTFT speedups up to 1.303× and throughput improvements up to 1.233×. For batchSize=16 with sequence length ranging from 128 to 64K, the TTFT ranges up to 1.274×, while requests throughput improvements up to 1.206×. The gains are more pronounced at longer sequence lengths, where KV transfer dominates the critical path, while slight slowdowns can occur in small-payload regimes due to fixed overheads. Overall, these results demonstrate that SplitZip translates its codec-level throughput advantage into tangible end-to-end benefits in a production serving system. Transmission-time breakdown. Figure 4 further breaks down the Qwen3-32B transfer time with batch size equal to 16. This breakdown uses additive accounting for encode, compressed transfer, and decode time. At sequence length 2K, SplitZip slightly reduces transfer time from 56.5 ms to 53.1 ms. However, the advantage increases with sequence length. At 16K, SplitZip reduces transfer time from 441.4 ms to 353.8 ms. At 64K, it reduces transfer time from 1749.3 ms to 1397.0 ms. In this long-context regime, compressed transfer accounts for 92.9% of the SplitZip time, while encoding and decoding account for only 5.7% and 1.4%, respectively. This confirms that SplitZip keeps codec overhead small enough for long-context KV-cache transfer to remain primarily communication-bound. 7
4.3 4.3.1
Ablation Study Ablation on Top-k Exponent Coding
Table 3 compares Top-8 3-bit coding with Top-16 4-bit coding. Top-8 reduces the dense code width, but the much larger escape stream overwhelms this saving and lowers the measured compression ratio from 1.324× to 1.038×. Encoding throughput is similar for the two variants, while Top-16 is 3.07× faster in decoding.
Table 3: Top-k exponent coding. Metric
Top-8
Top-16
Code width Coverage Ratio Encode(GB/s) Decode(GB/s) Escape rate
3-bit 92.11% 1.038× 440.1 710.5 7.89%
4-bit 99.84% 1.324× 613.3 2181.8 0.16%
The slowdown of Top-8 comes from both higher escape overhead and less GPU-friendly packing. Reducing the codebook from 16 to 8 entries lowers coverage from 99.84% to 92.11%, increasing the escape rate from 0.16% to 7.89%. This nearly 50× higher escape rate increases memory-intensive escape collection and sparse overwrite costs. In addition, 3-bit codes do not align naturally with byte-oriented GPU operations, unlike 4-bit codes where two exponents form one byte. 4.3.2
Ablation on Calibration Dataset
SplitZip uses a calibrated top-16 exponent Table 4: Cross-dataset calibration coverage. Dataset codebook to avoid online histogram construc- A is WikiText-2 train. tion. To test whether this codebook is datasetQwen3-32B Qwen3-Coder-30B Llama-3-8B specific, we calibrate on WikiText-2 train and Eval set evaluate coverage on datasets from different A→B B→B A→B B→B A→B B→B domains. Table 4 compares cross-dataset cal- WikiText-2 99.83% 99.83% 99.48% 99.48% 99.85% 99.85% ibration, denoted A→B, with oracle calibra- HumanEval 99.81% 99.83% 99.44% 99.49% 99.84% 99.85% GSM8K 99.79% 99.80% 99.41% 99.45% 99.78% 99.79% tion on each target dataset, denoted B→B. MMLU
99.78% 99.79% 99.38%
99.33%
99.77% 99.80%
99.76% 99.76% 99.34% 99.34% 99.73% 99.74% The calibrated exponent set generalizes well. PTB Coverage remains higher than 99% across all evaluated datasets, and almost matches the oracle result on WikiText-2 test, HumanEval [4], GSM8K [5], MMLU [8], and PTB [32] in this run. This shows that high-frequency KV-cache exponents are stable across domains, enabling SplitZip to use a fixed offline codebook.
4.3.3
Ablation on Calibration Granularity
Some prior compression designs use finer-grained code- Table 5: Calibration granularity ablation. books to improve locality, compression ratio, or paralMetric Tensor Token Channel lelism. We evaluate this idea on per-tensor, per-token and Coverage 99.84% 99.90% 99.90% per-channel codebooks. Compression
1.324×
1.324×
1.327×
0.082 0.020 Table 5 shows that finer-grained calibration increases top- Encode(GB/s) 613.3 Decode(GB/s) 2181.8 0.173 0.060 16 coverage, reaching 99.90% with per-token and perchannel codebooks. However, throughput drops by several orders of magnitude: per-token and per-channel calibration both fall from hundreds of GB/s to sub-GB/s performance. This is because fine-grained calibration requires many small codebooks, introducing extra storage, irregular lookup patterns, and more complex GPU execution. Therefore, SplitZip uses a single global codebook, which preserves nearly the same compression ratio while maintaining a simple and high-throughput dense path.
4.3.4
Ablation on Escape-Position Metadata
We also study whether SplitZip should explicitly store escape positions or reserve one code as an escape sentinel. The sentinel design uses only 15 frequent exponent values and reserves the remaining 4-bit code to mark escaped elements. This avoids storing explicit chunk-local positions and therefore slightly improves the compression ratio, from 1.324× to 1.331×. However, this metadata saving comes at a large decoding cost. 8
Table 6: Escape-position ablation. Metric Coverage Escape rate Ratio Encode(GB/s) Decode(GB/s)
Top-16 + Pos.
Top-15 + Sent.
99.84% 0.16% 1.324× 613.3 2181.8
99.73% 0.27% 1.331× 396.0 620.8
As shown in Table 6, the Top-15 sentinel design reduces decode throughput from 2181.8 GB/s to 620.8 GB/s, a 3.5× slowdown. The reason is that the decoder can no longer treat all 4-bit codes uniformly. It must inspect the dense code stream to identify sentinel values and then merge escaped exponents back into the output, introducing irregular control flow and memory access. In contrast, the Top-16 design decodes every element through the same dense lookup path and applies rare escape corrections through a separate sparse overwrite. This requires explicit position, making the metadata overhead a little higher. However, considering overall codec throughput, SplitZip chooses Top-16 coding with explicit escape positions to prioritize decoding throughput and GPU regularity over a marginal compression-ratio gain. 4.3.5
Ablation on Pre-Calibration
We compare SplitZip’s offline pre-calibrated codebook Table 7: Pre-calibration ablation. with a dynamic Top-16 variant that rebuilds the exponent Metric Pre-calib. Dynamic codebook for each input. As shown in Table 7, dynamic calibration achieves the same compression ratio and esRatio 1.324× 1.324× cape rate as the pre-calibrated design, and decode throughEscape rate 0.16% 0.16% Encode(GB/s) 613.3 80.7 put is almost unchanged. However, it substantially slows Decode(GB/s) 2181.8 2197.3 down online compression: encode throughput drops from 613.3 GB/s to 80.7 GB/s, 7.6× slower, because the dynamic variant adds an online histogram and top-k selection pass. 4.3.6
Ablation on Layer-Wise Coverage
.8%
99
.8%
-99
99 .5-
.5%
-99
99 .0-
<9
9.0
%
Number of Layers
We further test whether a small number of layers ex- Figure 5: Layer-wise coverage under a fixed hibit heavier-tailed exponent distributions that would shared Top-16 codebook on Qwen3-32B. increase the escape rate. For Qwen3-32B, we profile 61 64 BF16 KV caches from all 64 layers, select one shared K cache V cache Top-16 codebook from the aggregate K-cache distri44 48 bution and one shared Top-16 codebook from the ag32 gregate V-cache distribution, and then apply each fixed 13 16 codebook back to every layer. Figure 5 reports the re5 2 2 1 0 sulting layer-wise coverage histogram. It directly mea0 sures whether the deployed fixed codebook produces low-coverage outlier layers. The fixed K-cache codebook is stable across layers: all 64 layers remain above 99.0% coverage, and 61 layers remain above 99.8%. The V-cache codebook shows a modest lower-coverage tail, with two early layers below 99.0% and a worst-layer coverage of 98.77%. However, even this worst case corresponds to a 1.23% escape rate, and the median V layer still reaches 99.88% coverage. Thus, the tail is real but not large enough to dominate the codec.
5
Conclusion
This paper presents SplitZip, a GPU-friendly lossless compression scheme for KV-cache transfer in PD disaggregated LLM serving. SplitZip encodes frequent exponent values with fixed-length codes, and correcting rare values through an explicit escape stream, SplitZip provides a highly parallel codec for the latency-critical communication path. Our experiments show that SplitZip achieves higher compression and decompression throughput than prior lossless neural-network compressors, and translates this throughput into end-to-end KV-transfer speedups for long-context workloads. SplitZip is most effective when KV-cache communication is a major bottleneck; for short contexts or compute-bound regimes, codec overhead may outweigh the reduced transfer volume. Its compression ratio is also bounded by the redundancy in floating-point exponent values, making it complementary to more aggressive lossy or model-aware compression methods. From a societal perspective, SplitZip can reduce the cost, latency, and energy footprint of long-context LLM serving without changing model outputs. However, improved serving efficiency may also increase total LLM usage or make powerful long-context models easier to deploy for harmful applications, so it should be used together with appropriate governance and safety mechanisms. 9
References [1] Aditya Agrawal, Albert Magyar, Hiteshwar Eswaraiah, Patrick Sheridan, Pradeep Janedula, Ravi Krishnan Venkatesan, Krishna Nair, and Ravi Iyer. Quad length codes for lossless compression of e4m3, 2026. URL https://arxiv.org/abs/2602.17849. [2] Aditya Agrawal, Albert Magyar, Hiteshwar Eswaraiah, Patrick Sheridan, Pradeep Janedula, Ravi Krishnan Venkatesan, Krishna Nair, and Ravi Iyer. Single-stage huffman encoder for ml compression, 2026. URL https://arxiv.org/abs/2601.10673. [3] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150. [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel HerbertVoss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. [5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. [6] LMSYS Corp. Sglang. https://github.com/sgl-project/sglang, 2026. [7] Ruibo Fan, Xiangrui Yu, Xinglin Pan, Zeyu Li, Weile Luo, Qiang Wang, Wei Wang, and Xiaowen Chu. Zipserv: Fast and memory-efficient llm inference with hardware-aware lossless compression. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’26, page 2264–2280, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400723599. doi: 10.1145/3779212.3790250. URL https://doi.org/10.1145/3779212.3790250. [8] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/ abs/2009.03300. [9] Moshik Hershcovitch, Andrew Wood, Leshem Choshen, Guy Girmonsky, Roy Leibovitz, Ilias Ennmouri, Michal Malka, Peter Chin, Swaminathan Sundararaman, and Danny Harnik. Zipnn: Lossless compression for ai models, 2024. URL https://arxiv.org/abs/2411.05239. [10] Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=0LXotew9Du. [11] Hongyi Jia, Jinghui Zhang, Lu Fang, Stephen Chen, Yan Cui, Ye (Charlotte) Qi, and Zijing Liu. Disaggregated inference at scale with pytorch & vllm. https://pytorch.org/blog/ disaggregated-inference-at-scale-with-pytorch-vllm/, September 2025. PyTorch Blog. Accessed: April 26, 2026. [12] Weiqing Li, Guochao Jiang, Xiangyong Ding, Zhangcheng Tao, Chuzhan Hao, Chenfeng Xu, Yuewei Zhang, and Hao Wang. Flowkv: A disaggregated inference framework with low-latency kv cache transfer and load-aware scheduling, 2025. URL https://arxiv.org/abs/2504.03775. [13] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.net/forum?id=poE54GOq2l. [14] Zheng Li, Weiyan Wang, Ruiyuan Li, Chao Chen, Xianlei Long, Linjiang Zheng, Quanqing Xu, and Chuanhui Yang. A high-throughput gpu framework for adaptive lossless compression of floating-point data, 2025. URL https://arxiv.org/abs/2511.04140.
10
[15] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843. [16] Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. [17] MicroSoft. Phi-2: The surprising power of small language models. https://neurips.cc/media/ neurips-2023/Slides/83968_5GxuY2z.pdf, 2023. NeurIPS 2023. [18] NVIDIA. Dynamo. https://github.com/ai-dynamo/dynamo, 2026. [19] NVIDIA. Nvcomp. https://github.com/NVIDIA/nvcomp, 2026. [20] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting, 2024. URL https: //arxiv.org/abs/2311.18677. [21] Ruoyu Qin, Weiran He, Yaoyu Wang, Zheming Li, Xinran Xu, Yongwei Wu, Weimin Zheng, and Mingxing Zhang. Prefill-as-a-service: Kvcache of next-generation models could go cross-datacenter, 2026. URL https://arxiv.org/abs/2604.15039. [22] Qin Ruoyu, Li Zheming, He Weiran, Cui Jialei, Tang Heyi, Ren Feng, Ma Teng, Cai Shangming, Zhang Yineng, Zhang Mingxing, Wu Yongwei, Zheng Weimin, and Xu Xinran. Mooncake: A kvcache-centric disaggregated architecture for llm serving. ACM Trans. Storage, nov 2025. ISSN 1553-3077. doi: 10.1145/3773772. URL https://doi.org/10.1145/3773772. [23] vLLM Team. vllm. https://github.com/vllm-project/vllm, 2026. [24] Zirui Wang, Tingfeng Lan, Zhaoyuan Su, Juncheng Yang, and Yue Cheng. Zipllm: Efficient llm storage via model-aware synergistic data deduplication and compression, 2025. URL https://arxiv.org/abs/ 2505.06252. [25] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NG7sS51zVF. [26] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. [27] Patrick Yubeaton, Tareq Mahmoud, Shehab Naga, Pooria Taheri, Tianhua Xia, Arun George, Yasmein Khalil, Sai Qian Zhang, Siddharth Joshi, Chinmay Hegde, and Siddharth Garg. Huff-llm: End-to-end lossless compression for efficient llm inference, 2025. URL https://arxiv.org/abs/2502.00922. [28] Tianyi Zhang, Jonah Wonkyu Yi, Zhaozhuo Xu, and Anshumali Shrivastava. KV cache is 1 bit per channel: Efficient large language model inference with coupled quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum? id=pNnvzQsS4P. [29] Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, and Anshumali Shrivastava. 70% size, 100% accuracy: Lossless LLM compression for efficient GPU inference via dynamic-length float (DFloat11). In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=xdNAVP7TGy. [30] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=RkRrPp7GKO. [31] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, OSDI’24, USA, 2024. USENIX Association. ISBN 978-1-939133-40-3. [32] Junru Zhou and Hai Zhao. Head-driven phrase structure grammar parsing on penn treebank, 2020. URL https://arxiv.org/abs/1907.02684.
11
A
The Potential of Hiding Codec Overhead with Pipeline Overlap
In prefill–decode disaggregated serving, KV-cache compression, network transfer, and decompression can be executed as a streaming pipeline. This is important because the latency of a standalone codec kernel does not necessarily translate into additional end-to-end latency: if communication is the bottleneck, the encode and decode stages can be overlapped with data transfer. Let S denote the raw KV-cache size, ρ the compression ratio, Genc the compression throughput, Gdec the decompression throughput, and B the physical communication bandwidth for transferring compressed bytes. For one pipeline chunk, the three stage times are Tenc =
S , Genc
Txfer =
S , ρB
Tdec =
S . Gdec
For a long stream of chunks, the steady-state pipeline time is dominated by the slowest stage: S S S Tpipe = max , , . Genc ρB Gdec The codec overhead is fully hidden when the transfer stage remains the bottleneck, i.e. Txfer ≥ Tenc and Txfer ≥ Tdec . Equivalently, the maximum physical communication bandwidth for fully hiding both encoding and decoding overhead is Bhide =
min(Genc , Gdec ) . ρ
Thus, a faster codec increases the range of communication bandwidths under which SplitZip behaves as a purely bandwidth-saving transformation with no exposed codec latency. In our implementation, SplitZip achieves a compression throughput of Genc = 613.3 GB/s and a decompression throughput of Gdec = 2181.8 GB/s. Since compression is the slower codec stage, the hiding threshold is determined by Genc : min(613.3, 2181.8) 613.3 = ≈ 463.2 GB/s. 1.324 1.324 Therefore, SplitZip can fully hide its codec overhead as long as the physical communication bandwidth is no higher than approximately 463.2 GB/s. Bhide =
This threshold is well above the bandwidth of common inter-node communication technologies. For example, 400 Gb/s and 800 Gb/s Ethernet correspond to approximately 50 GB/s and 100 GB/s, respectively, and PCIe 5.0 x16 provides roughly 64 GB/s per direction. Thus, under typical datacenter interconnects, SplitZip’s encoding and decoding overheads can in principle be fully covered by communication time. In this regime, SplitZip behaves as a bandwidth-saving transformation rather than adding exposed codec latency to the end-to-end prefill–decode transfer path. Only very high-bandwidth intra-node GPU fabrics, such as modern NVLink-class interconnects, approach or exceed this threshold. In those settings, communication may no longer be the sole bottleneck, and codec throughput can again become visible in the end-to-end critical path.
B
SplitZip on FP8
SplitZip can also be applied to FP8 KV caches by adapting the exponent code width to the FP8 format. We consider both E5M2 and E4M3. For E5M2, the 5-bit exponent can be compressed with either a 4-bit top-16 code or a 3-bit top-8 code. For E4M3, a 4-bit code would not reduce the exponent size, so SplitZip uses a 3-bit top-8 code. FP8 offers less redundancy than BF16 because the original representation is already smaller, but the same fixed-code-plus-escape principle still applies. Table 8 report the FP8 results of SplitZip. Since FP8 already halves the KV-cache size compared with BF16, the remaining lossless compression opportunity is smaller. Nevertheless, SplitZip still provides additional compression over native E5M2 FP8. E4M3 Top-8 expands the payload because its exponent field is already compact and the escape metadata is too large for this distribution. E5M2 offers more redundancy, and the Top-16 variant achieves both the highest compression ratio and the lowest escape rate. 12
Table 8: FP8 codec throughput and compression ratio. Metric Coverage Ratio vs. FP8 Ratio vs. BF16 Encode(GB/s) Decode(GB/s) Escape rate
E4M3 Top-8
E5M2 Top-8
E5M2 Top-16
92.17% 0.933× 1.866× 219.6±1.5 366.9±2.5 7.83%
92.28% 1.049× 2.097× 221.6±14.6 340.8±0.9 7.72%
99.84% 1.136× 2.273× 249.7±0.8 564.9±6.5 0.16%
Table 9: Lossless BF16 correctness across context lengths. All rows reconstruct the KV cache bitwise and produce zero measured logit difference. Tokens
Text match
Max logit diff
Errors
128 256 494 970 1939 2840
Yes Yes Yes Yes Yes Yes
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0 0 0 0 0 0
The lower escape rate of E5M2 Top-16 also improves throughput: it reaches 249.7 GB/s encoding and 564.9 GB/s decoding, compared with 221.6 GB/s and 340.8 GB/s for E5M2 Top-8. This mirrors the BF16 trend that reducing escapes is more important than shrinking the dense code width when sparse correction dominates the tail cost. So, it’s suggested that Top-16 is the preferable FP8 setting for the measured Qwen3-32B workload.
C
Lossless check
We verify that SplitZip is strictly lossless by reconstructing the KV cache and comparing the resulting model outputs against the uncompressed baseline. As shown in Table 9, across context lengths from 128 to 2840 tokens, SplitZip achieves exact text match, zero reconstruction errors, and zero measured logit difference. This confirms that SplitZip preserves the BF16 KV cache bitwise and does not introduce any numerical deviation in model execution.
13