ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding

arXiv:2610.33719v1 [cs.AR] 24 Oct 2026

ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding Yuetao Chen

Peng Wang

Hong Xu

The Chinese University of Hong Kong

Unaffiliated

The Chinese University of Hong Kong

Abstract—The key-value (KV) cache places growing demands on GPU memory capacity and bandwidth as large language models serve longer contexts and more concurrent requests. Its footprint limits concurrency, while repeated KV reads during attention can make decoding memory-bandwidth bound. Prior lossless KV cache compression methods reduce transfers between devices but restore the uncompressed KV cache in GPU memory before attention. They therefore leave these repeated KV reads unchanged and require additional work and memory for decompression. We present Z IP F USION, which enables attention to consume the losslessly compressed KV cache directly. ZipFusion co-designs the compression format and attention kernel to reduce the bytes read from GPU memory during attention. The kernel reconstructs the original values directly in registers using simple bit operations, avoiding a separate decompression pass and easing the memory-bandwidth bound. ZipFusion also reassigns memory freed by compression across layers to store additional KV cache. Its memory manager reserves space for ongoing generation and makes the remaining capacity available to new requests. We implement ZipFusion on the top of FlashInfer and integrate it with SGLang. On an NVIDIA H100, ZipFusion achieves about 1.25× speedup over FlashInfer in decode attention. With 8K context, it improves model decode throughput by up to 26.3% over uncompressed SGLang under the same GPU memory budget. At each method’s largest supported batch, it also delivers up to 2.76× the throughput of the best-performing lossless KV cache compression baseline.

I. I NTRODUCTION Serving large language models (LLMs) with long contexts requires substantial GPU memory capacity and bandwidth. The key-value (KV) cache grows with context length and the number of concurrent requests, while dense attention reads the KV cache at every decoding step. Its footprint therefore limits how many requests can run together [1], and its repeated reads can limit decoding speed [2]. KV cache compression offers an opportunity to ease both constraints by reducing the amount of data stored and accessed during generation. To realize these benefits, attention must read the compressed KV cache efficiently, and the serving system must be able to use the saved memory for additional requests. Compression methods also differ in whether they preserve the values in the KV cache. Lossy compression methods, such as quantization [3]–[5], reduce storage by approximating the values in the original KV cache. Other methods retain only selected tokens in the KV cache [6], [7]. Their effects on model quality depend on the model and workload. In longcontext retrieval and question-answering experiments, Li et al. [8] find that most tested models are more sensitive to KV

KV decompression

Other decode time

16.2

SplitZip 32.7%

37.5

70.9%

IBP

41.8

73.8%

LZ4 0

10

20

30

40

Time per decode step (ms)

Fig. 1. Instrumented decode time for lossless KV cache codecs adapted to SGLang for Llama 3.1 8B, with BF16, batch size 32, 2048 input tokens. Each bar averages six repetitions of 256 decoding steps. Percentages show the decompression share. Prefill and initial compression are outside timing.

cache quantization than to weight quantization at the same bit width. We therefore focus on lossless KV cache compression for deployments that require the values in the original KV cache to be preserved. KV cache transfers arise both in offloaded inference [9] and in serving that separates prefill from decoding [10], [11]. Prior work uses lossless KV cache compression to reduce these transfer costs. IBP targets transfers between CPUs and GPUs, while SplitZip targets transfers between servers that process prompts and those that generate outputs [12], [13]. These approaches reduce the bytes transferred, but attention still reads the uncompressed KV cache from GPU memory after reconstruction. The repeated KV cache reads that can make decoding memory-bound therefore remain unchanged. Decompression also requires execution time and GPU memory for the restored values. Figure 1 measures this cost for three lossless codecs adapted to SGLang, each decompressing the KV cache before attention. Decompression accounts for 32.7% to 73.8% of instrumented decode time in this configuration. Using compressed KV cache directly for attention can reduce the bytes read at each decoding step. Reconstructing values within the attention kernel avoids a separate decompression pass and the need to store its uncompressed output in GPU memory. The space released by compression can then be used to admit additional requests. We present Z IP F USION, which enables attention to consume the losslessly compressed KV cache directly without maintaining an uncompressed copy in GPU memory. Completed KV cache pages remain compressed in GPU memory. Attention reads the compressed KV cache blocks and reconstructs the

values it needs directly in registers. The KV cache manager reports the additional GPU memory made available by compression to the scheduler, allowing it to admit more requests. Keeping the KV cache compressed reduces data movement both between devices and during attention, lowers the KV cache memory footprint, and eases the memory bandwidth bottleneck during decoding. Realizing these benefits requires addressing two challenges. First, reducing decode latency requires a lossless compression format that attention can use efficiently. The time saved by reading fewer bytes can otherwise be offset by the work needed to reconstruct values from the KV cache. We design the compression format and attention kernel together, arranging compressed keys and values to match their respective matrix computations and reconstructing values directly in registers with simple bit operations. Second, the memory saved by compression is needed both to serve additional requests and to accommodate KV cache growth for requests already running. The KV cache manager tracks how much space is available for KV cache writes as generation and compression proceed. The scheduler uses this information to admit additional requests while reserving space for ongoing generation. We implement ZipFusion for the KV cache in the 16-bit bfloat16 (BF16) format, on the top of FlashInfer [14] for attention and SGLang [15] for the KV cache manager. We evaluate Llama 3.1 8B Instruct and Qwen3-8B on an NVIDIA H100. Across the evaluated configurations, ZipFusion achieves 1.23× to 1.25× speedup over FlashInfer in decode attention. With 8K inputs, it increases model decode throughput by 26.3% for Llama and 25.8% for Qwen over uncompressed SGLang. Each system uses its largest supported batch under the same GPU memory budget. Relative to the best-performing lossless KV cache compression baseline, it delivers 2.76× throughput for Llama and 2.61× for Qwen. II. BACKGROUND AND M OTIVATION A. KV Cache and BF16 LLM inference begins with prefill, which processes the input prompt and builds the KV cache. Decoding then generates one token per request at each step and extends the KV cache. Dense attention compares the current query with stored keys to compute attention weights, then combines stored values using these weights. The same portion of the KV cache can therefore be read many times after it is written. Longer contexts increase these reads and the memory occupied by the KV cache [2]. Serving systems commonly organize the KV cache into pages, each covering a fixed number of tokens, and allocate more pages as requests grow [1]. We consider the KV cache stored in BF16, a 16-bit floatingpoint format. Each value contains one sign bit, eight exponent bits, and seven fraction bits. Lossless compression preserves the original bit pattern of every value, including all three fields. B. Compression for Faster Decode Attention Efficient attention kernels reduce GPU memory traffic through tiling [16] and improve execution through better work

partitioning [17] and reduced softmax synchronization [18]. KV cache compression can further reduce the bytes that these kernels read. Lossless compression exploits repeated exponents or other bit patterns in the KV cache. IBP and SplitZip use this redundancy to reduce transfer costs [12], [13]. The repeated use of the KV cache creates a further opportunity. Once a completed KV cache block is compressed, subsequent decoding steps can read fewer bytes each time they consume it. Restoring the KV cache after transfer leaves these repeated reads uncompressed. Keeping it compressed for attention instead allows one encoding operation to benefit many decoding steps. This makes attention execution an important target for lossless KV cache compression. Keeping the KV cache compressed also makes reconstruction part of the decoding cost. Figure 1 examines this cost in a common SGLang decode path. We run Llama 3.1 8B with 2048 input tokens and batch size 32 on one H100. SplitZip, IBP, and nvCOMP LZ4 [19] each retain the compressed prompt KV cache and restore one layer into a shared buffer before FlashInfer attention. We measure six repetitions of 256 decoding steps, with prefill and initial compression outside timing. Decompression takes 5.3 to 30.8 ms per step, while the remaining instrumented time stays near 10.9 ms across the three codecs. The decompression intervals sum K and V reconstruction across all layers. The remaining time includes attention and other model work. Attention still reads the restored KV cache from GPU memory, so separate reconstruction adds work without reducing these repeated reads. The measurement identifies an execution cost that compression ratio alone does not capture. For lossy KV cache compression, Gao et al. [20] find that throughput benefits depend on batch size and prompt length. Shannon entropy provides a reference for how compactly symbols can be encoded on average [21], but does not determine how quickly attention can locate and reconstruct the values it needs. Reconstruction within attention avoids that intermediate buffer, but its instruction cost still affects latency. The compression format therefore needs to support inexpensive reconstruction and arrange keys and values for their respective attention computations. Designing the format and attention together offers a way to retain the bandwidth savings while limiting the work needed to use the compressed KV cache. C. Compression for Higher Serving Capacity Compression can also make room for additional requests within the same GPU memory budget. The available space changes as completed KV cache pages are compressed and running requests extend their KV cache. Scheduling based on a fixed capacity derived from uncompressed page sizes cannot use this additional space. Yet admitting more requests requires accounting for both their KV cache and the space needed by ongoing generation. The relevant capacity is therefore the physical space available for further KV cache writes. Exposing

this capacity to the scheduler connects compression to request admission. More concurrent requests also increase the amount of KV cache read at each decoding step [2], linking this capacity opportunity to the need for efficient attention over the compressed KV cache. III. Z IP F USION D ESIGN A. Overview ZipFusion enables attention to consume compressed KV cache directly. A KV cache page groups data for a fixed number of tokens. Within each layer, the page’s keys and values are divided into smaller compression blocks that are encoded independently. GPU high-bandwidth memory (HBM) is allocated in larger chunks, each of which can hold multiple pages. Figure 2 shows how ZipFusion combines KV cache compression, attention, and memory management. Figure 2(a) illustrates page compression. Within each page, blocks that satisfy the compression condition shrink to 75% of their original size, while the remaining blocks stay unchanged. Attention reads these blocks in their stored formats and restores compressed values in registers immediately before computation, as shown in Figure 2(b). This reduces memory reads without writing the restored values back to GPU memory. Compression also frees space for additional KV cache pages. As shown in Figure 2(c), memory chunks released by one layer can be reassigned to another layer, allowing layers to share the memory saved by compression. B. Lossless KV Cache Format ZipFusion divides the KV cache within each page into blocks and compresses each block independently. When the exponents within a block span a small range, we encode them as offsets from a shared base while preserving every sign and fraction bit. Blocks that do not meet this condition remain uncompressed. Let s, e, and f denote the unsigned sign, exponent, and fraction fields of a BF16 value. Here, e is the unsigned integer stored in the eight-bit exponent field. The base b is the minimum exponent in the block rounded down to an even integer. A block is compressed when every exponent e lies in [b, b + 15]. Otherwise, it remains uncompressed. We retain the low byte ℓ of each BF16 value unchanged. This byte contains all seven fraction bits and the lowest exponent bit. For a compressed block, the exponent offset e−b fits in four bits. Because b is even, the lowest bit of this offset is already preserved in ℓ. We therefore store only its upper three bits,   e−b . (1) d= 2 For each value, we combine the sign bit s and the three bits of d into a four-bit code, denoted by c. Its highest bit stores s, and its lower three bits store d. Each value thus requires one retained byte and one four-bit code, or 12 bits in total. Two codes are packed into one byte, as illustrated in Figure 3(a).

(a) Compress completed pages Before compression

After compression

page 0

1024 B

encode

page 1

1024 B

raw

page 2

1024 B

encode

page 3

1024 B

low bytes

768 B 1024 B 768 B

encode 4-bit codes

768 B uncompressed values

(b) Attention over the compressed KV cache HBM

low bytes

Shared memory

low bytes

4-bit codes

page 2

copy compressed 4-bit codes reconstruct

Registers

BF16

QKᵀ

Q

Scores

(c) Share unused space across layers used chunk

Shared HBM pool

Layer 0

free reassign

Layer 1 data

needed reassignment

Fig. 2. KV cache compression, attention, and memory allocation. (a) blocks that satisfy the compression condition shrink to 75% of their original size, while the remaining blocks stay unchanged. (b) Data from the shown block of page 2 remain compressed through shared memory and are restored to BF16 in registers. (c) Each rectangle represents a memory chunk that can hold multiple pages. The empty chunk in layer 0 is reassigned to layer 1.

An eligible block therefore occupies 75% of its original size, excluding metadata. To restore the exponent, we read the three-bit offset d from the code c and the retained exponent bit r from the highest bit of the low byte ℓ. The original exponent is then ê = b + 2d + r.

(2)

The sign is recovered from the highest bit of c, and the seven fraction bits are retained in ℓ. All three fields are therefore restored exactly. These integer operations on bit fields introduce no floating-point rounding. For example, consider a 16-token page with a head dimension of 128. For each KV head, the page holds 16 × 128 = 2048 elements for keys and the same number for values. Dividing each set into 512-value blocks gives four blocks for keys and four for values. Each eligible block then occupies 768 bytes instead of 1024 bytes, excluding metadata. Within each compressed block, we store the retained low bytes contiguously, followed by the packed four-bit codes. To locate blocks within a page, attention also needs their starting positions and sizes. For each KV head, we store separate metadata for the four key blocks and the four value blocks. Each group uses eight bytes of metadata. Four bytes

(a) Encode two BF16 values · 4 B → 3 B b = 126

x₁ = 0xBFB6 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 0

1

s

s₁

e = 127

fraction

0

x₅ = 0xC1DA 1 1 0 0 0 0 0 1 1 1 0 1 1 0 1 0

1

0

s₅

e = 131

Code bytes

c₁

d = ⌊(e−b)/2⌋

0

0 s₁ s₅

d₅

d₁

c₅

1

010

000

1

ℓ₅ = 0xDA

0 8 bits

Retained low bytes + b/2 (0x3F/byte)

Sign + offset · 32 bits

⋯

10000000

⋯

⋯

0xB6

ℓ₂

ℓ₃ Restored BF16 values

High bytes · 32 bits

h₀

Add

0xBF

h₂

h₃

x₀

0xBFB6

x₂

x₃

x₆

x₇

Join

0x10 0xD0 0x19 0xD9

x₁, x₅ · (a)

1

d₅ = 2

ℓ₀

Code bytes · 32 bits

ℓ₁ = 0xB6

0xD0

d₁ = 0

(b) Restore eight BF16 values in registers

Mask

Low bytes

ℓ₄

0xDA

ℓ₆

ℓ₇

h₄

0xC1

h₆

h₇

32-bit register

+ b/2 (0x3F/byte)

Unpack

⋯

10000010

⋯

⋯

Add

x₄

0xC1DA

Join

Fig. 3. Lossless encoding and register reconstruction. Panel (a) follows two values from the eight-value group reconstructed by one thread in (b). Each branch in (b) restores four high bytes with one 32-bit addition. Join pairs each restored high byte with its retained low byte to recover the original BF16 value. Yellow and green track x1 and x5 across both panels. Subscripts identify the element to which each field belongs. Blue denotes retained low bytes ℓi , and hi denotes the high byte of xi . The prefix 0x marks hexadecimal values; bit strings are binary. Data byte counts exclude metadata.

store the starting offset of the first block, and the remaining four bytes describe the blocks individually. Each description either marks a block as uncompressed or stores its exponent base divided by two, b/2. This identifies whether the block occupies 1024 or 768 bytes and supplies the base needed for reconstruction. Attention locates each subsequent block by adding the sizes of the preceding blocks to the starting offset. C. Attention over the Compressed KV Cache Attention computes scores QK T from queries Q and keys K, then applies scaling and softmax to obtain attention probabilities P . The second matrix operation, P V , computes a weighted sum of the values V using the attention probabilities P. Tensor Core matrix instructions require each thread to hold specific matrix elements in its registers. We arrange the KV cache to match this requirement, storing eight elements needed by the same thread together within each compression block. Keys and values use different layouts because they serve as different operands in QK T and P V . The KV cache write kernel stores elements in these layouts, and compression preserves the grouping. For each compressed group, a thread reads eight retained low bytes and four code bytes, then reconstructs the eight BF16 elements in its registers for the Tensor Core instruction. Only the high bytes, which contain the sign and upper seven exponent bits, need to be reconstructed because the low bytes are retained unchanged. We pack the codes to simplify this reconstruction in registers. Each code byte combines the codes of two values from an eight-element group. Figure 3(a)

illustrates the pair x1 and x5 . We place the sign of x1 in bit 7 and its three-bit offset in bits 2 through 0, leaving the middle four bits for the code of x5 , producing the byte 0xD0 in the example. This places the sign and offset of x1 in the positions needed to reconstruct its high byte. The sign and offset of x5 can be moved into the same positions with bit operations. The other three code bytes follow the same pattern for (x0 , x4 ), (x2 , x6 ), and (x3 , x7 ). With this packing, the thread restores four high bytes at a time, as shown in Figure 3(b). The four code bytes form one 32-bit word. In the upper branch, masking keeps the signs and offsets of x0 through x3 . In the lower branch, unpacking moves those of x4 through x7 into the same bit positions. Each branch then adds a 32-bit word with b/2 in each byte. This restores four high bytes with one integer addition. The highlighted paths restore 0xBF for x1 and 0xC1 for x5 from the packed byte 0xD0. The exponent sum fits below the sign bit, so no carry changes the sign or crosses into another byte. The restored high bytes combine with the eight retained low bytes to form eight BF16 values, packed into four 32-bit registers for the matrix operation. The attention kernel reads the metadata and copies KV cache blocks from HBM into on-chip shared memory. Compressed blocks remain compressed during this copy. Threads read their groups from shared memory and restore the values in registers immediately before use. Figure 2(b) shows this path for keys. Values follow the same steps before P V , and uncompressed blocks supply the same operands without reconstruction. The kernel fetches metadata for the next portion of the KV cache while computing attention on the current

portion. Attention consumes the compressed blocks without a separate decompression pass or an uncompressed output buffer in GPU memory.

IV. E VALUATION A. Experimental Setup Models. We evaluate ZipFusion with Llama 3.1 8B Instruct [22] and Qwen3-8B [23]. Weights and the original KV cache use BF16 and reside on a single GPU. Testbed. We use an NVIDIA H100 with 80 GB of HBM3, CUDA 13.0, PyTorch 2.11.0, and FlashInfer 0.6.12. ZipFusion is integrated into SGLang with 16-token KV cache pages. All framework comparisons enable CUDA Graphs. Baselines. We compare against uncompressed SGLang [15] with FlashInfer [14] attention and three lossless codecs, IBP [12], SplitZip [13], and nvCOMP LZ4 [19]. Their public implementations are integrated into the same SGLang decode path. Each codec decompresses a layer’s KV cache into a GPU buffer before FlashInfer attention, while ZipFusion reconstructs values within attention. Workloads and measurements. Framework experiments use fixed essay prompts with 8192 input tokens and 256 output tokens per request. We time the 255 decode steps after the first token, excluding prefill and initial compression. Newly generated KV cache remains uncompressed for all methods.

Latency (µs)

ZipFusion keeps partially filled KV cache pages uncompressed so that each decoding step can append new values directly. Once a page is full, it can be compressed after the current attention reads finish. Figure 2(a) illustrates the compressed and uncompressed blocks that result. The encoded block sizes determine the allocation needed for each page. The page identifier stays the same even if compression moves its data. A lookup table maps this identifier to the page’s current location in GPU memory. If a separate destination is unavailable, compression can finish in place. Once compression finishes, the page has no extra uncompressed copy. Each new logical KV cache page needs physical space in every layer. The number of pages that can be added therefore depends on the space available across all layers. Each layer has private storage, and a shared pool of memory chunks can provide additional space. Once a chunk from this pool is empty and outstanding reads have finished, it can return to the pool and be reassigned to another layer, as shown in Figure 2(c). This allows shared space released in one layer to serve allocations in another. The allocator determines how many additional KV cache pages the remaining memory can support across all layers. It uses the uncompressed page size because new pages must accept writes before they can be compressed. As compression releases memory, the allocator updates the available capacity. The updated capacity allows more requests to be served while preserving space for ongoing generation.

Latency (µs)

D. KV Cache Management and Scheduling

FlashInfer

ZipFusion

(a) Llama 3.1 8B, 2K

(b) Qwen3-8B, 2K

(c) Llama 3.1 8B, 8K

(d) Qwen3-8B, 8K

300 200 100 0

1000 500 0 32

64

96

Batch size

32

64

96

Batch size

Fig. 4. Decode attention latency. Points show medians of 400 samples.

Throughput is total tokens generated during these steps divided by total decode time across runs. We verify KV cache reconstruction bitwise. B. Attention Kernel Performance We benchmark attention using queries and KV cache captured from a middle layer of each model, with separate KV cache copies for each batch entry. Figure 4 compares 2048- and 8192-token contexts at batch sizes 32, 64, and 96. We compare ZipFusion against production FlashInfer. L2 is cleared before each timed call. Timing includes attention, fused reconstruction, and any split-result merge, excluding input preparation and initial compression. Across the evaluated configurations, ZipFusion achieves 1.23–1.25× speedup over production FlashInfer, corresponding to 18.5%–20.2% lower attention latency. The improvement holds for both models at both context lengths and all three batch sizes. At an 8192-token context and batch size 64, Llama attention falls from 704.6 to 562.7 µs, while Qwen falls from 704.8 to 562.5 µs. These timings include reconstruction, showing that its cost is small enough to retain a net latency benefit from compressed attention. C. Framework Decode Throughput Figure 5 compares fixed batch sizes of 16, 32, and 48, and each method’s largest supported batch. All methods use a memory fraction of 0.8, including compression metadata and temporary buffers in the memory budget. At batch size 32, ZipFusion reaches 2044 tokens/s for Llama and 1882 tokens/s for Qwen, improving on uncompressed SGLang by 15.3% and 15.2%, respectively, and achieves the highest throughput among the evaluated methods at this batch size. Relative to SplitZip, the fastest compressed baseline at batch size 32, ZipFusion delivers 2.51× throughput for Llama and 2.57× for Qwen. These gains hold with the same number of requests in each batch. By reconstructing values within attention, ZipFusion avoids the separate decompression pass

SplitZip

IBP

LZ4

ZipFusion

× Exceeds measured capacity

(b) Batch size 32

(c) Batch size 48

(d) Maximum batch

2000

B40

B46

B57

3000

Llama 3.1 8B

Qwen3 8B

Llama 3.1 8B

Qwen3 8B

× ××

× ××

Llama 3.1 8B

Qwen3 8B

B52 B39 B39

0

B44 B44

1000

B50

(a) Batch size 16

B58

Decode throughput (tokens/s)

SGLang

Llama 3.1 8B

Qwen3 8B

Fig. 5. Model decode throughput with precompressed 8192-token prefixes. Panel (d) labels each method’s largest supported batch. Whiskers show the minimum and maximum over six runs. Crosses mark batches above measured capacity.

and reads less KV cache data from GPU memory during attention. The saved memory also supports larger batches within the same budget, as shown in Figure 5(d). ZipFusion increases the supported batch size from 46 to 57 for Llama and from 40 to 50 for Qwen. Compared with SGLang at its own largest batch, throughput improves by 26.3% and 25.8%, respectively. ZipFusion reaches 2453 tokens/s for Llama and 2042 tokens/s for Qwen, about 1.26× the throughput of SGLang in both cases. The larger gains at maximum batch size combine faster attention with the capacity to serve more requests within the same memory budget. V. R ELATED W ORK IBP [12] and SplitZip [13] use lossless KV cache compression to reduce transfers between CPUs and GPUs or between prefill and decode workers. Both restore the KV cache before attention. ZipFusion keeps completed pages compressed during attention, reducing their footprint and repeated GPU memory reads. For weights, NeuZip [24] and DFloat11 [25] exploit the low entropy of exponents. Tan et al. [26] align entropy decoding with matrix multiplication tiles. ZipServ [27] fuses weight decompression with matrix multiplication, restoring operands in registers. ZipServ emphasizes GDDR GPUs and reports that reconstruction overhead can offset bandwidth savings on A100 and H800. ZipFusion targets the growing KV cache and demonstrates benefits on an H100 with HBM3. For 8192-token inputs with precompressed KV cache prefixes, it achieves up to 15.3% higher model decode throughput than uncompressed SGLang at batch size 32, and up to 26.3% when each system uses its largest supported batch under the same GPU memory budget (Section IV-C). Orca [28] schedules requests at each generation step, while Sarathi-Serve [29] chunks prefill work to limit interference with decoding. vAttention [30] allocates physical memory on demand while keeping the KV cache contiguous in virtual memory. ZipFusion connects memory reclaimed by compression to request admission while reserving space for ongoing generation.

VI. C ONCLUSION We presented ZipFusion, which co-designs lossless KV cache compression, attention, and page management to reduce KV cache memory use and accelerate decoding. Attention reconstructs KV cache values directly in registers, avoiding a separate decompression pass and an uncompressed output buffer in GPU memory. The page manager makes saved memory available to the scheduler while reserving space for continued generation. On an NVIDIA H100, ZipFusion reduces attention latency by 18.5% to 20.2% relative to FlashInfer in the evaluated configurations. With 8K input contexts and precompressed KV cache prefixes, ZipFusion improves model decode throughput by up to 26.3% over uncompressed SGLang. Each system uses its largest batch within the same memory budget. These results show that lossless KV cache compression can support both larger batches and faster decoding. R EFERENCES [1] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611– 626. [2] M. Kim, S. Hong, R. Ko, S. Choi, H. Lee, J. Kim, J.-Y. Kim, and J. Park, “Oaken: Fast and efficient LLM serving with online-offline hybrid KV cache quantization,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 482–497. [3] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “KIVI: A tuning-free asymmetric 2bit quantization for KV cache,” in Proceedings of the 41st International Conference on Machine Learning, vol. 235. PMLR, 2024, pp. 32 332–32 344. [Online]. Available: https://proceedings.mlr.press/v235/liu24bz.html [4] C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024, pp. 1270–1303. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 028fcbcf85435d39a40c4d61b42c99a4-Abstract-Conference.html [5] L. Liu, L. Cheng, H. Ren, Z. Xu, Y. Pan, M. Wang, X. Li, Y. Han, and Y. Wang, “COMET: Towards practical W4A4KV4 LLMs serving,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2025, pp. 131–146.

[6] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen, “H2 O: HeavyHitter Oracle for Efficient Generative Inference of Large Language Models,” in Advances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023, pp. 34 661–34 710. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html [7] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient Streaming Language Models with Attention Sinks,” in International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=NG7sS51zVF [8] S. Li, X. Ning, L. Wang, T. Liu, X. Shi, S. Yan, G. Dai, H. Yang, and Y. Wang, “Evaluating quantized large language models,” in Proceedings of the 41st International Conference on Machine Learning, vol. 235. PMLR, 2024, pp. 28 480–28 524. [Online]. Available: https://proceedings.mlr.press/v235/li24bb.html [9] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Re, I. Stoica, and C. Zhang, “FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU,” in Proceedings of the 40th International Conference on Machine Learning, vol. 202. PMLR, 2023, pp. 31 094–31 116. [Online]. Available: https://proceedings.mlr.press/v202/sheng23a.html [10] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 2024, pp. 193–210. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin [11] P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient Generative LLM Inference Using Phase Splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132. [12] A. K. Kamath, A. Krishnamurthy, M. Canini, and S. Peter, “Reducing the GPU memory bottleneck with lossless compression for ML,” in Proceedings of the 21st European Conference on Computer Systems, 2026, pp. 899–918. [13] Y. Guo and S. Joshi, “SplitZip: Ultra fast lossless KV compression for disaggregated LLM serving,” arXiv:2605.01708, 2026. [Online]. Available: https://arxiv.org/abs/2605.01708 [14] Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, and L. Ceze, “FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving,” in Proceedings of Machine Learning and Systems, vol. 7. MLSys, 2025. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2025/ hash/dbf02b21d77409a2db30e56866a8ab3a-Abstract-Conference.html [15] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient Execution of Structured Language Model Programs,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024, pp. 62 557–62 583. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html [16] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” in Advances in Neural Information Processing Systems, vol. 35. Curran Associates, Inc., 2022, pp. 16 344–16 359. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html [17] T. Dao, “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” in International Conference on Learning Representations, 2024, pp. 35 549–35 562. [Online]. Available: https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 98ed250b203d1ac6b24bbcf263e3d4a7-Abstract-Conference.html [18] K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, Y. Dong, and Y. Wang, “FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics,” in Proceedings of Machine Learning and Systems, vol. 6, 2024, pp. 148–161. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2024/ hash/5321b1dabcd2be188d796c21b733e8c7-Abstract-Conference.html [19] NVIDIA, “nvCOMP documentation,” Online documentation, accessed October 22, 2026. [Online]. Available: https://docs.nvidia.com/cuda/ nvcomp/

[20] W. Gao, X. Zhou, P. Sun, T. Zhang, and Y. Wen, “Rethinking key-value cache compression techniques for large language model serving,” in Proceedings of Machine Learning and Systems, vol. 7, 2025. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2025/ hash/26289c647c6828e862e271ca3c490486-Abstract-Conference.html [21] C. E. Shannon, “A Mathematical Theory of Communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. [22] Meta, “Llama 3.1 8B Instruct model card,” Hugging Face, 2024. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.1-8BInstruct [23] Qwen Team, “Qwen3-8B model card,” Hugging Face, 2025. [Online]. Available: https://huggingface.co/Qwen/Qwen3-8B [24] Y. Hao, Y. Cao, and L. Mou, “NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks,” arXiv:2410.20650, 2024. [Online]. Available: https://arxiv.org/abs/2410. 20650 [25] T. Zhang, M. Hariri, S. H. Zhong, V. Chaudhary, Y. Sui, X. Hu, and A. Shrivastava, “70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11),” in Advances in Neural Information Processing Systems, vol. 38. Curran Associates, Inc., 2025, pp. 98 966–98 994. [Online]. Available: https://papers.nips.cc/paper_files/paper/2025/hash/ 8e8b4449c80879215ff2ee47b6dd01a6-Abstract-Conference.html [26] H. Tan, Y. Chen, G. Alonso, W.-F. Wong, and B. He, “Approaching shannon bound with lossless LLM weight compression,” in Proceedings of the 53rd Annual International Symposium on Computer Architecture, 2026. [Online]. Available: https://arxiv.org/abs/2606.15789 [27] R. Fan, X. Yu, X. Pan, Z. Li, W. Luo, Q. Wang, W. Wang, and X. Chu, “ZipServ: Fast and memory-efficient LLM inference with hardware-aware lossless compression,” in Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2026, pp. 2264–2280. [28] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A Distributed Serving System for Transformer-Based Generative Models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 2022, pp. 521– 538. [Online]. Available: https://www.usenix.org/conference/osdi22/ presentation/yu [29] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/agrawal [30] R. Prabhu, A. Nayak, J. Mohan, R. Ramjee, and A. Panwar, “vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, 2025, pp. 1133–1150.

Related documents

• ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding#1456542
• Resource-Efficient Speculative Decoding for Long-Context LLM Serving#1117896
• The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token#1412633
• The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token#1394205
• RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse#1118215